[Cuis] Screen Real Estate

[Phil (list)]

Casey,

On May 22, 2012, at 3:34 PM, Casey Ransberger wrote:

> When I was running on iPad, I turned on the big fonts and disabled the button bar in the browser. I found window management to be rather painful using Bert's experimental multitouch code, which is actually why I wrote the taskbar.
> 

I suspect this will be an issue on Android as well.  I'm fully expecting to need to come up with some sort of new interaction approach taking ideas from things like morphic/etoys/scratch to really get to a useful touch UI rather than the traditional Squeak/window based UI.  I'm not expecting to actually *write* much code *on* a touch device, rather I want to interact with it.

> I'm interested in knowing more about how Android makes app UI portable between touch screen sizes; at one point I thought of a meta-description framework that might be able to "do the right thing" automatically in various form factors, but however I thought about it, I couldn't help thinking that I was over-thinking (over-engineering) it. 
> 
> I couldn't think of a way to make it simple. 
> 

Android's solution isn't a free lunch as it still requires the app developer/designer to create and manage multiple screen configurations but it does provide some tools/concepts that work pretty well for multiple form factors...

First a few terms:

Layout - an XML file(s) that defines the arrangement and relationship of widgets on the screen similar to nibs created in Apple's Interface Builder.  Layouts provide similar functionality to the springs and struts mechanism and allow for growing/shrinking widgets and can contain widgets as well as other layouts and fragments.

Fragment - a logical grouping of widgets and functionality that is managed by an Activity.  Fragments were a new addition to Android as of 3.x when tablet support was formally added (Activities and Layouts were designed originally just with phones in mind.)  Fragments need to exist within an Activity for UI code and can be thought of as a refactoring of Activities into reusable sub-components.

Activity - A key building block of an app which is generally a screen that the user performs some unit of work in.  Activities are the code that manages the UI defined in a layout (which can include Fragments) and actually loads the layout(s) and fragment(s) and presents the UI to the user.  An Activity does not need to use Fragments but does need to manage them if it uses them.

A relatively simple UI, such as a basic picture viewing app, can be handled exclusively with Activities and Layouts across multiple form factors.  In fact many of the existing Android apps worked just fine when 3.0 (i.e. tablets) first came out without significant changes... for example, an Activity in a picture viewing app showing a grid of thumbnails would just display more thumbnails on a tablet screen and that might be exactly what is desired.  An example of an UI this won't work for is an email app.  That's where fragments come into play: you convert the appropriate logical components of your UI from Activities into Fragments, and what were Activities become effectively wrappers with glue code.

Using the email app example, let's assume the UI consists of a list of folders, a list of messages within a folder, and a message viewer.  The main entry point into the app would be the folder list Activity and corresponding folder list Fragment.  There would also be a message list Activity with corresponding Fragment, and a message viewer Activity with corresponding Fragment.  Most of the work gets done in the Fragments, and most of the Activities are just wrappers around their corresponding Fragment with the exception of the folder list Activity which handles the branching logic for multiple form factors.  On phones, the flow is folder list Activity->message list Activity->message viewer Activity with each Activity just loading its corresponding Fragment and handling the hand off to the next Activity as necessary.  Android handles getting back to the previous Activity as necessary.  On a tablet, the folder list Activity would load an alternate layout which displays all three Fragments on the same screen (i.e. a traditional 3 pane layout) and handles the communication and coordination between the Fragments (i.e. the user selected a different folder, update the message list and message viewer Fragments) and there would be no need to use the other Activities.  From a code standpoint both execution paths are handled in the folder list Activity depending on what kind of device the app is running on (i.e. 'oh, the tablet layout loaded... run in tablet mode' etc.)

Hopefully this all makes sense...

> It's nice to see so many folks excited about touch interfaces here. It was one of the original reasons I got into Cuis (Squeak and Pharo, at least at the time, draaaaged slowly on my iPad.)

My earlier discussion with Juan about my native views idea is specifically to address what appears to be a significant part of the problem related to display intensive code (i.e. given the less powerful ARM CPU, move a significant chunk of work out of the image/VM... though this should similarly benefit Intel-based systems as well since more VM cycles would be freed up for computation vs. display management).  Another issue that hopefully someone will take on at some point is that there is currently no JIT for ARM so we're just running the stack VM.

I'm glad to find other kindred spirits re: touch as well,
Phil