[Cuis] Time to break Morphic?

[Phil (list)]

On May 16, 2012, at 9:03 AM, Juan Vuletich wrote:

> Hi Phil,
> 
> Phil (list) wrote:
>> Juan,
>> 
>> It's great to have a list to discuss all things Cuis!  And it couldn't have come at a better time.  Cuis 4.0 has inspired me to start thinking a little crazy so I wanted to put this out for some feedback....
>>  
> 
> Great!
> 
>> One area that has been bugging me for a while is Morphic.  I the concept of it but one design decision has bothered me from the first time I used it: global coordinates.  Setting aside the question of whether or not this was the right way to go, 
> 
> It was not the right way to go. There was an illuminating interview with John Maloney (co-creator of Morphic, implementor of Squeak Morphic) in the SqueakNews e-zine. I could not find it online, so I uploaded all 5 issues of SqueakNews to http://www.jvuletich.org/SqueakNews/ . The relevant part is in the third part of the interview, in the September issue. After reading this, I awaited for a couple of years for someone to follow John's advice. Then, I decided to do Morphic 3.
> 
> The whole interview is great, and every bit of SqueakNews is pure gold.
> 

Thanks, I'll check it out.

>> it has a practical implication for something I've decided to take a whack at: graphics performance.  I've just started working on a plugin for hardware backed views that is a natural fit for a specialized type of Morph.  
> 
> Please tell us more about this!
> 

The basic idea is pretty simple: right now the Squeak display is just one large dumb frame buffer.  Makes perfect sense for the systems that Smalltalk was originally designed to run on... not so much for most systems in service today.  My thought was that a first step to modernizing it would be to break it up into multiple overlapping dumb frame buffers based on the host system's native view hierarchy.  This requires a very minor change to the VM, a plugin, and changes to the Smalltalk drawing logic.  For CogDroid the VM change is two lines of code in the interface logic (i.e. no need to touch the VM itself.)  The work to be done is in the plugin (relatively minor) and Smalltalk (the bulk of the work.)  The performance win from this is that you take a lot of the display management out of the hands of Smalltalk and the VM: overlapping, reordering, and moving views with alpha blending, damage detection and redrawing all becomes free (even better: on some systems it gets a multithreaded boost) as far as the Smalltalk system is concerned (though obviously it still needs to happen in the image for any Morphic structures that would not be implemented as a native view that are contained within one... the good news is that the scope of redraw is significantly reduced.)  If any of this sounds familiar, it is conceptually somewhat similar to how Croquet deals with OpenGL re: native OpenGL views (only dumber... Croquet uses the GPU for drawing, my initial approach would not change the 2D drawing logic within Smalltalk) and how pretty much every desktop GUI today works re: nested, overlapping views.

So a basic example of how this would work can be illustrated by a Morph sitting on top of a Transcript window.  Currently, any change to either results in a redraw of portions of both in Smalltalk.  Simply moving either one causes a redraw of both (and more.)  The approach I'm proposing would make the Morph and the Transcript independent hardware-backed views that would be unaware of each other, from a pixel point of view, at the Smalltalk level.  The only downside I can see is that a view not implemented as a native view (i.e. one that exists in the traditional root view aka the display) cannot be positioned above another native view.  But as long as major UI elements like windows, menus, and stand alone morphs get implemented as native views I don't see why that would be a problem.  Taken to the logical extreme, if every UI widget were implemented as a native view (i.e. close, minimize, scrollbars, etc.) there would be practically no overhead to damage detection and redraw left in the image: the host view would determine when it needs to redraw and the image would do so on demand.  Most of the redraws should be the result of the view actually changing on the Smalltalk side (i.e. something changed within the UI) rather than redrawing as the result of damage from another unrelated view changing.

Where local coordinates come into play is that native views are based on local coordinates (at least all the GUIs I can recall developing for recently.)  If Morphic used local coordinates, I'm pretty confident that once the infrastructure was in place, just about any Morph could be turned into a native view by changing one line of code (i.e. subclass from a hardware-backed parent rather than a software-based one) without knowing or caring that behind the scenes its pixels were being dumped into a different frame buffer.  Since we haven't changed the framebuffer-based drawing model and Smalltalk is still in control of every pixel drawn, behavior would still remain bit identical across platforms with and without native views.  I'm thinking that to keep things as simple as possible, all keyboard/mouse input handling would still be handled by the root view for now.

Of course once implemented as native views, nothing says they have to all necessarily remain dumb frame buffers as it would be a fairly straightforward matter to start adding a higher-level drawing interface to further improve performance (as well as the look of the Smalltalk system) by taking advantage of the drawing capabilities that modern GUI's provide.  Though one downside to this would likely be that you'd lose the bit-identical results as you changed platforms.  It's also possible that frame buffer performance could be improved by being smarter about how it interacts with the host environment via the plugin.  But those are projects for another day... I think the basic approach outlined above will keep me busy for a while.

>> To do so in the most straightforward way, Morphs would need to be based on a local coordinate system.  Right now I'm thinking about whether to try to rework the existing Morphic structure in Cuis or create a parallel local coordinate-based class hierarchy and am interested in feedback re: the pros and cons of each approach.
>>  
> 
> A parallel hierarchy based on local coordinates can be done. But many hacks would be necessary to make it run side by side with existing morphs. And the nice properties would be limited to the new morphs.
> 
> To me, the way to do it is converting all of Morphic to use local, Float coordinates. While doing so, it is also necessary to clean Morphic, so it can evolve.
> 

That was my initial thought re: the parallel approach... seems like the same order of magnitude of work to be done with little to no benefit.  I have no issue with float coordinates for Morphs (I assume they are floats up to the point of rendering when they would need to be converted to ints at the pixel level... if they don't, I'm not clear how you'd render to the frame buffer)  I'm also in favor of anything that makes Morphic better... hence the title of this thread: let's simplify it, clean it up, break it... so that we can fix it.

>> I seem to recall you writing/speaking in favor of this in the past re: your Morphic rewrite project (how's that going BTW?) and wanted to get your thoughts as to whether local coordinate based Morphs would be something you'd like to see in Cuis.  If so, any pointers on things I should be thinking about / watching out for are appreciated.
>> 
>> Thanks,
>> Phil
>>  
> 
> I've been silent for a while wrt Morphic 3, but I'm working on it. Much of the simplification and redesign I've been doing in Morphic in Cuis is to ease this transition, and to finally have a system that is simple, consistent, and based of Float local coordinate. This has been my objective for a long time, and it still is.
> 

I'm very glad to hear this!

> Right now I have a set of about 20 change sets that add 'position' and 'extent' instance variables to Morph, and reduce a lot the use of the now obsolete 'bounds' instance variable. This is not finished yet, but it could be better for you to start playing with this than to wait until Morphic 3 is ready for prime time...

Since the plan is still to implement Morphic 3 as the drawing toolkit for Cuis at some point in the future, I'd very much like to use this as a starting point rather than doing a lot of throw away work on the existing Morphic system and having to redo it down the road.  As long as it sort of mostly works (i.e. I don't need feature complete, just usable) I can live in an alpha/beta state until it's fully baked and possibly help out a bit (at the very least, I can provide feedback/bug reports).  Your approach sounds like exactly what I'd want for native views as that matches how things work in the host GUI.

> 
> Cheers,
> Juan Vuletich

Thanks,
Phil