[Cuis] Fwd: Re: DIRECT version number

[Phil (list)]

On Sun, 2015-07-19 at 11:20 -0300, Juan Vuletich wrote:
> Hi Folks,
> 
> (inline)
> 
> On 7/17/2015 3:45 PM, Phil (list) wrote:
> > ...On 7/16/2015 9:57 AM, H. Hirzel wrote:
> >>> Thank you Juan, for answering.
> >>>
> >>> At the moment I feel uncomfortable using Cuis because it is currently
> >>> a quite fast moving target. I perceive quite a number of API changes
> >>> though this might be wrong. But I do not know because there are no
> >>> release notes which summarize it.
> > I totally feel Hannes' pain having gone through it a few times myself
> > over the years.  At the same time, I agree with Ken's point about not
> > getting too bogged down in process since we're in the best position
> > right now to break things for the better longer term.
> >
> >> Yes, that's true. Besides, we don't have a specification of what is an
> >> API and what is not. So, almost any change can be considered an API change.
> > That's the problem!  Both for you and us (it's a general 'Smalltalk Way'
> > issue, not specific to Cuis)  I believe it is largely fixable without
> > tons of documentation and tests: the use of private (categories or
> > method naming convention).  Bernhard pointed out an existing capability
> > a few months ago
> > http://jvuletich.org/pipermail/cuis_jvuletich.org/2015-March/001784.html
> > While I personally like the idea of private categories, one of these
> > approaches should really be seriously considered.
> >
> > Other than a few specific examples (i.e. mostly the base data types
> > including the Collection and Number class hierarchies which are pretty
> > mature and don't change all that much anyway) I would guesstimate that
> > <20% (probably closer to 10%) of the methods, and and unknown % of
> > classes (a significantly larger fraction though), should even be
> > considered public.  If things were indicated as private that would
> > accomplish two things:
> >
> > 1) We could construct some simple tooling (some combination of up-front
> > warnings in the editor and/or a lint-type tester after the fact) that
> > would allow people to identify problematic calls within their code.
> > (i.e. just because it works now doesn't mean it always will if you keep
> > calling X, Y, Z...)
> >
> > 2) It would provide a way for people to know if something needs to be
> > discussed.  i.e. 'In class X method Y is private but I need that type of
> > functionality' so you would know that there's a need and could consider
> > either making it public, and therefore supported, or rethinking its
> > implementation if you'd like to offer a better public API solution.
> 
> Ok. Doing something along these lines would be great. But please note 
> that it is a rather large amount of work. It is not something I'm able 
> or willing to do it alone. If you (any of you all, not just Phil) wants 
> to start defining the API for any part of the system or optional 
> packages, please start doing it and send the changesets to the list. If 
> we get this working, I'll start the tools you mention.
> 

Agreed that this is a lot of work and I definitely wasn't expecting you
to do it (other than reviewing the work for approval/rejection.)  I'll
be happy to work on this with anyone else who is interested.

> <slowly diverging into OT rant>
> 
> On the other hand (and please forgive me if this kills a bit of 
> momentum), I don't really think we are changing APIs that much. And I 
> don't think we'll be doing it any more in the future than we do today, 
> at least in the base image. I hope to see most of the activity and 
> incompatible changes to happen in optional packages, and affecting 
> package dependencies... Mhhh, now that I write this I think that it 
> doesn't really go against the idea of defining APIs. Maybe APIs in the 
> base image will almost never change or be a problem, but packages should 
> provide APIs too... Optional packages will (I hope!) always be 
> improving, and new ones will appear all the time.
> 

You'd likely be very surprised.  I'd say there's usually at least a
couple changes a month that breaks something, somewhere in my code.  A
few recent examples that have been discussed on the list: removing clone
broke OpenGL a bit (easy fix thanks to the on-list discussion), removing
compilerClass killed OMeta, a while back you were considering removing
storeString which would again break OMeta. etc. (that's just a few
things off the top of my head from the last few months and note that all
were probably considered minor, inconsequential changes)  However the
issue snowballs quickly if one misses a few updates and some of the
breakage is related to changes that weren't discussed on the list or
otherwise documented.  I've been managing through this recently (the
last 6 months or so) by upgrading my image very quickly after you commit
changes but I doubt most people would be able / willing to have that
level of diligence.

A lot of this breakage is no one's 'fault', it's just how a lot of
Smalltalk code has been historically written.  Two examples of how this
happens:

1) For ports of existing code, you're subject to the whims of the
original author who often used techniques described in the next item to
write their code.  And of course, code coming from Squeak/Pharo means
that there were often 10 or more possible ways they could have
implemented parts of their solution.  So after porting a few different
packages, you have several different approaches to the same kinds of
problems.

2) For one's own code because there is currently no distinction between
public and private, you tend to go either with what you already know or
whatever the first thing you find poking around the image that appears
to make sense and work.  Even in Cuis, there are still usually multiple
ways to do most things, so you end up arbitrarily picking one or two
that you like.  As a result, you quickly end up with a potpourri of
calls that may or may not be the best way to do things.  Over time, you
start building new code on top of this code and quickly end up with a
house of cards that comes crashing down as the result of a subsequent
image update.  Again, this often results in each of us as authors
deciding on superficially different solutions to the same problems.

I'm not expecting you to not do things that might break my, or anyone
else's code, going forward.  Like Hannes, I'm just looking for ways to
help manage through the changes and breakage.  One of the key aspects of
this is to have an opinion and to say 'hey, here are the one or two ways
we should be doing this'.  There may be disagreement and discussion as
to what those one or two ways are, but at the end of the day we need to
settle these issues so that both your and our lives are manageable.
That's really the main thing that a public API is, IMO.  There may be
all sorts of internal helper methods and implementation details needed
to make it all work, but like the cells of a living organism, these
details should be hidden and their use discouraged.  Hey, didn't someone
say something along those lines once? :-)
 
> If all this work, the Cuis will be a lot less about my personal effort 
> and coding style, and the packages maintained by each one of us will 
> become the main characters in this story. And this will be a very good 
> thing. The base image should be about providing a good base for doing 
> the really interesting stuff on top...
> 

I both agree (that the packages become more important) and disagree
(that Cuis fades into the background).  I still view the core Cuis image
as the philosophical and stylistic backbone that all of my stuff is
based on.  It's a constant reminder to clean up and get rid of as much
cruft as possible in my code (especially for the packages that have yet
other packages dependent on them)  I freely admit that I am mostly
falling short in my execution on this front, but it would be easy to
lose sight of entirely if Cuis didn't stay the course.

> Let me tell you a bit about how Cuis got started. Many years ago I did 
> my thesis on audio/music processing. The I wanted to turn that into a 
> novel and fun application for musicians (both pros and anyone willing to 
> play with sound). I started to think about how a good GUI for that could 
> be, giving the feeling of modifying sound by direct manipulation of it, 
> as if it was clay, as if it was visual and not just auditory. Well, that 
> was the start of my questioning of the whole GUI business and Morphic 3. 
> And it became clear to me that Squeak had turned into something that was 
> not the best possible environment to do all this. The idea of "fixing" 
> Squeak, and thus make Cuis, was born. So, Cuis was not and end in 
> itself, but a means for allowing experimentation, freedom to build 
> applications in new styles, etc.
> 
> (The idea that Cuis is a truer Smalltalk-80 than Squeak or any other 
> came later, as an observation on facts.)
> 
> So, when the moment comes when Cuis stabilizes a bit, and interest moves 
> to packages and applications built by all us, Cuis will start to fulfill 
> its original objective.
> 

It's already this is as far as I'm concerned.  I can't tell you much
time I burned just trying to extract OpenGL from Croquet so that I could
load it into Squeak, for example.  It was painful on both sides, and by
the time I was done I had to think for a bit to remind myself what my
original objective was... I have *never* had this issue with Cuis.

It is absolutely enabling me to play around with ideas and build things
that I couldn't have / wouldn't have bothered in Squeak.  The only issue
I'm having is that once I've built it, I want to keep it running :-)

> </slowly diverging into OT rant>

Not at all a rant as it reaffirms *why* you're doing what you're doing.
If you lost sight of that, that's when I'd start worrying about Cuis.

> 
> >> ...
> >> The problem, I think, is not identifying well defined Cuis releases. The
> >> problem is knowing how updates could affect your code
> > By narrowing the scope of what the public API is using private
> > designators, this dramatically reduces what you need to be concerned
> > with on this front.  It's part of the reason that Apple, Google, and
> > even Microsoft are so adamant about public APIs these days.  They dealt
> > with the pain of even major developers depending on private APIs for
> > decades. It's only been in the last 10 years or so that they've been
> > able to dig themselves out from under this issue.
> >
> > This gets back to the 'Smalltalk Way' issue I referenced earlier.  The
> > attitude of 'you have the source, play around, have fun!' has morphed
> > into 'everything is a public API!  there are no rules! isn't this
> > great?'  Well no, it's not great.  It makes life impossible for image
> > maintainers (Squeak has been in a coma for the last decade or so as the
> > result of trying to keep everyone happy... yet no one seems to be) and
> > miserable for those who are trying to keep code running on it (stuff
> > still breaks all the time.)
> 
> Yes.
> 
> > I personally believe the minimalism and simplicity that Cuis is based on
> > has the best long term prospects but we have to get out of this
> > 'everything is public' mindset.  Sure, we have all the source: look at
> > it, learn from it, poke around, refactor it, submit fixes, etc.  But
> > there needs to be a manageable subset of it which is considered the
> > 'public API' that people can depend on to write frameworks and apps on
> > top of that is relatively stable longer term.  Let's start defining what
> > that is and backfill changes to the public API (which will happen from
> > time to time as we don't want things to be completely static) with tests
> > and documentation so you can focus your efforts on where they matter
> > most and you get the most enjoyment from.
> 
> I fully agree and support this.
> Maybe an alternative to yet-another-convention-on-categories could be to 
> mark public methods with a pragma, or make them send 'self publicAPI' or 
> such... This could be better for the tools, also.
> 

I like your idea better re: rather than having to go through and mark
the majority as private, just mark the minority as public.  I'm open to
either of your proposed approaches.  I just tossed out the two (method
categories and naming) I was aware of but agree that they would not do
much to ease developing tools to support/enforce.  Of the two ideas, am
I correct in believing that pragma would have the lowest runtime
overhead?  Are there any downsides you can think of to using pragma?

Speaking of enforcing, I am not proposing that we make this a
straight-jacket for anyone: it should be easy to opt-out the way things
like underscore assignment are (except for pp which I'm still trying to
get to an easily repro example with... :-) so that anyone who doesn't
care about this isn't forced into it.  On the other hand, for people who
do care, they should be able to leave it 'enabled', whatever that ends
up meaning, and get help from the tooling to help them minimize
potential future breakage.  The downside to opting out would be that
you, as the author opting out, have decided to take it entirely on
yourself to keep your code running across builds/releases.

> > ...
> >> I don't have that either. I run the tests from time to time (I'll start
> >> doing it before any commit, and add the xml report to the rep). But this
> >> is not the most important reason why Cuis is solid. Cuis is very
> >> reliable because:
> > reliable != stable
> 
> ;)
> 
> >> - I'm not too bad as a coder.
> >> - I do my own code revisions at least one day after each change.
> >> - I use Cuis every day, and spot most problems in the updates, before
> >> commtting them to the repo.
> >> - I really, really care about code quality.
> >> - I think Cuis keeps Smalltalk-80 and its ideals alive. And I think that
> >> is a big responsibility, given how important I think Smalltalk-80 is.
> >>
> > Agree to all this and more.  Cuis would not be where it is without your
> > continued effort and diligence. (i.e. setting aside that you obviously
> > created it, if you had just tossed it out into the world and let it go
> > at best it would have started looking something like Squeak or Pharo by
> > now.  It's a lot of effort over the long term to *keep* it small and
> > focused.  THANK YOU JUAN!)
> 
> Thanks for your nice words, Phil. I really appreciate that you all 
> accept this rather restrictive dev model for the base image, in contrast 
> with a more open trunk style. Hopefully, as the base image becomes 
> stable and almost irrelevant, this will be each time less of a problem.
> 
> >> So, I suggest running your tests when updating your image, or migrating
> >> your code to an updated image.
> > +1 to tests as they do help quite a bit and are a very good practice.
> > Just be judicious in what/where you test (I personally don't subscribe
> > to the TDD approach where you spend your days mostly writing tests.
> > That's just admitting defeat in that you spend all day writing
> > executable documentation.)
> 
> <OT>
> 
> I don't subscribe to "extreme" TDD either. To me design is a creative 
> activity, both technical and artistic. It should be "driven" by 
> creativity and intelligence, not by a process. Any xDD process is 
> admitting defeat on design itself!
> 

Heh... I like the way you put that.

> </OT>
> 
> >> In any case, I think this doesn't answer your concerns. What you need is
> >> some way to know which updates could affect you, to review them in
> >> detail, and understanding their effect on your code. A list of the
> >> affected classes and/or methods for each update makes no sense. Cuis can
> >> already show you that very easily. Perhaps each update should include:
> >>
> >> 1) parts of the system that are affected:
> >> - Kernel
> >> - Compiler
> >> - Tools
> >> - Morphic
> >> - etc
> >>
> >> 2) Level of risk involved
> >> - very unlikely to break code that depends on this part of the system
> >> - could perhaps break code that depends on this part of the system
> >> - will most likely break code that depends on this part of the system
> >>
> >> Or something like that.
> >>
> >> BTW, this is a very relevant topic for discussing on the mail list. Feel
> >> free to answer there if you agree.
> > One final thought on this:  I suspect a major cause of people abandoning
> > Cuis (and Smalltalk in general), behind the 'it's too different from
> > what I'm used to'/'not mainstream' reaction, is the instability of its
> > APIs.
> 
> Well, maybe. But the alternative (the whole Python 2.8 vs. 3.0 problem) 
> is a PITA too...

True, neither extreme (changing nothing vs. changing everything) works
very well. <cough>Perl 6</cough> :-)
 
> 
> As a bottom line, let me repeat: This is too big for me to do it. 
> Please, folks, start documenting the APIs you most care about.
> 

Ready to go... we just need to agree on how to do it (pragma/method
call/method category/method name?)  Also, this is most definitely
related to the 'Canonical test cases?' thread from a few months ago as
these types of test cases / docs are part of the backfill I was
referring to.

As I've been writing this it occurs to me that there's probably a pretty
easy way to go about getting started on this: IIRC, there are ~1,500
distinct method names in the core image.  If I do a count of how many
times my code is calling any of those methods, that should pop out what,
for my code, is the low hanging fruit.  The long-tail of say 1-5 calls
for the lesser used methods are the tedious work that will take more
effort for me to sort through.  We could use my 'short list' to at least
provide a starting point and using these most frequently called methods
you could scan through the list and could say 'hey, why are you calling
X instead of Y?' or maybe you'd get some visibility in the sense of
'wow, I didn't think that was being used'.  Hmm... I've got some
analysis to do...

> Cheers,
> Juan Vuletich