[Cuis] Fwd: Re: DIRECT version number

[Juan Vuletich]

Hi Phil,

This is  very good and stong advocacy on documented public APIs. Thank you!

(a few comments inline)
On 7/19/2015 6:38 PM, Phil (list) wrote:
> On Sun, 2015-07-19 at 11:20 -0300, Juan Vuletich wrote:
>> Hi Folks,
>>
>> (inline)
>>
>> On 7/17/2015 3:45 PM, Phil (list) wrote:
>>> ...On 7/16/2015 9:57 AM, H. Hirzel wrote:
>>>>> Thank you Juan, for answering.
>>>>>
>>>>> At the moment I feel uncomfortable using Cuis because it is currently
>>>>> a quite fast moving target. I perceive quite a number of API changes
>>>>> though this might be wrong. But I do not know because there are no
>>>>> release notes which summarize it.
>>> I totally feel Hannes' pain having gone through it a few times myself
>>> over the years.  At the same time, I agree with Ken's point about not
>>> getting too bogged down in process since we're in the best position
>>> right now to break things for the better longer term.
>>>
>>>> Yes, that's true. Besides, we don't have a specification of what is an
>>>> API and what is not. So, almost any change can be considered an API change.
>>> That's the problem!  Both for you and us (it's a general 'Smalltalk Way'
>>> issue, not specific to Cuis)  I believe it is largely fixable without
>>> tons of documentation and tests: the use of private (categories or
>>> method naming convention).  Bernhard pointed out an existing capability
>>> a few months ago
>>> http://jvuletich.org/pipermail/cuis_jvuletich.org/2015-March/001784.html
>>> While I personally like the idea of private categories, one of these
>>> approaches should really be seriously considered.
>>>
>>> Other than a few specific examples (i.e. mostly the base data types
>>> including the Collection and Number class hierarchies which are pretty
>>> mature and don't change all that much anyway) I would guesstimate that
>>> <20% (probably closer to 10%) of the methods, and and unknown % of
>>> classes (a significantly larger fraction though), should even be
>>> considered public.  If things were indicated as private that would
>>> accomplish two things:
>>>
>>> 1) We could construct some simple tooling (some combination of up-front
>>> warnings in the editor and/or a lint-type tester after the fact) that
>>> would allow people to identify problematic calls within their code.
>>> (i.e. just because it works now doesn't mean it always will if you keep
>>> calling X, Y, Z...)
>>>
>>> 2) It would provide a way for people to know if something needs to be
>>> discussed.  i.e. 'In class X method Y is private but I need that type of
>>> functionality' so you would know that there's a need and could consider
>>> either making it public, and therefore supported, or rethinking its
>>> implementation if you'd like to offer a better public API solution.
>> Ok. Doing something along these lines would be great. But please note
>> that it is a rather large amount of work. It is not something I'm able
>> or willing to do it alone. If you (any of you all, not just Phil) wants
>> to start defining the API for any part of the system or optional
>> packages, please start doing it and send the changesets to the list. If
>> we get this working, I'll start the tools you mention.
>>
> Agreed that this is a lot of work and I definitely wasn't expecting you
> to do it (other than reviewing the work for approval/rejection.)  I'll
> be happy to work on this with anyone else who is interested.
>
>> <slowly diverging into OT rant>
>>
>> On the other hand (and please forgive me if this kills a bit of
>> momentum), I don't really think we are changing APIs that much. And I
>> don't think we'll be doing it any more in the future than we do today,
>> at least in the base image. I hope to see most of the activity and
>> incompatible changes to happen in optional packages, and affecting
>> package dependencies... Mhhh, now that I write this I think that it
>> doesn't really go against the idea of defining APIs. Maybe APIs in the
>> base image will almost never change or be a problem, but packages should
>> provide APIs too... Optional packages will (I hope!) always be
>> improving, and new ones will appear all the time.
>>
> You'd likely be very surprised.  I'd say there's usually at least a
> couple changes a month that breaks something, somewhere in my code.  A
> few recent examples that have been discussed on the list: removing clone
> broke OpenGL a bit (easy fix thanks to the on-list discussion), removing
> compilerClass killed OMeta, a while back you were considering removing
> storeString which would again break OMeta. etc. (that's just a few
> things off the top of my head from the last few months and note that all
> were probably considered minor, inconsequential changes)  However the
> issue snowballs quickly if one misses a few updates and some of the
> breakage is related to changes that weren't discussed on the list or
> otherwise documented.  I've been managing through this recently (the
> last 6 months or so) by upgrading my image very quickly after you commit
> changes but I doubt most people would be able / willing to have that
> level of diligence.
>
> A lot of this breakage is no one's 'fault', it's just how a lot of
> Smalltalk code has been historically written.  Two examples of how this
> happens:
>
> 1) For ports of existing code, you're subject to the whims of the
> original author who often used techniques described in the next item to
> write their code.  And of course, code coming from Squeak/Pharo means
> that there were often 10 or more possible ways they could have
> implemented parts of their solution.  So after porting a few different
> packages, you have several different approaches to the same kinds of
> problems.
>
> 2) For one's own code because there is currently no distinction between
> public and private, you tend to go either with what you already know or
> whatever the first thing you find poking around the image that appears
> to make sense and work.  Even in Cuis, there are still usually multiple
> ways to do most things, so you end up arbitrarily picking one or two
> that you like.  As a result, you quickly end up with a potpourri of
> calls that may or may not be the best way to do things.  Over time, you
> start building new code on top of this code and quickly end up with a
> house of cards that comes crashing down as the result of a subsequent
> image update.  Again, this often results in each of us as authors
> deciding on superficially different solutions to the same problems.
>
> I'm not expecting you to not do things that might break my, or anyone
> else's code, going forward.  Like Hannes, I'm just looking for ways to
> help manage through the changes and breakage.  One of the key aspects of
> this is to have an opinion and to say 'hey, here are the one or two ways
> we should be doing this'.  There may be disagreement and discussion as
> to what those one or two ways are, but at the end of the day we need to
> settle these issues so that both your and our lives are manageable.
> That's really the main thing that a public API is, IMO.  There may be
> all sorts of internal helper methods and implementation details needed
> to make it all work, but like the cells of a living organism, these
> details should be hidden and their use discouraged.  Hey, didn't someone
> say something along those lines once? :-)

Yes. :)

>> If all this work, the Cuis will be a lot less about my personal effort
>> and coding style, and the packages maintained by each one of us will
>> become the main characters in this story. And this will be a very good
>> thing. The base image should be about providing a good base for doing
>> the really interesting stuff on top...
>>
> I both agree (that the packages become more important) and disagree
> (that Cuis fades into the background).  I still view the core Cuis image
> as the philosophical and stylistic backbone that all of my stuff is
> based on.  It's a constant reminder to clean up and get rid of as much
> cruft as possible in my code (especially for the packages that have yet
> other packages dependent on them)  I freely admit that I am mostly
> falling short in my execution on this front, but it would be easy to
> lose sight of entirely if Cuis didn't stay the course.
>
>> Let me tell you a bit about how Cuis got started. Many years ago I did
>> my thesis on audio/music processing. The I wanted to turn that into a
>> novel and fun application for musicians (both pros and anyone willing to
>> play with sound). I started to think about how a good GUI for that could
>> be, giving the feeling of modifying sound by direct manipulation of it,
>> as if it was clay, as if it was visual and not just auditory. Well, that
>> was the start of my questioning of the whole GUI business and Morphic 3.
>> And it became clear to me that Squeak had turned into something that was
>> not the best possible environment to do all this. The idea of "fixing"
>> Squeak, and thus make Cuis, was born. So, Cuis was not and end in
>> itself, but a means for allowing experimentation, freedom to build
>> applications in new styles, etc.
>>
>> (The idea that Cuis is a truer Smalltalk-80 than Squeak or any other
>> came later, as an observation on facts.)
>>
>> So, when the moment comes when Cuis stabilizes a bit, and interest moves
>> to packages and applications built by all us, Cuis will start to fulfill
>> its original objective.
>>
> It's already this is as far as I'm concerned.  I can't tell you much
> time I burned just trying to extract OpenGL from Croquet so that I could
> load it into Squeak, for example.  It was painful on both sides, and by
> the time I was done I had to think for a bit to remind myself what my
> original objective was... I have *never* had this issue with Cuis.
>
> It is absolutely enabling me to play around with ideas and build things
> that I couldn't have / wouldn't have bothered in Squeak.  The only issue
> I'm having is that once I've built it, I want to keep it running :-)
>
>> </slowly diverging into OT rant>
> Not at all a rant as it reaffirms *why* you're doing what you're doing.
> If you lost sight of that, that's when I'd start worrying about Cuis.
>
>>>> ...
>>>> The problem, I think, is not identifying well defined Cuis releases. The
>>>> problem is knowing how updates could affect your code
>>> By narrowing the scope of what the public API is using private
>>> designators, this dramatically reduces what you need to be concerned
>>> with on this front.  It's part of the reason that Apple, Google, and
>>> even Microsoft are so adamant about public APIs these days.  They dealt
>>> with the pain of even major developers depending on private APIs for
>>> decades. It's only been in the last 10 years or so that they've been
>>> able to dig themselves out from under this issue.
>>>
>>> This gets back to the 'Smalltalk Way' issue I referenced earlier.  The
>>> attitude of 'you have the source, play around, have fun!' has morphed
>>> into 'everything is a public API!  there are no rules! isn't this
>>> great?'  Well no, it's not great.  It makes life impossible for image
>>> maintainers (Squeak has been in a coma for the last decade or so as the
>>> result of trying to keep everyone happy... yet no one seems to be) and
>>> miserable for those who are trying to keep code running on it (stuff
>>> still breaks all the time.)
>> Yes.
>>
>>> I personally believe the minimalism and simplicity that Cuis is based on
>>> has the best long term prospects but we have to get out of this
>>> 'everything is public' mindset.  Sure, we have all the source: look at
>>> it, learn from it, poke around, refactor it, submit fixes, etc.  But
>>> there needs to be a manageable subset of it which is considered the
>>> 'public API' that people can depend on to write frameworks and apps on
>>> top of that is relatively stable longer term.  Let's start defining what
>>> that is and backfill changes to the public API (which will happen from
>>> time to time as we don't want things to be completely static) with tests
>>> and documentation so you can focus your efforts on where they matter
>>> most and you get the most enjoyment from.
>> I fully agree and support this.
>> Maybe an alternative to yet-another-convention-on-categories could be to
>> mark public methods with a pragma, or make them send 'self publicAPI' or
>> such... This could be better for the tools, also.
>>
> I like your idea better re: rather than having to go through and mark
> the majority as private, just mark the minority as public.  I'm open to
> either of your proposed approaches.  I just tossed out the two (method
> categories and naming) I was aware of but agree that they would not do
> much to ease developing tools to support/enforce.  Of the two ideas, am
> I correct in believing that pragma would have the lowest runtime
> overhead?  Are there any downsides you can think of to using pragma?

No that I can think of. A bit of experimentation is in order.

> Speaking of enforcing, I am not proposing that we make this a
> straight-jacket for anyone: it should be easy to opt-out the way things
> like underscore assignment are (except for pp which I'm still trying to
> get to an easily repro example with... :-) so that anyone who doesn't
> care about this isn't forced into it.  On the other hand, for people who
> do care, they should be able to leave it 'enabled', whatever that ends
> up meaning, and get help from the tooling to help them minimize
> potential future breakage.  The downside to opting out would be that
> you, as the author opting out, have decided to take it entirely on
> yourself to keep your code running across builds/releases.

Sure.

>>> ...
>>>> I don't have that either. I run the tests from time to time (I'll start
>>>> doing it before any commit, and add the xml report to the rep). But this
>>>> is not the most important reason why Cuis is solid. Cuis is very
>>>> reliable because:
>>> reliable != stable
>> ;)
>>
>>>> - I'm not too bad as a coder.
>>>> - I do my own code revisions at least one day after each change.
>>>> - I use Cuis every day, and spot most problems in the updates, before
>>>> commtting them to the repo.
>>>> - I really, really care about code quality.
>>>> - I think Cuis keeps Smalltalk-80 and its ideals alive. And I think that
>>>> is a big responsibility, given how important I think Smalltalk-80 is.
>>>>
>>> Agree to all this and more.  Cuis would not be where it is without your
>>> continued effort and diligence. (i.e. setting aside that you obviously
>>> created it, if you had just tossed it out into the world and let it go
>>> at best it would have started looking something like Squeak or Pharo by
>>> now.  It's a lot of effort over the long term to *keep* it small and
>>> focused.  THANK YOU JUAN!)
>> Thanks for your nice words, Phil. I really appreciate that you all
>> accept this rather restrictive dev model for the base image, in contrast
>> with a more open trunk style. Hopefully, as the base image becomes
>> stable and almost irrelevant, this will be each time less of a problem.
>>
>>>> So, I suggest running your tests when updating your image, or migrating
>>>> your code to an updated image.
>>> +1 to tests as they do help quite a bit and are a very good practice.
>>> Just be judicious in what/where you test (I personally don't subscribe
>>> to the TDD approach where you spend your days mostly writing tests.
>>> That's just admitting defeat in that you spend all day writing
>>> executable documentation.)
>> <OT>
>>
>> I don't subscribe to "extreme" TDD either. To me design is a creative
>> activity, both technical and artistic. It should be "driven" by
>> creativity and intelligence, not by a process. Any xDD process is
>> admitting defeat on design itself!
>>
> Heh... I like the way you put that.
>
>> </OT>
>>
>>>> In any case, I think this doesn't answer your concerns. What you need is
>>>> some way to know which updates could affect you, to review them in
>>>> detail, and understanding their effect on your code. A list of the
>>>> affected classes and/or methods for each update makes no sense. Cuis can
>>>> already show you that very easily. Perhaps each update should include:
>>>>
>>>> 1) parts of the system that are affected:
>>>> - Kernel
>>>> - Compiler
>>>> - Tools
>>>> - Morphic
>>>> - etc
>>>>
>>>> 2) Level of risk involved
>>>> - very unlikely to break code that depends on this part of the system
>>>> - could perhaps break code that depends on this part of the system
>>>> - will most likely break code that depends on this part of the system
>>>>
>>>> Or something like that.
>>>>
>>>> BTW, this is a very relevant topic for discussing on the mail list. Feel
>>>> free to answer there if you agree.
>>> One final thought on this:  I suspect a major cause of people abandoning
>>> Cuis (and Smalltalk in general), behind the 'it's too different from
>>> what I'm used to'/'not mainstream' reaction, is the instability of its
>>> APIs.
>> Well, maybe. But the alternative (the whole Python 2.8 vs. 3.0 problem)
>> is a PITA too...
> True, neither extreme (changing nothing vs. changing everything) works
> very well.<cough>Perl 6</cough>  :-)
>
>> As a bottom line, let me repeat: This is too big for me to do it.
>> Please, folks, start documenting the APIs you most care about.
>>
> Ready to go... we just need to agree on how to do it (pragma/method
> call/method category/method name?)  Also, this is most definitely
> related to the 'Canonical test cases?' thread from a few months ago as
> these types of test cases / docs are part of the backfill I was
> referring to.

I think pragmas are the best options. Also, "API existence tests" should 
ask for those APIs to be declared as public.

> As I've been writing this it occurs to me that there's probably a pretty
> easy way to go about getting started on this: IIRC, there are ~1,500
> distinct method names in the core image.  If I do a count of how many
> times my code is calling any of those methods, that should pop out what,
> for my code, is the low hanging fruit.  The long-tail of say 1-5 calls
> for the lesser used methods are the tedious work that will take more
> effort for me to sort through.  We could use my 'short list' to at least
> provide a starting point and using these most frequently called methods
> you could scan through the list and could say 'hey, why are you calling
> X instead of Y?' or maybe you'd get some visibility in the sense of
> 'wow, I didn't think that was being used'.  Hmm... I've got some
> analysis to do...

And some automated scripts to generate thos pragmas to write :)

Cheers,
Juan Vuletich