[Cuis] ContentPack
Casey Ransberger
casey.obrien.r at gmail.com
Mon Feb 4 15:54:42 CST 2013
I haven't had a chance to look at the implementation yet, but I should be able to do that sometime tomorrow (not near a VM today) if you'd like.
On Feb 4, 2013, at 5:43 AM, "H. Hirzel" <hannes.hirzel at gmail.com> wrote:
> P.S. the ABCpictures folder is 800kB on the disk whereas the source
> code is 8MB uncompressed and 4MB in mcz format.
>
> Casey,
> Did you have a look at the implementation?
>
> --Hannes
>
> On 2/4/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>> Casey,
>>
>> I am not surprised that loading a single class with 8MB takes some
>> time. I am actually pleased that this is possible at all.
>>
>> I think there is no need for further analysis in terms of speed as the
>> next iteration step is clear:
>>
>> 'Replace the storage mechanism with #storeString for the instances of
>> Form and ColorForm with writing PNG and JPG to a RWBinaryOrTextStream
>> (=memory internal stream which uses ByteArray) and store the resulting
>> ByteArray'
>>
>>
>> Regards
>>
>> --Hannes
>>
>>
>>
>> On 2/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>>> Have you tried profiling to look for bottlenecks? Odds are there's
>>> something in particular which is making it so slow overall.
>>>
>>> On Sat, Feb 2, 2013 at 11:17 AM, H. Hirzel <hannes.hirzel at gmail.com>
>>> wrote:
>>>
>>>> Hello all
>>>>
>>>> I published a rewrite of ContentPack [1] on
>>>>
>>>> https://github.com/hhzl/ContentPack
>>>>
>>>> It includes tests and documentation. It can deal with larger instances
>>>> of class Form and ColorForm. It splits them so that they are stored in
>>>> several compiled methods.
>>>>
>>>> Included is an example of a part of an ABC book, 4MB compressed, 8MB
>>>> expanded.
>>>>
>>>> My conclusion at the moment
>>>>
>>>> Load time is quite long (>5 min), so a bit of patience is needed. The
>>>> interesting fact is that in this test case Cuis can deal with a single
>>>> class with 8MB source code; on the other side the speed is not good.
>>>>
>>>> In the future we need to save the content in compressed format and
>>>> probably not all in the same class. However at the moment it is useful
>>>> for me as-is to move content around and therefore I publish it so that
>>>> I can focus on something else next.
>>>>
>>>> The ContentPack idea can be applied in other areas as well.
>>>>
>>>> Comments and code reviews are invited.
>>>>
>>>> Kind regards
>>>>
>>>> Hannes Hirzel
>>>>
>>>>
>>>>
>>>> [1]
>>>> From:
>>>>
>>>> http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html
>>>>
>>>> ContentPack
>>>>
>>>> The idea is to let collaborators of art resources (graphics and sound
>>>> designers) use whatever tool they prefer. For them, the resources are
>>>> PNG or WAV. But we need to get them in source code, to load them as
>>>> updates. ContentPack takes those external files and turns them into
>>>> code (that creates live instances). Then the change set is included in
>>>> the update stream. As we have the live objects, a later updates
>>>> removes all that code. And we are done. Later, if we want to do
>>>> another run of edition with external tools, ContentPack lets us export
>>>> the resources as files, so our artist updates them. Then the process
>>>> is repeated.
>>>>
>>>> The following is copied from the release notes of Cuis 3.3:
>>>>
>>>> ContentPack - A clean solution for a problem Squeak had for over a
>>>> decade! (by Casey Ransberger)
>>>>
>>>> Manages internal/external resources
>>>> Allows import / export to enable use of use existing artifacts and
>>>> external tools
>>>> Does not depend on external files for image update
>>>> Updates done with code (enabling change sets of Monticello packages)
>>>> Avoids cruft accumulation, code for resources is removed after
>>>> update
>>>>
>>>> All these properties are important, and ContentPack solves the issue
>>>> really well.
>>>>
>>>> On 1/18/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>>>>> Hello Juan and Casey
>>>>>
>>>>> Thank you Juan, for the elaboration with the added class comment.
>>>>> When looking at updates 966, 967 and 968 I realized that I actually
>>>>> have to subclass it to use it.
>>>>>
>>>>>
>>>>> The fact that ContentPack takes care of the conversions between
>>>>>
>>>>> a) Live objects (as of now instances of Form and ColorForm in
>>>>> dictionaries of dictionaries)
>>>>> b) their representation as Smalltalk methods in a single ContentPack
>>>>> subclass
>>>>> c) the storage on the disk (as *.png and *.bmp)
>>>>>
>>>>> makes it very valuable for constructing learning and other games as
>>>>> Casey points out. Actually it is a need.
>>>>>
>>>>> There is actually little code, but as it stands now it is difficult to
>>>>> understand because of the naming used and missing convenience methods.
>>>>>
>>>>> I have started on a rewrite of the class with factoring methods and
>>>>> more comments added.
>>>>>
>>>>> I will present the result soon for review.
>>>>>
>>>>>
>>>>> Kind regards
>>>>>
>>>>> Hannes Hirzel
>>>>>
>>>>>
>>>>> On 1/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>>>>>> Hannes, that's great. JPG support would be awesome to have;
>>>>>> eventually,
>>>>>> in
>>>>>> some wonderful someday, it'd be cool to add support for other media
>>>> types
>>>>>> like mpeg and mp3, it's just a matter of having a way to bring
>>>>>> disk-representations in, and some means to reduce them to base64 or
>>>>>> some
>>>>>> other convenient textual encoding (and also possibly the need to
>>>>>> adjust
>>>> a
>>>>>> limitation to the size of a change set, IIRC.)
>>>>>>
>>>>>> One of the things I'd like to do with Cuis is games, and so
>>>>>> multimedia
>>>>>> support is something I care about. It's just a matter of finding the
>>>>>> time.
>>>>>> I will try to work on the documentation part soon.
>>>>>>
>>>>>> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <hannes.hirzel at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you Juan, for giving more details. A follow up question
>>>>>>> regarding jpg support below....
>>>>>>>
>>>>>>> --Hannes
>>>>>>>
>>>>>>> On 1/3/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>>>>>>> Hi Hannes,
>>>>>>>>
>>>>>>>> H. Hirzel wrote:
>>>>>>>>> In the meantime I found out that
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ContentPack seems to be a persistence mechanism for a dictionary
>>>>>>>>> which
>>>>>>>>> may contain other dictionaries and instances of Form and
>>>>>>>>> ColorForm.
>>>>>>>>>
>>>>>>>>> Forms are written out to the file system as *.png files and
>>>>>>>>> ColorForms
>>>>>>>>> are written as *.bmp files.
>>>>>>>>>
>>>>>>>>> Is this correct?
>>>>>>>>>
>>>>>>>>> --Hannes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yes it is correct. It is a bit more than that, though. Forms (and
>>>>>>>> potentially other media types) can exist in three forms:
>>>>>>>> 1) As external files, such as jpg, png, etc. This is the
>>>>>>>> representation
>>>>>>>> we need to use external tools (such as image processing apps,
>>>> cameras,
>>>>>>>> scanners, web, etc) to work on them.
>>>>>>>
>>>>>>> Speaking of cameras. I'd like to include jpg files. However I do not
>>>>>>> see a support of for them. On the class side of ContentPack we have
>>>>>>> the method
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> mapping
>>>>>>>
>>>>>>> ^ {
>>>>>>> ColorForm -> #bmp .
>>>>>>> Form -> #png
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>> 2) As methods. Non human readable, base-64 encoded binary data. We
>>>>>>>> need
>>>>>>>> this to be able to include such stuff in the update stream, or in
>>>>>>>> packages. After we update an image, we usually delete these
>>>>>>>> methods,
>>>>>>>> just keeping 3).
>>>>>>>> 3) Live objects in the image, for example, stored in class
>>>>>>>> variables.
>>>>>>>> This is to make use of them in Cuis.
>>>>>>>>
>>>>>>>> Most of the time, we use 3). But we need 2) for the update stream.
>>>>>>>> We
>>>>>>>> also need 1) sometimes to work on them. ContentPack supports the
>>>>>>>> conversion between these 3 formats. The implementation is quite
>>>>>>>> simple.
>>>>>>>> What is really great is that Casey realized we need some tool to
>>>>>>>> move
>>>>>>>> comfortably between these 3 representations. And he also
>>>>>>>> implemented
>>>>>>>> it.
>>>>>>>>
>>>>>>>> Please grab
>>>>>>>> http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zipand
>>>>>>>> take a look at updates 966, 967 and 968.
>>>>>>>>
>>>>>>>> Maybe it is time for a bit more documentation, and usage
>>>>>>>> examples...
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Juan Vuletich
>>>>>>>>
>>>>>>>>> On 1/2/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Juan and Casey
>>>>>>>>>>
>>>>>>>>>> Is the ContentPack something like Fuel?
>>>>>>>>>> http://rmod.lille.inria.fr/web/pier/software/Fuel
>>>>>>>>>> (BTW it is now available for Squeak as well, see announcement on
>>>> the
>>>>>>>>>> Squeak list)?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I found class 'ContentPack', I copy it in below. The nice thing
>>>>>>>>>> is
>>>>>>>>>> that it only has 11 instance methods and 7 class methods.
>>>>>>>>>> However
>>>>>>>>>> most
>>>>>>>>>> of the comment I do not understand.
>>>>>>>>>>
>>>>>>>>>> I copy in the class comment below.
>>>>>>>>>> I put in comments in uppercase.
>>>>>>>>>>
>>>>>>>>>> Kind regards
>>>>>>>>>> Hannes
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>>>>>>>>> ContentPack lets you read in and write out the (supported files
>>>>>>>>>> in
>>>>>>>>>> the) contents of a directory on your file system. It also allows
>>>> you
>>>>>>>>>> to trivially create "messenger" subclasses that capture the
>>>>>>>>>> information containted (TYPO) in these directory trees,
>>>>>>>>>> including
>>>>>>>>>> any
>>>>>>>>>> implicit communication that's there in the structure of the
>>>>>>>>>> directory
>>>>>>>>>> hierarchy itself, which are captured in your changes file.
>>>>>>>>>>
>>>>>>>>>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported
>>>>>>>>>> file
>>>>>>>>>> types?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You can then file out a change set that contains a
>>>>>>>>>> representation
>>>> of
>>>>>>>>>> the (supported file/object types and directory structurein) TYPO
>>>> the
>>>>>>>>>> stuff on your disk, or in your image. This subclass is a dummy
>>>> which
>>>>>>>>>> ContentPack compiles methods into containing base 64 encoded
>>>>>>>>>> data.
>>>>>>>>>> You
>>>>>>>>>> can load this into another image, as long as that image has
>>>>>>>>>> ContentPack loaded. The filed in class can then recreate the
>>>>>>>>>> ContentPack on the other end with the media files and structure
>>>>>>>>>> intact.
>>>>>>>>>>
>>>>>>>>>> The current implementation is based on #storeString, but the
>>>>>>>>>> plan
>>>> is
>>>>>>>>>> to change that to SmartRefStream in the long run to support
>>>>>>>>>> serializing things like morphs.
>>>>>>>>>>
>>>>>>>>>> ContentPack instances hang onto the actual tree of media
>>>>>>>>>> objects.(I
>>>>>>>>>> DO
>>>>>>>>>> NOT UNDERSTAND) It has a nice simple EDSL ???? that just
>>>> interprets
>>>>>>>>>> an array of strings from beginning to end as a "path" to a file
>>>>>>>>>> (really a series of dictionary lookups to a Smalltalk object,
>>>> wherin
>>>>>>>>>> the dictionaries mirror the structure of what was on the disk,
>>>>>>>>>> sans
>>>>>>>>>> unsupported files.) This mechanism will likely change a little
>>>>>>>>>> bit
>>>>>>>>>> at
>>>>>>>>>> some point,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ContentPack came into the world a little faster than I expected,
>>>>>>>>>> as
>>>>>>>>>> I
>>>>>>>>>> ended up using it to send some icons back in time to fix the
>>>>>>>>>> Cuis
>>>>>>>>>> update stream without having to sort my changes all over again.
>>>>>>>>>> As
>>>>>>>>>> such it had some unusual design pressures... it had to be able
>>>>>>>>>> to
>>>>>>>>>> carry information in and out of both the change set stream and
>>>>>>>>>> the
>>>>>>>>>> filesystem, as well as function in a slightly earlier
>>>>>>>>>> (unreleased)
>>>>>>>>>> version of Cuis than it was written in, and not break anything
>>>>>>>>>> on
>>>>>>>>>> it's
>>>>>>>>>> way back up through the build to head.
>>>>>>>>>>
>>>>>>>>>> SENDING ICONS BACK IN TIME????
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The code, in particular the way things are named, has not
>>>>>>>>>> settled
>>>>>>>>>> yet,
>>>>>>>>>> and that's why this comment contains no code examples. Use with
>>>> care
>>>>>>>>>> and read the code first, for now.
>>>>>>>>>>
>>>>>>>>>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT
>>>>>>>>>> WOULD
>>>>>>>>>> NOT
>>>>>>>>>> BE IN CORE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Currently, .bmp import and .png import are implemented, and both
>>>> can
>>>>>>>>>> be exported. Anything you can import, you can also shuffle into
>>>>>>>>>> a
>>>>>>>>>> change set. Plans are in the works to support audio, change
>>>>>>>>>> sets,
>>>>>>>>>> and
>>>>>>>>>> text files.
>>>>>>>>>>
>>>>>>>>>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
>>>>>>>>>> THIS
>>>>>>>>>> SHOULD NOT BE A LARGE EFFORT.
>>>>>>>>>>
>>>>>>>>>> I'll support video if someone has a good importer, exporter, and
>>>>>>>>>> player under the MIT license that'll work under Cuis.
>>>>>>>>>>
>>>>>>>>>> Currently, objects are serialized into single methods, which
>>>>>>>>>> works
>>>>>>>>>> for
>>>>>>>>>> small icons, but likely doesn't work well (if at all) for larger
>>>>>>>>>> files. My intent is to add some behavior that breaks up large
>>>>>>>>>> objects
>>>>>>>>>> into smaller chunks so that this becomes a non-issue. I'll
>>>>>>>>>> likely
>>>>>>>>>> get
>>>>>>>>>> to that when I've removed most of the repetitive subtle
>>>>>>>>>> variations
>>>>>>>>>> of
>>>>>>>>>> the same recursive tree walking visitor-trick from the code, and
>>>>>>>>>> renamed everything. I think in essence this class is slightly
>>>>>>>>>> smaller
>>>>>>>>>> than it is as represented currently.
>>>>>>>>>>
>>>>>>>>>> Hopefully I will be able to explain all of this better once I've
>>>>>>>>>> clarified the code a bit so that I can show off some examples.
>>>>>>>>>>
>>>>>>>>>> YES. EXAMPLES WILL HELP A LOT :-)
>>>>>>>>>>
>>>>>>>>>> - cbr
>>>>>>>>>>
>>>>>>>>>> On 1/2/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Folks,
>>>>>>>>>>>
>>>>>>>>>>> I think it should work ok. I don't recall doing any changes
>>>>>>>>>>> that
>>>>>>>>>>> would
>>>>>>>>>>> obviously affect it.
>>>>>>>>>>>
>>>>>>>>>>> BTW, a bit more of documentation wouldn't hurt, but the code is
>>>> all
>>>>>>>>>>> there, and there's a reasonable class comment. It is just a
>>>>>>>>>>> matter
>>>>>>>>>>> of
>>>>>>>>>>> learning and playing a bit with it.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Juan Vuletich
>>>>>>>>>>>
>>>>>>>>>>> Casey Ransberger wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
>>>>>>>>>>>> expected, after I'd used it to muck with the time stream, that
>>>>>>>>>>>> we'd
>>>>>>>>>>>> throw it away once the paradox was resolved, but Juan liked
>>>>>>>>>>>> it,
>>>>>>>>>>>> and
>>>>>>>>>>>> wanted to keep it.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, it's my dog and I've been terrible about feeding it. I
>>>>>>>>>>>> should
>>>>>>>>>>>> fix that.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <
>>>> hannes.hirzel at gmail.com
>>>>>>>>>>>> <mailto:hannes.hirzel at gmail.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Casey and Juan
>>>>>>>>>>>>
>>>>>>>>>>>> Good to see you active on this list.
>>>>>>>>>>>> How to I try out the ContentPack in Cuis 4.1?
>>>>>>>>>>>>
>>>>>>>>>>>> Casey mentions in another thread that it might not work
>>>>>>>>>>>> anymore.
>>>>>>>>>>>>
>>>>>>>>>>>> --Hannes
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/30/12, Juan Vuletich <juan at jvuletich.org
>>>>>>>>>>>> <mailto:juan at jvuletich.org>> wrote:
>>>>>>>>>>>>> Hi Hannes,
>>>>>>>>>>>>>
>>>>>>>>>>>>> You might be thinking on Casey's ContentPack, that is
>>>>>>>>>>>> part
>>>>>>>>>>>> of
>>>>>>>>>>>> Cuis. It
>>>>>>>>>>>>> allows us to use only change sets for updating Cuis,
>>>>>>>>>>>> while
>>>>>>>>>>>> at
>>>>>>>>>>>> the same
>>>>>>>>>>>>> time, using external tools for editing resources (like
>>>> .bmp,
>>>>>>>>>>>> .png and
>>>>>>>>>>>>> jpg files, etc).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Juan Vuletich
>>>>>>>>>>>>>
>>>>>>>>>>>>> H. Hirzel wrote:
>>>>>>>>>>>> .....
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If I remember well you once did a package for managing
>>>>>>>>>>>> resources, right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Where is it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --Hannes
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Cuis mailing list
>>>>>>>>>>>> Cuis at jvuletich.org <mailto:Cuis at jvuletich.org>
>>>>>>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Casey Ransberger
>>>>>>>>>>>>
>>>>>>>
>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Cuis mailing list
>>>>>>>>>>>> Cuis at jvuletich.org
>>>>>>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Cuis mailing list
>>>>>>>>>>> Cuis at jvuletich.org
>>>>>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Cuis mailing list
>>>>>>>>> Cuis at jvuletich.org
>>>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Cuis mailing list
>>>>>>>> Cuis at jvuletich.org
>>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Cuis mailing list
>>>>>>> Cuis at jvuletich.org
>>>>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Casey Ransberger
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Cuis mailing list
>>>> Cuis at jvuletich.org
>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>>
>>>
>>>
>>>
>>> --
>>> Casey Ransberger
>>>
>>
>
> _______________________________________________
> Cuis mailing list
> Cuis at jvuletich.org
> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
More information about the Cuis
mailing list