[Cuis] ContentPack
H. Hirzel
hannes.hirzel at gmail.com
Sat Feb 2 13:17:51 CST 2013
Hello all
I published a rewrite of ContentPack [1] on
https://github.com/hhzl/ContentPack
It includes tests and documentation. It can deal with larger instances
of class Form and ColorForm. It splits them so that they are stored in
several compiled methods.
Included is an example of a part of an ABC book, 4MB compressed, 8MB expanded.
My conclusion at the moment
Load time is quite long (>5 min), so a bit of patience is needed. The
interesting fact is that in this test case Cuis can deal with a single
class with 8MB source code; on the other side the speed is not good.
In the future we need to save the content in compressed format and
probably not all in the same class. However at the moment it is useful
for me as-is to move content around and therefore I publish it so that
I can focus on something else next.
The ContentPack idea can be applied in other areas as well.
Comments and code reviews are invited.
Kind regards
Hannes Hirzel
[1]
From:
http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html
ContentPack
The idea is to let collaborators of art resources (graphics and sound
designers) use whatever tool they prefer. For them, the resources are
PNG or WAV. But we need to get them in source code, to load them as
updates. ContentPack takes those external files and turns them into
code (that creates live instances). Then the change set is included in
the update stream. As we have the live objects, a later updates
removes all that code. And we are done. Later, if we want to do
another run of edition with external tools, ContentPack lets us export
the resources as files, so our artist updates them. Then the process
is repeated.
The following is copied from the release notes of Cuis 3.3:
ContentPack - A clean solution for a problem Squeak had for over a
decade! (by Casey Ransberger)
Manages internal/external resources
Allows import / export to enable use of use existing artifacts and
external tools
Does not depend on external files for image update
Updates done with code (enabling change sets of Monticello packages)
Avoids cruft accumulation, code for resources is removed after update
All these properties are important, and ContentPack solves the issue
really well.
On 1/18/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> Hello Juan and Casey
>
> Thank you Juan, for the elaboration with the added class comment.
> When looking at updates 966, 967 and 968 I realized that I actually
> have to subclass it to use it.
>
>
> The fact that ContentPack takes care of the conversions between
>
> a) Live objects (as of now instances of Form and ColorForm in
> dictionaries of dictionaries)
> b) their representation as Smalltalk methods in a single ContentPack
> subclass
> c) the storage on the disk (as *.png and *.bmp)
>
> makes it very valuable for constructing learning and other games as
> Casey points out. Actually it is a need.
>
> There is actually little code, but as it stands now it is difficult to
> understand because of the naming used and missing convenience methods.
>
> I have started on a rewrite of the class with factoring methods and
> more comments added.
>
> I will present the result soon for review.
>
>
> Kind regards
>
> Hannes Hirzel
>
>
> On 1/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>> Hannes, that's great. JPG support would be awesome to have; eventually,
>> in
>> some wonderful someday, it'd be cool to add support for other media types
>> like mpeg and mp3, it's just a matter of having a way to bring
>> disk-representations in, and some means to reduce them to base64 or some
>> other convenient textual encoding (and also possibly the need to adjust a
>> limitation to the size of a change set, IIRC.)
>>
>> One of the things I'd like to do with Cuis is games, and so multimedia
>> support is something I care about. It's just a matter of finding the
>> time.
>> I will try to work on the documentation part soon.
>>
>> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <hannes.hirzel at gmail.com>
>> wrote:
>>
>>> Thank you Juan, for giving more details. A follow up question
>>> regarding jpg support below....
>>>
>>> --Hannes
>>>
>>> On 1/3/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>> > Hi Hannes,
>>> >
>>> > H. Hirzel wrote:
>>> >> In the meantime I found out that
>>> >>
>>> >>
>>> >> ContentPack seems to be a persistence mechanism for a dictionary
>>> >> which
>>> >> may contain other dictionaries and instances of Form and ColorForm.
>>> >>
>>> >> Forms are written out to the file system as *.png files and
>>> >> ColorForms
>>> >> are written as *.bmp files.
>>> >>
>>> >> Is this correct?
>>> >>
>>> >> --Hannes
>>> >>
>>> >>
>>> >
>>> > Yes it is correct. It is a bit more than that, though. Forms (and
>>> > potentially other media types) can exist in three forms:
>>> > 1) As external files, such as jpg, png, etc. This is the
>>> > representation
>>> > we need to use external tools (such as image processing apps, cameras,
>>> > scanners, web, etc) to work on them.
>>>
>>> Speaking of cameras. I'd like to include jpg files. However I do not
>>> see a support of for them. On the class side of ContentPack we have
>>> the method
>>>
>>>
>>>
>>> mapping
>>>
>>> ^ {
>>> ColorForm -> #bmp .
>>> Form -> #png
>>> }
>>>
>>>
>>> > 2) As methods. Non human readable, base-64 encoded binary data. We
>>> > need
>>> > this to be able to include such stuff in the update stream, or in
>>> > packages. After we update an image, we usually delete these methods,
>>> > just keeping 3).
>>> > 3) Live objects in the image, for example, stored in class variables.
>>> > This is to make use of them in Cuis.
>>> >
>>> > Most of the time, we use 3). But we need 2) for the update stream. We
>>> > also need 1) sometimes to work on them. ContentPack supports the
>>> > conversion between these 3 formats. The implementation is quite
>>> > simple.
>>> > What is really great is that Casey realized we need some tool to move
>>> > comfortably between these 3 representations. And he also implemented
>>> > it.
>>> >
>>> > Please grab http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zip and
>>> > take a look at updates 966, 967 and 968.
>>> >
>>> > Maybe it is time for a bit more documentation, and usage examples...
>>> >
>>> > Cheers,
>>> > Juan Vuletich
>>> >
>>> >> On 1/2/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>>> >>
>>> >>> Hello Juan and Casey
>>> >>>
>>> >>> Is the ContentPack something like Fuel?
>>> >>> http://rmod.lille.inria.fr/web/pier/software/Fuel
>>> >>> (BTW it is now available for Squeak as well, see announcement on the
>>> >>> Squeak list)?
>>> >>>
>>> >>>
>>> >>>
>>> >>> I found class 'ContentPack', I copy it in below. The nice thing is
>>> >>> that it only has 11 instance methods and 7 class methods. However
>>> >>> most
>>> >>> of the comment I do not understand.
>>> >>>
>>> >>> I copy in the class comment below.
>>> >>> I put in comments in uppercase.
>>> >>>
>>> >>> Kind regards
>>> >>> Hannes
>>> >>>
>>> >>>
>>> >>>
>>> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>> >>> ContentPack lets you read in and write out the (supported files in
>>> >>> the) contents of a directory on your file system. It also allows you
>>> >>> to trivially create "messenger" subclasses that capture the
>>> >>> information containted (TYPO) in these directory trees, including
>>> >>> any
>>> >>> implicit communication that's there in the structure of the
>>> >>> directory
>>> >>> hierarchy itself, which are captured in your changes file.
>>> >>>
>>> >>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported file
>>> >>> types?
>>> >>>
>>> >>>
>>> >>> You can then file out a change set that contains a representation of
>>> >>> the (supported file/object types and directory structurein) TYPO the
>>> >>> stuff on your disk, or in your image. This subclass is a dummy which
>>> >>> ContentPack compiles methods into containing base 64 encoded data.
>>> >>> You
>>> >>> can load this into another image, as long as that image has
>>> >>> ContentPack loaded. The filed in class can then recreate the
>>> >>> ContentPack on the other end with the media files and structure
>>> >>> intact.
>>> >>>
>>> >>> The current implementation is based on #storeString, but the plan is
>>> >>> to change that to SmartRefStream in the long run to support
>>> >>> serializing things like morphs.
>>> >>>
>>> >>> ContentPack instances hang onto the actual tree of media objects.(I
>>> >>> DO
>>> >>> NOT UNDERSTAND) It has a nice simple EDSL ???? that just interprets
>>> >>> an array of strings from beginning to end as a "path" to a file
>>> >>> (really a series of dictionary lookups to a Smalltalk object, wherin
>>> >>> the dictionaries mirror the structure of what was on the disk, sans
>>> >>> unsupported files.) This mechanism will likely change a little bit
>>> >>> at
>>> >>> some point,
>>> >>>
>>> >>>
>>> >>> ContentPack came into the world a little faster than I expected, as
>>> >>> I
>>> >>> ended up using it to send some icons back in time to fix the Cuis
>>> >>> update stream without having to sort my changes all over again. As
>>> >>> such it had some unusual design pressures... it had to be able to
>>> >>> carry information in and out of both the change set stream and the
>>> >>> filesystem, as well as function in a slightly earlier (unreleased)
>>> >>> version of Cuis than it was written in, and not break anything on
>>> >>> it's
>>> >>> way back up through the build to head.
>>> >>>
>>> >>> SENDING ICONS BACK IN TIME????
>>> >>>
>>> >>>
>>> >>> The code, in particular the way things are named, has not settled
>>> >>> yet,
>>> >>> and that's why this comment contains no code examples. Use with care
>>> >>> and read the code first, for now.
>>> >>>
>>> >>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT WOULD
>>> >>> NOT
>>> >>> BE IN CORE.
>>> >>>
>>> >>>
>>> >>> Currently, .bmp import and .png import are implemented, and both can
>>> >>> be exported. Anything you can import, you can also shuffle into a
>>> >>> change set. Plans are in the works to support audio, change sets,
>>> >>> and
>>> >>> text files.
>>> >>>
>>> >>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
>>> >>> THIS
>>> >>> SHOULD NOT BE A LARGE EFFORT.
>>> >>>
>>> >>> I'll support video if someone has a good importer, exporter, and
>>> >>> player under the MIT license that'll work under Cuis.
>>> >>>
>>> >>> Currently, objects are serialized into single methods, which works
>>> >>> for
>>> >>> small icons, but likely doesn't work well (if at all) for larger
>>> >>> files. My intent is to add some behavior that breaks up large
>>> >>> objects
>>> >>> into smaller chunks so that this becomes a non-issue. I'll likely
>>> >>> get
>>> >>> to that when I've removed most of the repetitive subtle variations
>>> >>> of
>>> >>> the same recursive tree walking visitor-trick from the code, and
>>> >>> renamed everything. I think in essence this class is slightly
>>> >>> smaller
>>> >>> than it is as represented currently.
>>> >>>
>>> >>> Hopefully I will be able to explain all of this better once I've
>>> >>> clarified the code a bit so that I can show off some examples.
>>> >>>
>>> >>> YES. EXAMPLES WILL HELP A LOT :-)
>>> >>>
>>> >>> - cbr
>>> >>>
>>> >>> On 1/2/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>> >>>
>>> >>>> Hi Folks,
>>> >>>>
>>> >>>> I think it should work ok. I don't recall doing any changes that
>>> >>>> would
>>> >>>> obviously affect it.
>>> >>>>
>>> >>>> BTW, a bit more of documentation wouldn't hurt, but the code is all
>>> >>>> there, and there's a reasonable class comment. It is just a matter
>>> >>>> of
>>> >>>> learning and playing a bit with it.
>>> >>>>
>>> >>>> Cheers,
>>> >>>> Juan Vuletich
>>> >>>>
>>> >>>> Casey Ransberger wrote:
>>> >>>>
>>> >>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
>>> >>>>> expected, after I'd used it to muck with the time stream, that
>>> >>>>> we'd
>>> >>>>> throw it away once the paradox was resolved, but Juan liked it,
>>> >>>>> and
>>> >>>>> wanted to keep it.
>>> >>>>>
>>> >>>>> Anyway, it's my dog and I've been terrible about feeding it. I
>>> >>>>> should
>>> >>>>> fix that.
>>> >>>>>
>>> >>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <hannes.hirzel at gmail.com
>>> >>>>> <mailto:hannes.hirzel at gmail.com>> wrote:
>>> >>>>>
>>> >>>>> Hello Casey and Juan
>>> >>>>>
>>> >>>>> Good to see you active on this list.
>>> >>>>> How to I try out the ContentPack in Cuis 4.1?
>>> >>>>>
>>> >>>>> Casey mentions in another thread that it might not work
>>> >>>>> anymore.
>>> >>>>>
>>> >>>>> --Hannes
>>> >>>>>
>>> >>>>> On 12/30/12, Juan Vuletich <juan at jvuletich.org
>>> >>>>> <mailto:juan at jvuletich.org>> wrote:
>>> >>>>> > Hi Hannes,
>>> >>>>> >
>>> >>>>> > You might be thinking on Casey's ContentPack, that is part
>>> >>>>> of
>>> >>>>> Cuis. It
>>> >>>>> > allows us to use only change sets for updating Cuis, while
>>> >>>>> at
>>> >>>>> the same
>>> >>>>> > time, using external tools for editing resources (like .bmp,
>>> >>>>> .png and
>>> >>>>> > jpg files, etc).
>>> >>>>> >
>>> >>>>> > Cheers,
>>> >>>>> > Juan Vuletich
>>> >>>>> >
>>> >>>>> > H. Hirzel wrote:
>>> >>>>> .....
>>> >>>>> >>
>>> >>>>> >> If I remember well you once did a package for managing
>>> >>>>> resources, right?
>>> >>>>> >>
>>> >>>>> >> Where is it?
>>> >>>>> >>
>>> >>>>> >> --Hannes
>>> >>>>> >>
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> Cuis mailing list
>>> >>>>> Cuis at jvuletich.org <mailto:Cuis at jvuletich.org>
>>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Casey Ransberger
>>> >>>>>
>>> ------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> Cuis mailing list
>>> >>>>> Cuis at jvuletich.org
>>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>>
>>> >>>>>
>>> >>>> _______________________________________________
>>> >>>> Cuis mailing list
>>> >>>> Cuis at jvuletich.org
>>> >>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>
>>> >>>>
>>> >>
>>> >> _______________________________________________
>>> >> Cuis mailing list
>>> >> Cuis at jvuletich.org
>>> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Cuis mailing list
>>> > Cuis at jvuletich.org
>>> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >
>>>
>>> _______________________________________________
>>> Cuis mailing list
>>> Cuis at jvuletich.org
>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>
>>
>>
>>
>> --
>> Casey Ransberger
>>
>
More information about the Cuis
mailing list