[Cuis] ContentPack
H. Hirzel
hannes.hirzel at gmail.com
Mon Feb 4 07:43:27 CST 2013
P.S. the ABCpictures folder is 800kB on the disk whereas the source
code is 8MB uncompressed and 4MB in mcz format.
Casey,
Did you have a look at the implementation?
--Hannes
On 2/4/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> Casey,
>
> I am not surprised that loading a single class with 8MB takes some
> time. I am actually pleased that this is possible at all.
>
> I think there is no need for further analysis in terms of speed as the
> next iteration step is clear:
>
> 'Replace the storage mechanism with #storeString for the instances of
> Form and ColorForm with writing PNG and JPG to a RWBinaryOrTextStream
> (=memory internal stream which uses ByteArray) and store the resulting
> ByteArray'
>
>
> Regards
>
> --Hannes
>
>
>
> On 2/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>> Have you tried profiling to look for bottlenecks? Odds are there's
>> something in particular which is making it so slow overall.
>>
>> On Sat, Feb 2, 2013 at 11:17 AM, H. Hirzel <hannes.hirzel at gmail.com>
>> wrote:
>>
>>> Hello all
>>>
>>> I published a rewrite of ContentPack [1] on
>>>
>>> https://github.com/hhzl/ContentPack
>>>
>>> It includes tests and documentation. It can deal with larger instances
>>> of class Form and ColorForm. It splits them so that they are stored in
>>> several compiled methods.
>>>
>>> Included is an example of a part of an ABC book, 4MB compressed, 8MB
>>> expanded.
>>>
>>> My conclusion at the moment
>>>
>>> Load time is quite long (>5 min), so a bit of patience is needed. The
>>> interesting fact is that in this test case Cuis can deal with a single
>>> class with 8MB source code; on the other side the speed is not good.
>>>
>>> In the future we need to save the content in compressed format and
>>> probably not all in the same class. However at the moment it is useful
>>> for me as-is to move content around and therefore I publish it so that
>>> I can focus on something else next.
>>>
>>> The ContentPack idea can be applied in other areas as well.
>>>
>>> Comments and code reviews are invited.
>>>
>>> Kind regards
>>>
>>> Hannes Hirzel
>>>
>>>
>>>
>>> [1]
>>> From:
>>>
>>> http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html
>>>
>>> ContentPack
>>>
>>> The idea is to let collaborators of art resources (graphics and sound
>>> designers) use whatever tool they prefer. For them, the resources are
>>> PNG or WAV. But we need to get them in source code, to load them as
>>> updates. ContentPack takes those external files and turns them into
>>> code (that creates live instances). Then the change set is included in
>>> the update stream. As we have the live objects, a later updates
>>> removes all that code. And we are done. Later, if we want to do
>>> another run of edition with external tools, ContentPack lets us export
>>> the resources as files, so our artist updates them. Then the process
>>> is repeated.
>>>
>>> The following is copied from the release notes of Cuis 3.3:
>>>
>>> ContentPack - A clean solution for a problem Squeak had for over a
>>> decade! (by Casey Ransberger)
>>>
>>> Manages internal/external resources
>>> Allows import / export to enable use of use existing artifacts and
>>> external tools
>>> Does not depend on external files for image update
>>> Updates done with code (enabling change sets of Monticello packages)
>>> Avoids cruft accumulation, code for resources is removed after
>>> update
>>>
>>> All these properties are important, and ContentPack solves the issue
>>> really well.
>>>
>>> On 1/18/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>>> > Hello Juan and Casey
>>> >
>>> > Thank you Juan, for the elaboration with the added class comment.
>>> > When looking at updates 966, 967 and 968 I realized that I actually
>>> > have to subclass it to use it.
>>> >
>>> >
>>> > The fact that ContentPack takes care of the conversions between
>>> >
>>> > a) Live objects (as of now instances of Form and ColorForm in
>>> > dictionaries of dictionaries)
>>> > b) their representation as Smalltalk methods in a single ContentPack
>>> > subclass
>>> > c) the storage on the disk (as *.png and *.bmp)
>>> >
>>> > makes it very valuable for constructing learning and other games as
>>> > Casey points out. Actually it is a need.
>>> >
>>> > There is actually little code, but as it stands now it is difficult to
>>> > understand because of the naming used and missing convenience methods.
>>> >
>>> > I have started on a rewrite of the class with factoring methods and
>>> > more comments added.
>>> >
>>> > I will present the result soon for review.
>>> >
>>> >
>>> > Kind regards
>>> >
>>> > Hannes Hirzel
>>> >
>>> >
>>> > On 1/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>>> >> Hannes, that's great. JPG support would be awesome to have;
>>> >> eventually,
>>> >> in
>>> >> some wonderful someday, it'd be cool to add support for other media
>>> types
>>> >> like mpeg and mp3, it's just a matter of having a way to bring
>>> >> disk-representations in, and some means to reduce them to base64 or
>>> >> some
>>> >> other convenient textual encoding (and also possibly the need to
>>> >> adjust
>>> a
>>> >> limitation to the size of a change set, IIRC.)
>>> >>
>>> >> One of the things I'd like to do with Cuis is games, and so
>>> >> multimedia
>>> >> support is something I care about. It's just a matter of finding the
>>> >> time.
>>> >> I will try to work on the documentation part soon.
>>> >>
>>> >> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <hannes.hirzel at gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Thank you Juan, for giving more details. A follow up question
>>> >>> regarding jpg support below....
>>> >>>
>>> >>> --Hannes
>>> >>>
>>> >>> On 1/3/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>> >>> > Hi Hannes,
>>> >>> >
>>> >>> > H. Hirzel wrote:
>>> >>> >> In the meantime I found out that
>>> >>> >>
>>> >>> >>
>>> >>> >> ContentPack seems to be a persistence mechanism for a dictionary
>>> >>> >> which
>>> >>> >> may contain other dictionaries and instances of Form and
>>> >>> >> ColorForm.
>>> >>> >>
>>> >>> >> Forms are written out to the file system as *.png files and
>>> >>> >> ColorForms
>>> >>> >> are written as *.bmp files.
>>> >>> >>
>>> >>> >> Is this correct?
>>> >>> >>
>>> >>> >> --Hannes
>>> >>> >>
>>> >>> >>
>>> >>> >
>>> >>> > Yes it is correct. It is a bit more than that, though. Forms (and
>>> >>> > potentially other media types) can exist in three forms:
>>> >>> > 1) As external files, such as jpg, png, etc. This is the
>>> >>> > representation
>>> >>> > we need to use external tools (such as image processing apps,
>>> cameras,
>>> >>> > scanners, web, etc) to work on them.
>>> >>>
>>> >>> Speaking of cameras. I'd like to include jpg files. However I do not
>>> >>> see a support of for them. On the class side of ContentPack we have
>>> >>> the method
>>> >>>
>>> >>>
>>> >>>
>>> >>> mapping
>>> >>>
>>> >>> ^ {
>>> >>> ColorForm -> #bmp .
>>> >>> Form -> #png
>>> >>> }
>>> >>>
>>> >>>
>>> >>> > 2) As methods. Non human readable, base-64 encoded binary data. We
>>> >>> > need
>>> >>> > this to be able to include such stuff in the update stream, or in
>>> >>> > packages. After we update an image, we usually delete these
>>> >>> > methods,
>>> >>> > just keeping 3).
>>> >>> > 3) Live objects in the image, for example, stored in class
>>> >>> > variables.
>>> >>> > This is to make use of them in Cuis.
>>> >>> >
>>> >>> > Most of the time, we use 3). But we need 2) for the update stream.
>>> >>> > We
>>> >>> > also need 1) sometimes to work on them. ContentPack supports the
>>> >>> > conversion between these 3 formats. The implementation is quite
>>> >>> > simple.
>>> >>> > What is really great is that Casey realized we need some tool to
>>> >>> > move
>>> >>> > comfortably between these 3 representations. And he also
>>> >>> > implemented
>>> >>> > it.
>>> >>> >
>>> >>> > Please grab
>>> >>> > http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zipand
>>> >>> > take a look at updates 966, 967 and 968.
>>> >>> >
>>> >>> > Maybe it is time for a bit more documentation, and usage
>>> >>> > examples...
>>> >>> >
>>> >>> > Cheers,
>>> >>> > Juan Vuletich
>>> >>> >
>>> >>> >> On 1/2/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>>> >>> >>
>>> >>> >>> Hello Juan and Casey
>>> >>> >>>
>>> >>> >>> Is the ContentPack something like Fuel?
>>> >>> >>> http://rmod.lille.inria.fr/web/pier/software/Fuel
>>> >>> >>> (BTW it is now available for Squeak as well, see announcement on
>>> the
>>> >>> >>> Squeak list)?
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> I found class 'ContentPack', I copy it in below. The nice thing
>>> >>> >>> is
>>> >>> >>> that it only has 11 instance methods and 7 class methods.
>>> >>> >>> However
>>> >>> >>> most
>>> >>> >>> of the comment I do not understand.
>>> >>> >>>
>>> >>> >>> I copy in the class comment below.
>>> >>> >>> I put in comments in uppercase.
>>> >>> >>>
>>> >>> >>> Kind regards
>>> >>> >>> Hannes
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>>
>>> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>> >>> >>> ContentPack lets you read in and write out the (supported files
>>> >>> >>> in
>>> >>> >>> the) contents of a directory on your file system. It also allows
>>> you
>>> >>> >>> to trivially create "messenger" subclasses that capture the
>>> >>> >>> information containted (TYPO) in these directory trees,
>>> >>> >>> including
>>> >>> >>> any
>>> >>> >>> implicit communication that's there in the structure of the
>>> >>> >>> directory
>>> >>> >>> hierarchy itself, which are captured in your changes file.
>>> >>> >>>
>>> >>> >>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported
>>> >>> >>> file
>>> >>> >>> types?
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> You can then file out a change set that contains a
>>> >>> >>> representation
>>> of
>>> >>> >>> the (supported file/object types and directory structurein) TYPO
>>> the
>>> >>> >>> stuff on your disk, or in your image. This subclass is a dummy
>>> which
>>> >>> >>> ContentPack compiles methods into containing base 64 encoded
>>> >>> >>> data.
>>> >>> >>> You
>>> >>> >>> can load this into another image, as long as that image has
>>> >>> >>> ContentPack loaded. The filed in class can then recreate the
>>> >>> >>> ContentPack on the other end with the media files and structure
>>> >>> >>> intact.
>>> >>> >>>
>>> >>> >>> The current implementation is based on #storeString, but the
>>> >>> >>> plan
>>> is
>>> >>> >>> to change that to SmartRefStream in the long run to support
>>> >>> >>> serializing things like morphs.
>>> >>> >>>
>>> >>> >>> ContentPack instances hang onto the actual tree of media
>>> >>> >>> objects.(I
>>> >>> >>> DO
>>> >>> >>> NOT UNDERSTAND) It has a nice simple EDSL ???? that just
>>> interprets
>>> >>> >>> an array of strings from beginning to end as a "path" to a file
>>> >>> >>> (really a series of dictionary lookups to a Smalltalk object,
>>> wherin
>>> >>> >>> the dictionaries mirror the structure of what was on the disk,
>>> >>> >>> sans
>>> >>> >>> unsupported files.) This mechanism will likely change a little
>>> >>> >>> bit
>>> >>> >>> at
>>> >>> >>> some point,
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> ContentPack came into the world a little faster than I expected,
>>> >>> >>> as
>>> >>> >>> I
>>> >>> >>> ended up using it to send some icons back in time to fix the
>>> >>> >>> Cuis
>>> >>> >>> update stream without having to sort my changes all over again.
>>> >>> >>> As
>>> >>> >>> such it had some unusual design pressures... it had to be able
>>> >>> >>> to
>>> >>> >>> carry information in and out of both the change set stream and
>>> >>> >>> the
>>> >>> >>> filesystem, as well as function in a slightly earlier
>>> >>> >>> (unreleased)
>>> >>> >>> version of Cuis than it was written in, and not break anything
>>> >>> >>> on
>>> >>> >>> it's
>>> >>> >>> way back up through the build to head.
>>> >>> >>>
>>> >>> >>> SENDING ICONS BACK IN TIME????
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> The code, in particular the way things are named, has not
>>> >>> >>> settled
>>> >>> >>> yet,
>>> >>> >>> and that's why this comment contains no code examples. Use with
>>> care
>>> >>> >>> and read the code first, for now.
>>> >>> >>>
>>> >>> >>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT
>>> >>> >>> WOULD
>>> >>> >>> NOT
>>> >>> >>> BE IN CORE.
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> Currently, .bmp import and .png import are implemented, and both
>>> can
>>> >>> >>> be exported. Anything you can import, you can also shuffle into
>>> >>> >>> a
>>> >>> >>> change set. Plans are in the works to support audio, change
>>> >>> >>> sets,
>>> >>> >>> and
>>> >>> >>> text files.
>>> >>> >>>
>>> >>> >>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
>>> >>> >>> THIS
>>> >>> >>> SHOULD NOT BE A LARGE EFFORT.
>>> >>> >>>
>>> >>> >>> I'll support video if someone has a good importer, exporter, and
>>> >>> >>> player under the MIT license that'll work under Cuis.
>>> >>> >>>
>>> >>> >>> Currently, objects are serialized into single methods, which
>>> >>> >>> works
>>> >>> >>> for
>>> >>> >>> small icons, but likely doesn't work well (if at all) for larger
>>> >>> >>> files. My intent is to add some behavior that breaks up large
>>> >>> >>> objects
>>> >>> >>> into smaller chunks so that this becomes a non-issue. I'll
>>> >>> >>> likely
>>> >>> >>> get
>>> >>> >>> to that when I've removed most of the repetitive subtle
>>> >>> >>> variations
>>> >>> >>> of
>>> >>> >>> the same recursive tree walking visitor-trick from the code, and
>>> >>> >>> renamed everything. I think in essence this class is slightly
>>> >>> >>> smaller
>>> >>> >>> than it is as represented currently.
>>> >>> >>>
>>> >>> >>> Hopefully I will be able to explain all of this better once I've
>>> >>> >>> clarified the code a bit so that I can show off some examples.
>>> >>> >>>
>>> >>> >>> YES. EXAMPLES WILL HELP A LOT :-)
>>> >>> >>>
>>> >>> >>> - cbr
>>> >>> >>>
>>> >>> >>> On 1/2/13, Juan Vuletich <juan at jvuletich.org> wrote:
>>> >>> >>>
>>> >>> >>>> Hi Folks,
>>> >>> >>>>
>>> >>> >>>> I think it should work ok. I don't recall doing any changes
>>> >>> >>>> that
>>> >>> >>>> would
>>> >>> >>>> obviously affect it.
>>> >>> >>>>
>>> >>> >>>> BTW, a bit more of documentation wouldn't hurt, but the code is
>>> all
>>> >>> >>>> there, and there's a reasonable class comment. It is just a
>>> >>> >>>> matter
>>> >>> >>>> of
>>> >>> >>>> learning and playing a bit with it.
>>> >>> >>>>
>>> >>> >>>> Cheers,
>>> >>> >>>> Juan Vuletich
>>> >>> >>>>
>>> >>> >>>> Casey Ransberger wrote:
>>> >>> >>>>
>>> >>> >>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
>>> >>> >>>>> expected, after I'd used it to muck with the time stream, that
>>> >>> >>>>> we'd
>>> >>> >>>>> throw it away once the paradox was resolved, but Juan liked
>>> >>> >>>>> it,
>>> >>> >>>>> and
>>> >>> >>>>> wanted to keep it.
>>> >>> >>>>>
>>> >>> >>>>> Anyway, it's my dog and I've been terrible about feeding it. I
>>> >>> >>>>> should
>>> >>> >>>>> fix that.
>>> >>> >>>>>
>>> >>> >>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <
>>> hannes.hirzel at gmail.com
>>> >>> >>>>> <mailto:hannes.hirzel at gmail.com>> wrote:
>>> >>> >>>>>
>>> >>> >>>>> Hello Casey and Juan
>>> >>> >>>>>
>>> >>> >>>>> Good to see you active on this list.
>>> >>> >>>>> How to I try out the ContentPack in Cuis 4.1?
>>> >>> >>>>>
>>> >>> >>>>> Casey mentions in another thread that it might not work
>>> >>> >>>>> anymore.
>>> >>> >>>>>
>>> >>> >>>>> --Hannes
>>> >>> >>>>>
>>> >>> >>>>> On 12/30/12, Juan Vuletich <juan at jvuletich.org
>>> >>> >>>>> <mailto:juan at jvuletich.org>> wrote:
>>> >>> >>>>> > Hi Hannes,
>>> >>> >>>>> >
>>> >>> >>>>> > You might be thinking on Casey's ContentPack, that is
>>> >>> >>>>> part
>>> >>> >>>>> of
>>> >>> >>>>> Cuis. It
>>> >>> >>>>> > allows us to use only change sets for updating Cuis,
>>> >>> >>>>> while
>>> >>> >>>>> at
>>> >>> >>>>> the same
>>> >>> >>>>> > time, using external tools for editing resources (like
>>> .bmp,
>>> >>> >>>>> .png and
>>> >>> >>>>> > jpg files, etc).
>>> >>> >>>>> >
>>> >>> >>>>> > Cheers,
>>> >>> >>>>> > Juan Vuletich
>>> >>> >>>>> >
>>> >>> >>>>> > H. Hirzel wrote:
>>> >>> >>>>> .....
>>> >>> >>>>> >>
>>> >>> >>>>> >> If I remember well you once did a package for managing
>>> >>> >>>>> resources, right?
>>> >>> >>>>> >>
>>> >>> >>>>> >> Where is it?
>>> >>> >>>>> >>
>>> >>> >>>>> >> --Hannes
>>> >>> >>>>> >>
>>> >>> >>>>>
>>> >>> >>>>> _______________________________________________
>>> >>> >>>>> Cuis mailing list
>>> >>> >>>>> Cuis at jvuletich.org <mailto:Cuis at jvuletich.org>
>>> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>> --
>>> >>> >>>>> Casey Ransberger
>>> >>> >>>>>
>>> >>>
>>> ------------------------------------------------------------------------
>>> >>> >>>>>
>>> >>> >>>>> _______________________________________________
>>> >>> >>>>> Cuis mailing list
>>> >>> >>>>> Cuis at jvuletich.org
>>> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>> _______________________________________________
>>> >>> >>>> Cuis mailing list
>>> >>> >>>> Cuis at jvuletich.org
>>> >>> >>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>> >>>>
>>> >>> >>>>
>>> >>> >>
>>> >>> >> _______________________________________________
>>> >>> >> Cuis mailing list
>>> >>> >> Cuis at jvuletich.org
>>> >>> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > Cuis mailing list
>>> >>> > Cuis at jvuletich.org
>>> >>> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>> >
>>> >>>
>>> >>> _______________________________________________
>>> >>> Cuis mailing list
>>> >>> Cuis at jvuletich.org
>>> >>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Casey Ransberger
>>> >>
>>> >
>>>
>>> _______________________________________________
>>> Cuis mailing list
>>> Cuis at jvuletich.org
>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>
>>
>>
>>
>> --
>> Casey Ransberger
>>
>
More information about the Cuis
mailing list