[Cuis] ContentPack
H. Hirzel
hannes.hirzel at gmail.com
Mon Feb 4 07:35:42 CST 2013
Casey,
I am not surprised that loading a single class with 8MB takes some
time. I am actually pleased that this is possible at all.
I think there is no need for further analysis in terms of speed as the
next iteration step is clear:
'Replace the storage mechanism with #storeString for the instances of
Form and ColorForm with writing PNG and JPG to a RWBinaryOrTextStream
(=memory internal stream which uses ByteArray) and store the resulting
ByteArray'
Regards
--Hannes
On 2/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
> Have you tried profiling to look for bottlenecks? Odds are there's
> something in particular which is making it so slow overall.
>
> On Sat, Feb 2, 2013 at 11:17 AM, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>
>> Hello all
>>
>> I published a rewrite of ContentPack [1] on
>>
>> https://github.com/hhzl/ContentPack
>>
>> It includes tests and documentation. It can deal with larger instances
>> of class Form and ColorForm. It splits them so that they are stored in
>> several compiled methods.
>>
>> Included is an example of a part of an ABC book, 4MB compressed, 8MB
>> expanded.
>>
>> My conclusion at the moment
>>
>> Load time is quite long (>5 min), so a bit of patience is needed. The
>> interesting fact is that in this test case Cuis can deal with a single
>> class with 8MB source code; on the other side the speed is not good.
>>
>> In the future we need to save the content in compressed format and
>> probably not all in the same class. However at the moment it is useful
>> for me as-is to move content around and therefore I publish it so that
>> I can focus on something else next.
>>
>> The ContentPack idea can be applied in other areas as well.
>>
>> Comments and code reviews are invited.
>>
>> Kind regards
>>
>> Hannes Hirzel
>>
>>
>>
>> [1]
>> From:
>>
>> http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html
>>
>> ContentPack
>>
>> The idea is to let collaborators of art resources (graphics and sound
>> designers) use whatever tool they prefer. For them, the resources are
>> PNG or WAV. But we need to get them in source code, to load them as
>> updates. ContentPack takes those external files and turns them into
>> code (that creates live instances). Then the change set is included in
>> the update stream. As we have the live objects, a later updates
>> removes all that code. And we are done. Later, if we want to do
>> another run of edition with external tools, ContentPack lets us export
>> the resources as files, so our artist updates them. Then the process
>> is repeated.
>>
>> The following is copied from the release notes of Cuis 3.3:
>>
>> ContentPack - A clean solution for a problem Squeak had for over a
>> decade! (by Casey Ransberger)
>>
>> Manages internal/external resources
>> Allows import / export to enable use of use existing artifacts and
>> external tools
>> Does not depend on external files for image update
>> Updates done with code (enabling change sets of Monticello packages)
>> Avoids cruft accumulation, code for resources is removed after update
>>
>> All these properties are important, and ContentPack solves the issue
>> really well.
>>
>> On 1/18/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>> > Hello Juan and Casey
>> >
>> > Thank you Juan, for the elaboration with the added class comment.
>> > When looking at updates 966, 967 and 968 I realized that I actually
>> > have to subclass it to use it.
>> >
>> >
>> > The fact that ContentPack takes care of the conversions between
>> >
>> > a) Live objects (as of now instances of Form and ColorForm in
>> > dictionaries of dictionaries)
>> > b) their representation as Smalltalk methods in a single ContentPack
>> > subclass
>> > c) the storage on the disk (as *.png and *.bmp)
>> >
>> > makes it very valuable for constructing learning and other games as
>> > Casey points out. Actually it is a need.
>> >
>> > There is actually little code, but as it stands now it is difficult to
>> > understand because of the naming used and missing convenience methods.
>> >
>> > I have started on a rewrite of the class with factoring methods and
>> > more comments added.
>> >
>> > I will present the result soon for review.
>> >
>> >
>> > Kind regards
>> >
>> > Hannes Hirzel
>> >
>> >
>> > On 1/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
>> >> Hannes, that's great. JPG support would be awesome to have;
>> >> eventually,
>> >> in
>> >> some wonderful someday, it'd be cool to add support for other media
>> types
>> >> like mpeg and mp3, it's just a matter of having a way to bring
>> >> disk-representations in, and some means to reduce them to base64 or
>> >> some
>> >> other convenient textual encoding (and also possibly the need to
>> >> adjust
>> a
>> >> limitation to the size of a change set, IIRC.)
>> >>
>> >> One of the things I'd like to do with Cuis is games, and so multimedia
>> >> support is something I care about. It's just a matter of finding the
>> >> time.
>> >> I will try to work on the documentation part soon.
>> >>
>> >> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <hannes.hirzel at gmail.com>
>> >> wrote:
>> >>
>> >>> Thank you Juan, for giving more details. A follow up question
>> >>> regarding jpg support below....
>> >>>
>> >>> --Hannes
>> >>>
>> >>> On 1/3/13, Juan Vuletich <juan at jvuletich.org> wrote:
>> >>> > Hi Hannes,
>> >>> >
>> >>> > H. Hirzel wrote:
>> >>> >> In the meantime I found out that
>> >>> >>
>> >>> >>
>> >>> >> ContentPack seems to be a persistence mechanism for a dictionary
>> >>> >> which
>> >>> >> may contain other dictionaries and instances of Form and
>> >>> >> ColorForm.
>> >>> >>
>> >>> >> Forms are written out to the file system as *.png files and
>> >>> >> ColorForms
>> >>> >> are written as *.bmp files.
>> >>> >>
>> >>> >> Is this correct?
>> >>> >>
>> >>> >> --Hannes
>> >>> >>
>> >>> >>
>> >>> >
>> >>> > Yes it is correct. It is a bit more than that, though. Forms (and
>> >>> > potentially other media types) can exist in three forms:
>> >>> > 1) As external files, such as jpg, png, etc. This is the
>> >>> > representation
>> >>> > we need to use external tools (such as image processing apps,
>> cameras,
>> >>> > scanners, web, etc) to work on them.
>> >>>
>> >>> Speaking of cameras. I'd like to include jpg files. However I do not
>> >>> see a support of for them. On the class side of ContentPack we have
>> >>> the method
>> >>>
>> >>>
>> >>>
>> >>> mapping
>> >>>
>> >>> ^ {
>> >>> ColorForm -> #bmp .
>> >>> Form -> #png
>> >>> }
>> >>>
>> >>>
>> >>> > 2) As methods. Non human readable, base-64 encoded binary data. We
>> >>> > need
>> >>> > this to be able to include such stuff in the update stream, or in
>> >>> > packages. After we update an image, we usually delete these
>> >>> > methods,
>> >>> > just keeping 3).
>> >>> > 3) Live objects in the image, for example, stored in class
>> >>> > variables.
>> >>> > This is to make use of them in Cuis.
>> >>> >
>> >>> > Most of the time, we use 3). But we need 2) for the update stream.
>> >>> > We
>> >>> > also need 1) sometimes to work on them. ContentPack supports the
>> >>> > conversion between these 3 formats. The implementation is quite
>> >>> > simple.
>> >>> > What is really great is that Casey realized we need some tool to
>> >>> > move
>> >>> > comfortably between these 3 representations. And he also
>> >>> > implemented
>> >>> > it.
>> >>> >
>> >>> > Please grab
>> >>> > http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zipand
>> >>> > take a look at updates 966, 967 and 968.
>> >>> >
>> >>> > Maybe it is time for a bit more documentation, and usage
>> >>> > examples...
>> >>> >
>> >>> > Cheers,
>> >>> > Juan Vuletich
>> >>> >
>> >>> >> On 1/2/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
>> >>> >>
>> >>> >>> Hello Juan and Casey
>> >>> >>>
>> >>> >>> Is the ContentPack something like Fuel?
>> >>> >>> http://rmod.lille.inria.fr/web/pier/software/Fuel
>> >>> >>> (BTW it is now available for Squeak as well, see announcement on
>> the
>> >>> >>> Squeak list)?
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> I found class 'ContentPack', I copy it in below. The nice thing
>> >>> >>> is
>> >>> >>> that it only has 11 instance methods and 7 class methods. However
>> >>> >>> most
>> >>> >>> of the comment I do not understand.
>> >>> >>>
>> >>> >>> I copy in the class comment below.
>> >>> >>> I put in comments in uppercase.
>> >>> >>>
>> >>> >>> Kind regards
>> >>> >>> Hannes
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>>
>> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>> >>> >>> ContentPack lets you read in and write out the (supported files
>> >>> >>> in
>> >>> >>> the) contents of a directory on your file system. It also allows
>> you
>> >>> >>> to trivially create "messenger" subclasses that capture the
>> >>> >>> information containted (TYPO) in these directory trees, including
>> >>> >>> any
>> >>> >>> implicit communication that's there in the structure of the
>> >>> >>> directory
>> >>> >>> hierarchy itself, which are captured in your changes file.
>> >>> >>>
>> >>> >>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported
>> >>> >>> file
>> >>> >>> types?
>> >>> >>>
>> >>> >>>
>> >>> >>> You can then file out a change set that contains a representation
>> of
>> >>> >>> the (supported file/object types and directory structurein) TYPO
>> the
>> >>> >>> stuff on your disk, or in your image. This subclass is a dummy
>> which
>> >>> >>> ContentPack compiles methods into containing base 64 encoded
>> >>> >>> data.
>> >>> >>> You
>> >>> >>> can load this into another image, as long as that image has
>> >>> >>> ContentPack loaded. The filed in class can then recreate the
>> >>> >>> ContentPack on the other end with the media files and structure
>> >>> >>> intact.
>> >>> >>>
>> >>> >>> The current implementation is based on #storeString, but the plan
>> is
>> >>> >>> to change that to SmartRefStream in the long run to support
>> >>> >>> serializing things like morphs.
>> >>> >>>
>> >>> >>> ContentPack instances hang onto the actual tree of media
>> >>> >>> objects.(I
>> >>> >>> DO
>> >>> >>> NOT UNDERSTAND) It has a nice simple EDSL ???? that just
>> interprets
>> >>> >>> an array of strings from beginning to end as a "path" to a file
>> >>> >>> (really a series of dictionary lookups to a Smalltalk object,
>> wherin
>> >>> >>> the dictionaries mirror the structure of what was on the disk,
>> >>> >>> sans
>> >>> >>> unsupported files.) This mechanism will likely change a little
>> >>> >>> bit
>> >>> >>> at
>> >>> >>> some point,
>> >>> >>>
>> >>> >>>
>> >>> >>> ContentPack came into the world a little faster than I expected,
>> >>> >>> as
>> >>> >>> I
>> >>> >>> ended up using it to send some icons back in time to fix the Cuis
>> >>> >>> update stream without having to sort my changes all over again.
>> >>> >>> As
>> >>> >>> such it had some unusual design pressures... it had to be able to
>> >>> >>> carry information in and out of both the change set stream and
>> >>> >>> the
>> >>> >>> filesystem, as well as function in a slightly earlier
>> >>> >>> (unreleased)
>> >>> >>> version of Cuis than it was written in, and not break anything on
>> >>> >>> it's
>> >>> >>> way back up through the build to head.
>> >>> >>>
>> >>> >>> SENDING ICONS BACK IN TIME????
>> >>> >>>
>> >>> >>>
>> >>> >>> The code, in particular the way things are named, has not settled
>> >>> >>> yet,
>> >>> >>> and that's why this comment contains no code examples. Use with
>> care
>> >>> >>> and read the code first, for now.
>> >>> >>>
>> >>> >>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT WOULD
>> >>> >>> NOT
>> >>> >>> BE IN CORE.
>> >>> >>>
>> >>> >>>
>> >>> >>> Currently, .bmp import and .png import are implemented, and both
>> can
>> >>> >>> be exported. Anything you can import, you can also shuffle into a
>> >>> >>> change set. Plans are in the works to support audio, change sets,
>> >>> >>> and
>> >>> >>> text files.
>> >>> >>>
>> >>> >>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
>> >>> >>> THIS
>> >>> >>> SHOULD NOT BE A LARGE EFFORT.
>> >>> >>>
>> >>> >>> I'll support video if someone has a good importer, exporter, and
>> >>> >>> player under the MIT license that'll work under Cuis.
>> >>> >>>
>> >>> >>> Currently, objects are serialized into single methods, which
>> >>> >>> works
>> >>> >>> for
>> >>> >>> small icons, but likely doesn't work well (if at all) for larger
>> >>> >>> files. My intent is to add some behavior that breaks up large
>> >>> >>> objects
>> >>> >>> into smaller chunks so that this becomes a non-issue. I'll likely
>> >>> >>> get
>> >>> >>> to that when I've removed most of the repetitive subtle
>> >>> >>> variations
>> >>> >>> of
>> >>> >>> the same recursive tree walking visitor-trick from the code, and
>> >>> >>> renamed everything. I think in essence this class is slightly
>> >>> >>> smaller
>> >>> >>> than it is as represented currently.
>> >>> >>>
>> >>> >>> Hopefully I will be able to explain all of this better once I've
>> >>> >>> clarified the code a bit so that I can show off some examples.
>> >>> >>>
>> >>> >>> YES. EXAMPLES WILL HELP A LOT :-)
>> >>> >>>
>> >>> >>> - cbr
>> >>> >>>
>> >>> >>> On 1/2/13, Juan Vuletich <juan at jvuletich.org> wrote:
>> >>> >>>
>> >>> >>>> Hi Folks,
>> >>> >>>>
>> >>> >>>> I think it should work ok. I don't recall doing any changes that
>> >>> >>>> would
>> >>> >>>> obviously affect it.
>> >>> >>>>
>> >>> >>>> BTW, a bit more of documentation wouldn't hurt, but the code is
>> all
>> >>> >>>> there, and there's a reasonable class comment. It is just a
>> >>> >>>> matter
>> >>> >>>> of
>> >>> >>>> learning and playing a bit with it.
>> >>> >>>>
>> >>> >>>> Cheers,
>> >>> >>>> Juan Vuletich
>> >>> >>>>
>> >>> >>>> Casey Ransberger wrote:
>> >>> >>>>
>> >>> >>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
>> >>> >>>>> expected, after I'd used it to muck with the time stream, that
>> >>> >>>>> we'd
>> >>> >>>>> throw it away once the paradox was resolved, but Juan liked it,
>> >>> >>>>> and
>> >>> >>>>> wanted to keep it.
>> >>> >>>>>
>> >>> >>>>> Anyway, it's my dog and I've been terrible about feeding it. I
>> >>> >>>>> should
>> >>> >>>>> fix that.
>> >>> >>>>>
>> >>> >>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <
>> hannes.hirzel at gmail.com
>> >>> >>>>> <mailto:hannes.hirzel at gmail.com>> wrote:
>> >>> >>>>>
>> >>> >>>>> Hello Casey and Juan
>> >>> >>>>>
>> >>> >>>>> Good to see you active on this list.
>> >>> >>>>> How to I try out the ContentPack in Cuis 4.1?
>> >>> >>>>>
>> >>> >>>>> Casey mentions in another thread that it might not work
>> >>> >>>>> anymore.
>> >>> >>>>>
>> >>> >>>>> --Hannes
>> >>> >>>>>
>> >>> >>>>> On 12/30/12, Juan Vuletich <juan at jvuletich.org
>> >>> >>>>> <mailto:juan at jvuletich.org>> wrote:
>> >>> >>>>> > Hi Hannes,
>> >>> >>>>> >
>> >>> >>>>> > You might be thinking on Casey's ContentPack, that is
>> >>> >>>>> part
>> >>> >>>>> of
>> >>> >>>>> Cuis. It
>> >>> >>>>> > allows us to use only change sets for updating Cuis,
>> >>> >>>>> while
>> >>> >>>>> at
>> >>> >>>>> the same
>> >>> >>>>> > time, using external tools for editing resources (like
>> .bmp,
>> >>> >>>>> .png and
>> >>> >>>>> > jpg files, etc).
>> >>> >>>>> >
>> >>> >>>>> > Cheers,
>> >>> >>>>> > Juan Vuletich
>> >>> >>>>> >
>> >>> >>>>> > H. Hirzel wrote:
>> >>> >>>>> .....
>> >>> >>>>> >>
>> >>> >>>>> >> If I remember well you once did a package for managing
>> >>> >>>>> resources, right?
>> >>> >>>>> >>
>> >>> >>>>> >> Where is it?
>> >>> >>>>> >>
>> >>> >>>>> >> --Hannes
>> >>> >>>>> >>
>> >>> >>>>>
>> >>> >>>>> _______________________________________________
>> >>> >>>>> Cuis mailing list
>> >>> >>>>> Cuis at jvuletich.org <mailto:Cuis at jvuletich.org>
>> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> --
>> >>> >>>>> Casey Ransberger
>> >>> >>>>>
>> >>>
>> ------------------------------------------------------------------------
>> >>> >>>>>
>> >>> >>>>> _______________________________________________
>> >>> >>>>> Cuis mailing list
>> >>> >>>>> Cuis at jvuletich.org
>> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>> _______________________________________________
>> >>> >>>> Cuis mailing list
>> >>> >>>> Cuis at jvuletich.org
>> >>> >>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>> >>>>
>> >>> >>>>
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> Cuis mailing list
>> >>> >> Cuis at jvuletich.org
>> >>> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>> > _______________________________________________
>> >>> > Cuis mailing list
>> >>> > Cuis at jvuletich.org
>> >>> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>> >
>> >>>
>> >>> _______________________________________________
>> >>> Cuis mailing list
>> >>> Cuis at jvuletich.org
>> >>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Casey Ransberger
>> >>
>> >
>>
>> _______________________________________________
>> Cuis mailing list
>> Cuis at jvuletich.org
>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>
>
>
>
> --
> Casey Ransberger
>
More information about the Cuis
mailing list