[Cuis] ContentPack
Casey Ransberger
casey.obrien.r at gmail.com
Mon Feb 4 07:06:54 CST 2013
Have you tried profiling to look for bottlenecks? Odds are there's
something in particular which is making it so slow overall.
On Sat, Feb 2, 2013 at 11:17 AM, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> Hello all
>
> I published a rewrite of ContentPack [1] on
>
> https://github.com/hhzl/ContentPack
>
> It includes tests and documentation. It can deal with larger instances
> of class Form and ColorForm. It splits them so that they are stored in
> several compiled methods.
>
> Included is an example of a part of an ABC book, 4MB compressed, 8MB
> expanded.
>
> My conclusion at the moment
>
> Load time is quite long (>5 min), so a bit of patience is needed. The
> interesting fact is that in this test case Cuis can deal with a single
> class with 8MB source code; on the other side the speed is not good.
>
> In the future we need to save the content in compressed format and
> probably not all in the same class. However at the moment it is useful
> for me as-is to move content around and therefore I publish it so that
> I can focus on something else next.
>
> The ContentPack idea can be applied in other areas as well.
>
> Comments and code reviews are invited.
>
> Kind regards
>
> Hannes Hirzel
>
>
>
> [1]
> From:
>
> http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html
>
> ContentPack
>
> The idea is to let collaborators of art resources (graphics and sound
> designers) use whatever tool they prefer. For them, the resources are
> PNG or WAV. But we need to get them in source code, to load them as
> updates. ContentPack takes those external files and turns them into
> code (that creates live instances). Then the change set is included in
> the update stream. As we have the live objects, a later updates
> removes all that code. And we are done. Later, if we want to do
> another run of edition with external tools, ContentPack lets us export
> the resources as files, so our artist updates them. Then the process
> is repeated.
>
> The following is copied from the release notes of Cuis 3.3:
>
> ContentPack - A clean solution for a problem Squeak had for over a
> decade! (by Casey Ransberger)
>
> Manages internal/external resources
> Allows import / export to enable use of use existing artifacts and
> external tools
> Does not depend on external files for image update
> Updates done with code (enabling change sets of Monticello packages)
> Avoids cruft accumulation, code for resources is removed after update
>
> All these properties are important, and ContentPack solves the issue
> really well.
>
> On 1/18/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> > Hello Juan and Casey
> >
> > Thank you Juan, for the elaboration with the added class comment.
> > When looking at updates 966, 967 and 968 I realized that I actually
> > have to subclass it to use it.
> >
> >
> > The fact that ContentPack takes care of the conversions between
> >
> > a) Live objects (as of now instances of Form and ColorForm in
> > dictionaries of dictionaries)
> > b) their representation as Smalltalk methods in a single ContentPack
> > subclass
> > c) the storage on the disk (as *.png and *.bmp)
> >
> > makes it very valuable for constructing learning and other games as
> > Casey points out. Actually it is a need.
> >
> > There is actually little code, but as it stands now it is difficult to
> > understand because of the naming used and missing convenience methods.
> >
> > I have started on a rewrite of the class with factoring methods and
> > more comments added.
> >
> > I will present the result soon for review.
> >
> >
> > Kind regards
> >
> > Hannes Hirzel
> >
> >
> > On 1/4/13, Casey Ransberger <casey.obrien.r at gmail.com> wrote:
> >> Hannes, that's great. JPG support would be awesome to have; eventually,
> >> in
> >> some wonderful someday, it'd be cool to add support for other media
> types
> >> like mpeg and mp3, it's just a matter of having a way to bring
> >> disk-representations in, and some means to reduce them to base64 or some
> >> other convenient textual encoding (and also possibly the need to adjust
> a
> >> limitation to the size of a change set, IIRC.)
> >>
> >> One of the things I'd like to do with Cuis is games, and so multimedia
> >> support is something I care about. It's just a matter of finding the
> >> time.
> >> I will try to work on the documentation part soon.
> >>
> >> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <hannes.hirzel at gmail.com>
> >> wrote:
> >>
> >>> Thank you Juan, for giving more details. A follow up question
> >>> regarding jpg support below....
> >>>
> >>> --Hannes
> >>>
> >>> On 1/3/13, Juan Vuletich <juan at jvuletich.org> wrote:
> >>> > Hi Hannes,
> >>> >
> >>> > H. Hirzel wrote:
> >>> >> In the meantime I found out that
> >>> >>
> >>> >>
> >>> >> ContentPack seems to be a persistence mechanism for a dictionary
> >>> >> which
> >>> >> may contain other dictionaries and instances of Form and ColorForm.
> >>> >>
> >>> >> Forms are written out to the file system as *.png files and
> >>> >> ColorForms
> >>> >> are written as *.bmp files.
> >>> >>
> >>> >> Is this correct?
> >>> >>
> >>> >> --Hannes
> >>> >>
> >>> >>
> >>> >
> >>> > Yes it is correct. It is a bit more than that, though. Forms (and
> >>> > potentially other media types) can exist in three forms:
> >>> > 1) As external files, such as jpg, png, etc. This is the
> >>> > representation
> >>> > we need to use external tools (such as image processing apps,
> cameras,
> >>> > scanners, web, etc) to work on them.
> >>>
> >>> Speaking of cameras. I'd like to include jpg files. However I do not
> >>> see a support of for them. On the class side of ContentPack we have
> >>> the method
> >>>
> >>>
> >>>
> >>> mapping
> >>>
> >>> ^ {
> >>> ColorForm -> #bmp .
> >>> Form -> #png
> >>> }
> >>>
> >>>
> >>> > 2) As methods. Non human readable, base-64 encoded binary data. We
> >>> > need
> >>> > this to be able to include such stuff in the update stream, or in
> >>> > packages. After we update an image, we usually delete these methods,
> >>> > just keeping 3).
> >>> > 3) Live objects in the image, for example, stored in class variables.
> >>> > This is to make use of them in Cuis.
> >>> >
> >>> > Most of the time, we use 3). But we need 2) for the update stream. We
> >>> > also need 1) sometimes to work on them. ContentPack supports the
> >>> > conversion between these 3 formats. The implementation is quite
> >>> > simple.
> >>> > What is really great is that Casey realized we need some tool to move
> >>> > comfortably between these 3 representations. And he also implemented
> >>> > it.
> >>> >
> >>> > Please grab http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zipand
> >>> > take a look at updates 966, 967 and 968.
> >>> >
> >>> > Maybe it is time for a bit more documentation, and usage examples...
> >>> >
> >>> > Cheers,
> >>> > Juan Vuletich
> >>> >
> >>> >> On 1/2/13, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> >>> >>
> >>> >>> Hello Juan and Casey
> >>> >>>
> >>> >>> Is the ContentPack something like Fuel?
> >>> >>> http://rmod.lille.inria.fr/web/pier/software/Fuel
> >>> >>> (BTW it is now available for Squeak as well, see announcement on
> the
> >>> >>> Squeak list)?
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> I found class 'ContentPack', I copy it in below. The nice thing is
> >>> >>> that it only has 11 instance methods and 7 class methods. However
> >>> >>> most
> >>> >>> of the comment I do not understand.
> >>> >>>
> >>> >>> I copy in the class comment below.
> >>> >>> I put in comments in uppercase.
> >>> >>>
> >>> >>> Kind regards
> >>> >>> Hannes
> >>> >>>
> >>> >>>
> >>> >>>
> >>>
> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
> >>> >>> ContentPack lets you read in and write out the (supported files in
> >>> >>> the) contents of a directory on your file system. It also allows
> you
> >>> >>> to trivially create "messenger" subclasses that capture the
> >>> >>> information containted (TYPO) in these directory trees, including
> >>> >>> any
> >>> >>> implicit communication that's there in the structure of the
> >>> >>> directory
> >>> >>> hierarchy itself, which are captured in your changes file.
> >>> >>>
> >>> >>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported file
> >>> >>> types?
> >>> >>>
> >>> >>>
> >>> >>> You can then file out a change set that contains a representation
> of
> >>> >>> the (supported file/object types and directory structurein) TYPO
> the
> >>> >>> stuff on your disk, or in your image. This subclass is a dummy
> which
> >>> >>> ContentPack compiles methods into containing base 64 encoded data.
> >>> >>> You
> >>> >>> can load this into another image, as long as that image has
> >>> >>> ContentPack loaded. The filed in class can then recreate the
> >>> >>> ContentPack on the other end with the media files and structure
> >>> >>> intact.
> >>> >>>
> >>> >>> The current implementation is based on #storeString, but the plan
> is
> >>> >>> to change that to SmartRefStream in the long run to support
> >>> >>> serializing things like morphs.
> >>> >>>
> >>> >>> ContentPack instances hang onto the actual tree of media objects.(I
> >>> >>> DO
> >>> >>> NOT UNDERSTAND) It has a nice simple EDSL ???? that just
> interprets
> >>> >>> an array of strings from beginning to end as a "path" to a file
> >>> >>> (really a series of dictionary lookups to a Smalltalk object,
> wherin
> >>> >>> the dictionaries mirror the structure of what was on the disk, sans
> >>> >>> unsupported files.) This mechanism will likely change a little bit
> >>> >>> at
> >>> >>> some point,
> >>> >>>
> >>> >>>
> >>> >>> ContentPack came into the world a little faster than I expected, as
> >>> >>> I
> >>> >>> ended up using it to send some icons back in time to fix the Cuis
> >>> >>> update stream without having to sort my changes all over again. As
> >>> >>> such it had some unusual design pressures... it had to be able to
> >>> >>> carry information in and out of both the change set stream and the
> >>> >>> filesystem, as well as function in a slightly earlier (unreleased)
> >>> >>> version of Cuis than it was written in, and not break anything on
> >>> >>> it's
> >>> >>> way back up through the build to head.
> >>> >>>
> >>> >>> SENDING ICONS BACK IN TIME????
> >>> >>>
> >>> >>>
> >>> >>> The code, in particular the way things are named, has not settled
> >>> >>> yet,
> >>> >>> and that's why this comment contains no code examples. Use with
> care
> >>> >>> and read the code first, for now.
> >>> >>>
> >>> >>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT WOULD
> >>> >>> NOT
> >>> >>> BE IN CORE.
> >>> >>>
> >>> >>>
> >>> >>> Currently, .bmp import and .png import are implemented, and both
> can
> >>> >>> be exported. Anything you can import, you can also shuffle into a
> >>> >>> change set. Plans are in the works to support audio, change sets,
> >>> >>> and
> >>> >>> text files.
> >>> >>>
> >>> >>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
> >>> >>> THIS
> >>> >>> SHOULD NOT BE A LARGE EFFORT.
> >>> >>>
> >>> >>> I'll support video if someone has a good importer, exporter, and
> >>> >>> player under the MIT license that'll work under Cuis.
> >>> >>>
> >>> >>> Currently, objects are serialized into single methods, which works
> >>> >>> for
> >>> >>> small icons, but likely doesn't work well (if at all) for larger
> >>> >>> files. My intent is to add some behavior that breaks up large
> >>> >>> objects
> >>> >>> into smaller chunks so that this becomes a non-issue. I'll likely
> >>> >>> get
> >>> >>> to that when I've removed most of the repetitive subtle variations
> >>> >>> of
> >>> >>> the same recursive tree walking visitor-trick from the code, and
> >>> >>> renamed everything. I think in essence this class is slightly
> >>> >>> smaller
> >>> >>> than it is as represented currently.
> >>> >>>
> >>> >>> Hopefully I will be able to explain all of this better once I've
> >>> >>> clarified the code a bit so that I can show off some examples.
> >>> >>>
> >>> >>> YES. EXAMPLES WILL HELP A LOT :-)
> >>> >>>
> >>> >>> - cbr
> >>> >>>
> >>> >>> On 1/2/13, Juan Vuletich <juan at jvuletich.org> wrote:
> >>> >>>
> >>> >>>> Hi Folks,
> >>> >>>>
> >>> >>>> I think it should work ok. I don't recall doing any changes that
> >>> >>>> would
> >>> >>>> obviously affect it.
> >>> >>>>
> >>> >>>> BTW, a bit more of documentation wouldn't hurt, but the code is
> all
> >>> >>>> there, and there's a reasonable class comment. It is just a matter
> >>> >>>> of
> >>> >>>> learning and playing a bit with it.
> >>> >>>>
> >>> >>>> Cheers,
> >>> >>>> Juan Vuletich
> >>> >>>>
> >>> >>>> Casey Ransberger wrote:
> >>> >>>>
> >>> >>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
> >>> >>>>> expected, after I'd used it to muck with the time stream, that
> >>> >>>>> we'd
> >>> >>>>> throw it away once the paradox was resolved, but Juan liked it,
> >>> >>>>> and
> >>> >>>>> wanted to keep it.
> >>> >>>>>
> >>> >>>>> Anyway, it's my dog and I've been terrible about feeding it. I
> >>> >>>>> should
> >>> >>>>> fix that.
> >>> >>>>>
> >>> >>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <
> hannes.hirzel at gmail.com
> >>> >>>>> <mailto:hannes.hirzel at gmail.com>> wrote:
> >>> >>>>>
> >>> >>>>> Hello Casey and Juan
> >>> >>>>>
> >>> >>>>> Good to see you active on this list.
> >>> >>>>> How to I try out the ContentPack in Cuis 4.1?
> >>> >>>>>
> >>> >>>>> Casey mentions in another thread that it might not work
> >>> >>>>> anymore.
> >>> >>>>>
> >>> >>>>> --Hannes
> >>> >>>>>
> >>> >>>>> On 12/30/12, Juan Vuletich <juan at jvuletich.org
> >>> >>>>> <mailto:juan at jvuletich.org>> wrote:
> >>> >>>>> > Hi Hannes,
> >>> >>>>> >
> >>> >>>>> > You might be thinking on Casey's ContentPack, that is part
> >>> >>>>> of
> >>> >>>>> Cuis. It
> >>> >>>>> > allows us to use only change sets for updating Cuis, while
> >>> >>>>> at
> >>> >>>>> the same
> >>> >>>>> > time, using external tools for editing resources (like
> .bmp,
> >>> >>>>> .png and
> >>> >>>>> > jpg files, etc).
> >>> >>>>> >
> >>> >>>>> > Cheers,
> >>> >>>>> > Juan Vuletich
> >>> >>>>> >
> >>> >>>>> > H. Hirzel wrote:
> >>> >>>>> .....
> >>> >>>>> >>
> >>> >>>>> >> If I remember well you once did a package for managing
> >>> >>>>> resources, right?
> >>> >>>>> >>
> >>> >>>>> >> Where is it?
> >>> >>>>> >>
> >>> >>>>> >> --Hannes
> >>> >>>>> >>
> >>> >>>>>
> >>> >>>>> _______________________________________________
> >>> >>>>> Cuis mailing list
> >>> >>>>> Cuis at jvuletich.org <mailto:Cuis at jvuletich.org>
> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>> Casey Ransberger
> >>> >>>>>
> >>>
> ------------------------------------------------------------------------
> >>> >>>>>
> >>> >>>>> _______________________________________________
> >>> >>>>> Cuis mailing list
> >>> >>>>> Cuis at jvuletich.org
> >>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>> >>>>>
> >>> >>>>>
> >>> >>>> _______________________________________________
> >>> >>>> Cuis mailing list
> >>> >>>> Cuis at jvuletich.org
> >>> >>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>> >>>>
> >>> >>>>
> >>> >>
> >>> >> _______________________________________________
> >>> >> Cuis mailing list
> >>> >> Cuis at jvuletich.org
> >>> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Cuis mailing list
> >>> > Cuis at jvuletich.org
> >>> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>> >
> >>>
> >>> _______________________________________________
> >>> Cuis mailing list
> >>> Cuis at jvuletich.org
> >>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
> >>>
> >>
> >>
> >>
> >> --
> >> Casey Ransberger
> >>
> >
>
> _______________________________________________
> Cuis mailing list
> Cuis at jvuletich.org
> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>
--
Casey Ransberger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://jvuletich.org/pipermail/cuis_jvuletich.org/attachments/20130204/f402ff43/attachment-0002.html>
More information about the Cuis
mailing list