[Cuis] image segments

[H. Hirzel]

And this seems to cover what you want to do

How to use an ImageSegment to determine memory usage
http://wiki.squeak.org/squeak/1260



On 11/26/15, H. Hirzel <hannes.hirzel at gmail.com> wrote:
> That's interesting,
>
> see also http://wiki.squeak.org/squeak/2316 with a reference to an
> article evaluation ImageSegments at the end.
>
>
>
> On 11/26/15, Luciano Notarfrancesco <luchiano at gmail.com> wrote:
>> Cool, maybe it's possible to make it into a package, I'll take a look.
>>
>> I've never used it to store Projects, tho..  I used before to store
>> things
>> that must be loaded very fast on demand and can be forgotten once there
>> are
>> no more references to them.. like image fragments that compose a map of
>> the
>> world, and small sound fragments that are used to synthesize speech
>> (diphones). Yesterday I just wanted to know how big are my objects, and I
>> wanted to make some experiments to see how the bit-size of my objects
>> grow
>> as the problems I'm modeling grow by some other measure (degree of
>> polynomials, size of matrices are the most basic) and to know how much
>> memory are my algorithms actually using at every step... image segments
>> are
>> ideal for this, you can use them to know exactly how much memory would be
>> freed if a reference to a given object were forgotten.
>>
>>
>> On Thu, Nov 26, 2015 at 7:36 AM, H. Hirzel <hannes.hirzel at gmail.com>
>> wrote:
>>
>>> Squeak 5.0 still has ImageSegments (class comment below). Saving and
>>> loading of  projects currently is blocked in Squeak because of an
>>> unfinished name space (environments implementation).
>>>
>>> http://wiki.squeak.org/squeak/6218
>>>
>>> And there is Fuel
>>> http://rmod.inria.fr/web/software/Fuel
>>>
>>> ----------
>>>
>>> ImageSegment
>>>
>>> I represent a segment of Squeak address space.  I am created from an
>>> array of root objects.  After storing, my segment contains a binary
>>> encoding of every object accessible from my roots but not otherwise
>>> accessible from anywhere else in the system.  My segment contains
>>> outward pointers that are indices into my table of outPointers.
>>> On load my segment is converted back into objects and becommed
>>> into an Array of the loaded objects, so they can be enumerated.
>>>
>>>         The main use of ImageSegments is to store Projects.  A dummy
>>> version of SmartRefStream traverses the Project.  Everything it finds
>>> is classified as either an object that is owned by the project (only
>>> pointed to inside the project), or an object outside the project that
>>> is pointed to from inside the project.  The objects that are
>>> completely owned by the project are compressed into pure binary form
>>> in an ImageSegment.  The outside objects are put in the 'outPointers'
>>> array.  The entire ImageSegment (binary part plus outPointers) is
>>> encoded in a SmartRefStream, and saved on the disk.  (aProject
>>> exportSegmentWithChangeSet:fileName:directory:) calls (anImageSegment
>>> writeForExportWithSources:inDirectory:changeSet:).
>>>
>>>         Note that every object inside the project is put into the
>>> segment's arrayOfRoots.  This is because a dummy SmartRefStream to
>>> scan the project, in order to make intelligent decisions about what
>>> belongs in the project.
>>>         See Project's class comment for what messages are sent to
>>> objects as they are unpacked in a new image.
>>>
>>> ---- Older Details ------
>>>
>>>         The primary kind of image segment is an Export Segment.  It
>>> can be saved on a server and read into a completely different Squeak
>>> image.
>>> Old way to create one:
>>> (ImageSegment new copyFromRootsForExport: (Array with: Baz with: Baz
>>> class))
>>>                 writeForExport: 'myFile.extSeg'.
>>> Old way to create one for a project:
>>>         (Project named: 'Play With Me - 3') exportSegment.
>>> To read it into another image:  Select 'myFile.extSeg' in a FileList,
>>> Menu 'load as project'.  It will install its classes automatically.
>>> If you need to see the roots array, it is temporarily stored in
>>> (SmartRefStream scannedObject).
>>>
>>> Most of 'states' of an ImageSegment are not used to export a project,
>>> and have been abandoned.
>>>
>>>         When a segment is written out onto a file, it goes in a
>>> folder called <image name>_segs.  If your image is called
>>> "Squeak2.6.image", the folder "Squeak2.6_segs" must accompany the
>>> image whenever your move, copy, or rename it.
>>>         Whenever a Class is in arrayOfRoots, its class (aClass class)
>>> must also be in the arrayOfRoots.
>>>         There are two kinds of image segments.  Normal image segments
>>> are a piece of a specific Squeak image, and can only be read back
>>> into that image.  The image holds the array of outPointers that are
>>> necessary to turn the bits in the file into objects.
>>>         To put out a normal segment that holds a Project (not the
>>> current project), execute (Project named: 'xxx') storeSegment.
>>>
>>>
>>> arrayOfRoots    The objects that head the tree we will trace.
>>> segment                 The WordArray of raw bits of all objects in the
>>> tree.
>>> outPointers             Oops of all objects outside the segment
>>> pointed to from inside.
>>> state                   (see below)
>>> segmentName     Its basic name.  Often the name of a Project.
>>> fileName                The local name of the file.  'Foo-23.seg'
>>> userRootCnt             number of roots submitted by caller.  Extras
>>> are added in preparation for saving.
>>>
>>> state that an ImageSegment may exist in...
>>>
>>> #activeCopy                     (has been copied, with the intent to
>>> become active)
>>> arrayOfRoots, segment, and outPointers have been created by
>>> copyFromRoots:.  The tree of objects has been encoded in the segment,
>>> but those objects are still present in the Squeak system.
>>>
>>> #active                         (segment is actively holding objects)
>>> The segment is now the only holder of tree of objects.  Each of the
>>> original roots has been transmuted into an ImageSegmentRootStub that
>>> refers back to this image segment.  The original objects in the
>>> segment will all be garbageCollected.
>>>
>>> #onFile
>>> The segment has been written out to a file and replaced by a file
>>> pointer.  Only ImageSegmentRootStubs and the array of outPointers
>>> remains in the image.  To get this far:
>>> (ImageSegment new copyFromRoots: (Array with: Baz with: Baz class))
>>>                 writeToFile: 'myFile.seg'.
>>>
>>> #inactive
>>> The segment has been brought back into memory and turned back into
>>> objects.  rootsArray is set, but the segment is invalid.
>>>
>>> #onFileWithSymbols
>>> The segment has been written out to a file, along with the text of
>>> all the symbols in the outPointers array, and replaced by a file
>>> pointer.  This reduces the size of the outPointers array, and also
>>> allows the system to reclaim any symbols that are not referred to
>>> from elsewhere in the image.  The specific format used is that of a
>>> literal array as follows:
>>>         #(symbol1 symbol2 # symbol3 symbol4 'symbolWithSpaces' #
>>> symbol5).
>>> In this case, the original outPointers array was 8 long, but the
>>> compacted table of outPointers retains only two entries.  These get
>>> inserted in place of the #'s in the array of symbols after it is read
>>> back in.  Symbols with embedded spaces or other strange characters
>>> are written as strings, and converted back to symbols when read back
>>> in.  The symbol # is never written out.
>>>         NOTE: All IdentitySets or dictionaries must be rehashed when
>>> being read back from this format.  The symbols are effectively
>>> internal.  (No, not if read back into same image.  If a different
>>> image, then use #imported.  -tk)
>>>
>>> #imported
>>> The segment is on an external file or just read in from one.  The
>>> segment and outPointers are meant to be read into a foreign image.
>>> In this form, the image segment can be read from a URL, and
>>> installed.  A copy of the original array of root objects is
>>> constructed, with former outPointers bound to existing objects in the
>>> host system.
>>>         (Any Class inside the segment MUST be in the arrayOfRoots.
>>> This is so its association can be inserted into Smalltalk.  The
>>> class's metaclass must be in roots also.  Methods that are in
>>> outPointers because blocks point at them, were found and added to the
>>> roots.
>>>         All IdentitySets and dictionaries are rehashed when being
>>> read back from exported segments.)
>>>
>>>
>>> To discover why only some of the objects in a project are being
>>> written out, try this (***Destructive Test***).  This breaks lots of
>>> backpointers in the target project, and puts up an array of
>>> suspicious objects, a list of the classes of the outPointers, and a
>>> debugger.
>>> "Close any transcripts in the target project"
>>> World currentHand objectToPaste ifNotNil: [
>>>         self inform: 'Hand is holding a Morph in its paste buffer:\'
>>> withCRs,
>>>                 World currentHand objectToPaste printString].
>>> PV := Project named: 'xxxx'.
>>> (IS := ImageSegment new) findRogueRootsImSeg:
>>>         (Array with: PV world presenter with: PV world).
>>> IS findOwnersOutPtrs.   "Optionally: write a file with owner chains"
>>> "Quit and DO NOT save"
>>>
>>> When an export image segment is brought into an image, it is like an
>>> image starting up.  Certain startUp messages need to be run.  These
>>> are byte and word reversals for nonPointer data that comes from a
>>> machine of the opposite endianness.  #startUpProc passes over all
>>> objects in the segment, and:
>>>         The first time an instance of class X is encountered, (msg _
>>> X startUpFrom: anImageSegment) is sent.  If msg is nil, the usual
>>> case, it means that instances of X do not need special work.  X is
>>> included in the IdentitySet, noStartUpNeeded.  If msg is not nil,
>>> store it in the dictionary, startUps (aClass -> aMessage).
>>>         When a later instance of X is encountered, if X is in
>>> noStartUpNeeded, do nothing.  If X is in startUps, send the message
>>> to the instance.  Typically this is a message like #swapShortObjects.
>>>         Every class that implements #startUp, should see if it needs
>>> a parallel implementation of #startUpFrom:.
>>>
>>> On 11/26/15, Luciano Notarfrancesco <luchiano at gmail.com> wrote:
>>> > I don't see image segments in Cuis... I don't even know if they still
>>> exist
>>> > in Squeak... Anyone is still using it?
>>> >
>>> > It would be cool to have it at least as a package. If I remember
>>> correctly
>>> > it was very simple, like two primitives to load and save, but I think
>>> > it
>>> > depended on something like mark and sweep or some particular kind of
>>> > garbage collector and it might not work in new VMs.
>>> >
>>>
>>> _______________________________________________
>>> Cuis mailing list
>>> Cuis at jvuletich.org
>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>
>>
>