[Cuis] Unicode Support

[Juan Vuletich]

Hi Euan,

On 12/4/2015 8:42 AM, EuanM wrote:
> I'm currently groping my way to seeing how feature-complete our
> Unicode support is.  I am doing this to establish what still needs to
> be done to provide full Unicode support.
>
> This seems to me to be an area where it would be best to write it
> once, and then have the same codebase incorporated into the Smalltalks
> that most share a common ancestry.
>
> I am keen to get: equality-testing for strings; sortability for
> strings which have ligatures and diacritic characters; and correct
> round-tripping of data.
>

Current state (as I understand it):

- Squeak: M17N by Yoshiki

- Pharo: Inherited Squeak code. I don't know how much it has diverged.

- Cuis: Chose not to use Squeak approach. Chose to make the base image 
include and use only 1-byte strings. Chose to use ISO-8859-15, the most 
complete standard for the Latin alphabet, including diacritics, etc. 
Includes limited support for Unicode. See 
https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/blob/master/Documentation/UnicodeNotes.md


Unicode is a complex area, and no system can claim to do it all right. I 
see several possible levels of Unicode support. These are (with 
Smalltalk examples):

0) No Unicode support at all. Like ST-80 or Squeak previous to 3.7.

1) Limited Unicode support (as in Cuis). Can handle Unicode character in 
Strings. Comfortable Display / edit of text is restricted to Latin 
alphabet. Non ISO-8859-15 characters are represented as NCRs, but are 
not instances of Character themselves. For example, an NCR such as 
'α' (made of 5 8-bit Characters) represents the greek letter Alpha, 
and is properly handled if such string is converted to an UTF-8 
ByteArray (for example for copying into the Clipboard or for serving Web 
pages). In short, you can not directly edit or display general Unicode, 
but you can embed it in code, include it in Strings, copy&paste, and 
serve web pages with it.

2) Ken's Cuis-Smalltalk-Unicode. Can display and edit. Includes the 
great Ropes representation for Strings. Limited font support.

3) Squeak. Can display and edit Unicode strings. Includes broad font 
support with TrueType / OpenType. Does not do grapheme composition, 
right to left, or ligatures.

4) Scratch. Most complete support, including all that's missing in the 
previous approaches. Uses PanGo. This means that text composition is not 
in Smalltalk, but in a large and complex external library. This sounds 
appropriate: Full Unicode is so complex that it took PanGo many years do 
it reasonably well, and most projects rely on it.

I believe that your objectives can be done both with Squeak's approach 
and with Cuis' approach. Cuis, for instance, is only missing methods for 
normalization (for characters with multiple code point representations) 
and sorting; all done on UTF-8 ByteArrays.

Cheers,
Juan Vuletich

References:
NCR: http://en.wikipedia.org/wiki/Numeric_character_reference
Ken's: https://github.com/KenDickey/Cuis-Smalltalk-Unicode

> Call to action:
> ==========
>
> If you have comments on these proposals - such as "but we already have
> that facility" or "the reason we do not have these facilities is
> because they are dog-slow" - please let me know them.
>
> If you would like to help out, please let me know.
>
> If you have Unicode experience and expertise, and would like to be, or
> would be willing to be, in the  'council of experts' for this project,
> please let me know.
>
> If you have comments or ideas on anything mentioned in this email
>
> In the first instance, the initiative's website will be:
> http://smalltalk.uk.to/unicode.html
>
> I have created a SqueakSource.com project called UnicodeSupport
>
> I want to avoid re-inventing any facilities which already exist.
> Except where they prevent us reaching the goals of:
>    - sortable UTF8 strings
>    - sortable UTF16 strings
>    - equivalence testing of 2 UTF8 strings
>    - equivalence testing of 2 UTF16 strings
>    - round-tripping UTF8 strings through Smalltalk
>    - roundtripping UTF16 strings through Smalltalk.
> As I understand it, we have limited Unicode support atm.
>
> Current state of play
> ===============
> ByteString gets converted to WideString when need is automagically detected.
>
> Is there anything else that currently exists?
>
> Definition of Terms
> ==============
> A quick definition of terms before I go any further:
>
> Standard terms from the Unicode standard
> ===============================
> a compatibility character : an additional encoding of a *normal*
> character, for compatibility and round-trip conversion purposes.  For
> instance, a 1-byte encoding of a Latin character with a diacritic.
>
> Made-up terms
> ============
> a convenience codepoint :  a single codepoint which represents an item
> that is also encoded as a string of codepoints.
>
> (I tend to use the terms compatibility character and compatibility
> codepoint interchangably.  The standard only refers to them as
> compatibility characters.  However, the standard is determined to
> emphasise that characters are abstract and that codepoints are
> concrete.  So I think it is often more useful and productive to think
> of compatibility or convenience codepoints).
>
> a composed character :  a character made up of several codepoints
>
> Unicode encoding explained
> =====================
> A convenience codepoint can therefore be thought of as a code point
> used for a character which also has a composed form.
>
> The way Unicode works is that sometimes you can encode a character in
> one byte, sometimes not.  Sometimes you can encode it in two bytes,
> sometimes not.
>
> You can therefore have a long stream of ASCII which is single-byte
> Unicode.  If there is an occasional Cyrillic or Greek character in the
> stream, it would be represented either by a compatibility character or
> by a multi-byte combination.
>
> Using compatibility characters can prevent proper sorting and
> equivalence testing.
>
> Using "pure" Unicode, ie. "normal encodings", can cause compatibility
> and round-tripping probelms.  Although avoiding them can *also* cause
> compatibility issues and round-tripping problems.
>
> Currently my thinking is:
>
> a Utf8String class
> an Ordered collection, with 1 byte characters as the modal element,
> but short arrays of wider strings where necessary
> a Utf16String class
> an Ordered collection, with 2 byte characters as the modal element,
> but short arrays of wider strings
> beginning with a 2-byte endianness indicator.
>
> Utf8Strings sometimes need to be sortable, and sometimes need to be compatible.
>
> So my thinking is that Utf8String will contain convenience codepoints,
> for round-tripping.  And where there are multiple convenience
> codepoints for a character, that it standardises on one.
>
> And that there is a Utf8SortableString which uses *only* normal characters.
>
> We then need methods to convert between the two.
>
> aUtf8String asUtf8SortableString
>
> and
>
> aUtf8SortableString asUtf8String
>
>
> Sort orders are culture and context dependent - Sweden and Germany
> have different sort orders for the same diacritic-ed characters.  Some
> countries have one order in general usage, and another for specific
> usages, such as phone directories (e.g. UK and France)
>
> Similarly for Utf16 :  Utf16String and Utf16SortableString and
> conversion methods
>
> A list of sorted words would be a SortedCollection, and there could be
> pre-prepared sortBlocks for them, e.g. frPhoneBookOrder, deOrder,
> seOrder, ukOrder, etc
>
> along the lines of
> aListOfWords := SortedCollection sortBlock: deOrder
>
> If a word is either a Utf8SortableString, or a well-formed Utf8String,
> then we can perform equivalence testing on them trivially.
>
> To make sure a Utf8String is well formed, we would need to have a way
> of cleaning up any convenience codepoints which were valid, but which
> were for a character which has multiple equally-valid alternative
> convenience codepoints, and for which the string currently had the
> "wrong" convenience codepoint.  (i.e for any character with valid
> alternative convenience codepoints, we would choose one to be in the
> well-formed Utf8String, and we would need a method for cleaning the
> alternative convenience codepoints out of the string, and replacing
> them with the chosen approved convenience codepoint.
>
> aUtf8String cleanUtf8String
>
> With WideString, a lot of the issues disappear - except
> round-tripping(although I'm sure I have seen something recently about
> 4-byte strings that also have an additional bit.  Which would make
> some Unicode characters 5-bytes long.)
>
>
> (I'm starting to zone out now - if I've overlooked anything - obvious,
> subtle, or somewhere in between, please let me know)
>
> Cheers,
>      Euan