Squawks of the Parrot

The Pain of Text

Yeah, this stuff's all getting cranky. Deal. :)

At the moment, I'm trying to work on specifying text stuff for Parrot. Not simple, of course, because text is such a massive pain. Right now I'm just trying to sort the various functions on characters and strings into the right spot so they can be properly overridden, thumped, assaulted, and generally beaten about.

If you've been following along, you've no doubt seen the rants about text, so I won't reprise them (much) and instead go for the actual useful bits. As far as I can tell (and this is all welded deep into parrot's string handling), there are three basic parts to this:

1. Encoding to turn a stream of octets into integer code points
2. Character set To give meaning to the code points
3. Language To determine the behaviour of the code points

And yeah, there is some overlap between what the character set and the language does. That's part of the problem I'm facing--this stuff's all been invented a dozen or more times and the decisions that were made were all very reasonable but not entirely compatible.

The encoding bit's the least controversial of the layers. Even with the multibyte non-Unicode encodings that do escape-byte stuff there's a pretty straightforward way to map to a 32 bit integer, so that part's easy. (Note that easy and boring have no relation -- putting together the different byte-to-codepoint mapping tables and corresponding code is going to be terribly dull)

The character set's generally non-controversial as long as you don't pay attention to how the individual characters actually look. (If you do, then fights break out and it's not pretty) Unfortunately Unicode adds in the twist of combining characters ¹ so it's not quite enough to look at a single code point to get a single character -- in many common cases for me (most of the languages of western Europe) you've the potential of needing two or more code points to represent a single character. I'm not sure if there are cases where it's reasonable to deal with the parts of a character², but people seem to insist. I dunno, I can't see any circumstances where n and ñ (that second character's an n with a tilde over it, which can be represented as two code points in Unicode) could be in any way equivalent, but what the heck do I know?

The language bit... that's where things get interesting. Not fun, mind, but interesting. Language is where I put the transforms and meaning-association stuff. Case-folding is a matter for language, as are character classification, sorting, and more complex things like word-break determination.

Part of the problem with the language bit is in defaulting -- it arguably ought to be pulled in from the character set, since the language code ought to be independent of the character set but some of the sets are huge (Unicode!) and often you just don't care for characters out of your language. If you've got a string of Chinese text with "llama" thrown in there you're probably going to treat that as a five character word rather than a four character one. (Yes, I know, these days in most places even text that's really marked as spanish treats ll as a two-character sequence rather than one--humor me, I couldn't find an accented character in a non-roman-based character set in the 20 seconds I took to look) Then there's the issue of whether some of these characters ought to be classified one way or another. Even if you have Unicode text, should 一 (ichi, one in Japanese) (which should look like a horizontal bar, assuming it pasted in right, and I got the right character, and you can view it... isn't text fun?) be considered a digit if it's in a string tagged as French? (Heck, should it be in there in a string tagged as Japanese? The number/non-number-word distinction's a bit fuzzy there. Or at least I'm fuzzy on it) Should it even be considered a word character? And yes, you can argue that this is a good reason to put in restrictions on allowable data, but that's not something Parrot can really do, so it needs dealing with.

Parts of the language handling code are also intimately tied to the character set (you can't upcase "a" to "A" if you don't know that the code point you were handed was an "a") so you almost need to have a multiple-dispatch system set up with per-charset language tables and/or code. Fun enough with roman-based alphabets but it gets potentially really fun when you start throwing in all the asian text variants. (I think, ultimately, it'll be relatively simple. Despite the fact that it looks like you could use Shift-JIS (a Japanese character set) to write out a chunk of Chinese text, I'm not sure it'd be considered Chinese, in which case we have a much more restricted set of charset/language pairs. Except for Unicode, which'll sleep with anyone)

Anyway, I think we can do layering stuff enough to hide this. The encoding layer can hand codepoints back and forth, some transcoding, and that's about it. (well, that and some metadata -- lengths and such) Easy enough.

The character set layer can hand you characters, and provide some defaults for the language code to work with. Most of the per-character informational code can live here (is it a letter, is it a digit, is it upper-case, and so on) though the language code potentially ought to get in the way. That'll be simple delegation for the most part.

The language layer is where the transformational and multicharacter fun lives. Case-mangling and word break detection live here (and yes, I know, for some languages word break detection requires an extensive dictionary, complex heuristics, a lunar calendar, and a good random number generator, but...) as do a few other things.

So, for the moment, the list is (and yeah, I'll post it to the internals list):

Encoding
read_codepoint
write_codepoint
to_encoding
from_encoding
knows_encoding

Character Set
read_character
write_character
substring
defaults for language

Language
is_upper
is_lower
upcase
downcase
titlecase
is_alpha
is_number
is_space
charname (maybe)
is_wordbreak (this'll have to take a position)
compare (takes two strings for sorting)

Ah, damn, then there's the question of whether sorting and equivalence testing should take encoding into account or not. (This is for Unicode mainly where you can have composed and decomposed versions of the same character) Though the answer there is "it depends", I'm sure. Bugger.

¹ Something I think may be unique to Unicode. Might be wrong about that, though.
² As opposed to the individual bytes of the encoded, which are useful to deal with