TechEssence.info

The only thing that computers understand, fundamentally, is the difference between a one and a zero. Numbers. That's all. To express letters so that computers can manipulate them, each letter has to be assigned a unique number.

Self-evidently, the total of the world's languages includes a lot more letters than just the English alphabet, so when computers migrated out of Anglophone countries, letter-representation systems had to expand and adjust. The eventual result was Unicode.

The 1960s and 1970s witnessed many competing attempts to assign numbers to letters. Increasing internationalization only made the problem worse and more urgent. Eventually (simplifying extremely!) the "solution" settled upon was a plethora of "character encodings" using the same set of numbers to represent different sets of letters; any document sent had to signal somehow what encoding it was using, so that the computer could display the proper letters.

If you've ever gotten a spam email that looked like gobbledygook in your email application, you've been the victim of a character encoding problem. Either the email didn't correctly signal which encoding it was using, or your computer doesn't know how to handle that particular encoding. Or, of course, both.

The real nail in the coffin of these character-encoding systems is a multilingual document or database. Such as—a research library's catalogue.

Finally in the late 1980s, the International Standards Organization decided enough was enough. They determined that the way forward was to expand the set of numbers used to represent letters hugely, and assign each letter its own unique number, so that there was no possibility of confusion.

Of course it didn't turn out to be that simple. Exactly what a "letter" is became a vexed question. Also there are several shortcuts (each, confusingly, called an "encoding;" examples include UTF-8 and UTF-16) that computers can use to represent Unicode characters. Still, Unicode is and will likely remain the gold standard for computer representation of the world's languages.

What does this mean for you? Well, if you have been using MARC's anglicizations of foreign alphabets in your catalogue, now may be the time to look into converting your records to Unicode. This has been done mechanically on very large and linguistically-diverse catalogues with impressively accurate results.

If you have internationalized content on your web pages, you should check what character encoding it is in; if that encoding is not Unicode, you should absolutely consider conversion. Be aware, also, that some popular web-design applications (FrontPage is an egregious offender) do not use Unicode—they use Microsoft-proprietary and heavily discredited encodings instead. Fix these, or ask someone else to! Future web-page maintainers will thank you.

Jukka Korpela has an occasionally techie but still readable tutorial on character encodings. Recommended.