ICU (International Components for Unicode) for PascalAI. Unicode normalization, locale-aware collation and sorting, date/number formatting, transliteration between scripts, text boundary detection, charset encoding conversion — the gold standard for i18n.
ppm install libicu
System dependency: requires
libicu-dev (Ubuntu/Debian:
sudo apt install libicu-dev). On macOS:
brew install icu4c. On Windows: download the ICU binary distribution from
icu.unicode.org.
Overview
ICU is the Unicode Consortium's reference implementation for internationalization. It handles
the full complexity of human language that ASCII-era string libraries ignore: Turkish dotless i, German umlaut
collation, right-to-left Arabic text, Japanese word boundaries, French date formats, and the thousand other
cases where naive string handling produces wrong results for non-English users.
The PascalAI libicu binding wraps ICU's C API in a clean, resource-safe interface. All opaque
handle types (collators, formatters, break iterators, converters) are allocated with a New*
function and released with the corresponding Free* function.
Quick Start
Normalization and Case
uses libicu;
{ Normalize to NFC (standard form for storage and comparison) }
var s := Normalize('café', nfNFC);
{ Locale-aware case (Turkish 'I' -> 'i' with dot, not plain 'i') }
WriteLn(ToLower('ISTANBUL', 'tr_TR')); { 'istanbul' }
WriteLn(ToLower('ISTANBUL', 'en_US')); { 'istanbul' }
WriteLn(ToUpper('istanbul', 'tr_TR')); { 'ISTANBUL' with dotted I }
Locale-Aware Sorting
uses libicu;
var col := NewCollator('de'); { German collation rules }
var words := ['Arger', 'Apfel', 'Zebra', 'ahnlich'];
SortStrings(col, words);
{ Result: ['ahnlich', 'Apfel', 'Arger', 'Zebra'] -- a-umlaut sorts near a }
FreeCollator(col);
Date and Number Formatting
uses libicu;
var dateFmt := NewDateFormat(dsMedium, dsShort, 'fr_FR');
WriteLn(FormatDate(dateFmt, 1700000000000)); { '14 nov. 2023, 22:13' }
FreeDateFormat(dateFmt);
var numFmt := NewCurrencyFormat('de_DE', 'EUR');
WriteLn(FormatNumber(numFmt, 1234567.89)); { '1.234.567,89 EUR' }
FreeNumberFormat(numFmt);
API Reference
Normalization
ICU supports four Unicode normalization forms. NFC (Canonical Decomposition followed by Canonical
Composition) is the standard form for storage and string comparison. NFD decomposes characters
into base + combining marks. NFKC/NFKD additionally fold compatibility characters (e.g. ligatures,
half-width katakana) into their canonical equivalents.
| Function | Description |
|---|
| Normalize(s, form) | Return s in the given normalization form; form is one of nfNFC, nfNFD, nfNFKC, nfNFKD |
| IsNormalized(s, form) | Return True if s is already in the given normalization form (fast check, no allocation) |
Case Conversion
| Function | Description |
|---|
| ToUpper(s, locale) | Locale-aware uppercase conversion; handles Turkish dotted I, Greek final sigma, etc. |
| ToLower(s, locale) | Locale-aware lowercase conversion; pass an empty string for the default locale |
| ToTitle(s, locale) | Title-case conversion (first letter of each word uppercased) following Unicode word-break rules for the locale |
| CaseFold(s) | Locale-independent case folding for case-insensitive comparison; more aggressive than ToLower |
Collation
| Function | Description |
|---|
| NewCollator(locale) | Create a collator for the given locale string (e.g. 'de', 'zh_CN', 'es_419'); returns a collator handle |
| FreeCollator(col) | Release a collator handle and its associated resources |
| Compare(col, a, b) | Compare two strings using locale collation rules; returns -1, 0, or 1 |
| SortStrings(col, arr) | In-place sort of a string array using the collator's ordering; stable sort |
| GetSortKey(col, s) | Return a byte array sort key for s; memcmp on sort keys gives the same order as Compare |
Date Formatting
| Function | Description |
|---|
| NewDateFormat(dateStyle, timeStyle, locale) | Create a date/time formatter; styles are dsNone, dsShort, dsMedium, dsLong, dsFull |
| NewDateFormatPattern(pattern, locale) | Create a formatter with an explicit ICU skeleton pattern (e.g. 'yyyy-MM-dd HH:mm') |
| FreeDateFormat(fmt) | Release a date format handle |
| FormatDate(fmt, epochMs) | Format a Unix timestamp (milliseconds since epoch) as a locale string |
| ParseDate(fmt, s) | Parse a locale-formatted date string and return a Unix timestamp in milliseconds |
| FormatRelativeTime(locale, diffSec) | Format a signed seconds offset as a relative string like '3 hours ago' or 'in 2 days' |
Number Formatting
| Function | Description |
|---|
| NewNumberFormat(locale) | Create a decimal number formatter for the given locale; handles decimal separators, grouping, etc. |
| NewCurrencyFormat(locale, currency) | Create a currency formatter; currency is a 3-letter ISO 4217 code (e.g. 'EUR', 'JPY') |
| NewPercentFormat(locale) | Create a percentage formatter that multiplies by 100 and appends the locale's percent symbol |
| FreeNumberFormat(fmt) | Release a number format handle |
| FormatNumber(fmt, value) | Format a Float value as a locale string using the formatter's rules |
| ParseNumber(fmt, s) | Parse a locale-formatted number string and return a Float |
Transliteration
Transliterators convert text between scripts or apply Unicode transformations. Common IDs:
'Latin-Cyrillic', 'Cyrillic-Latin', 'Any-Latin',
'Latin-Arabic', 'Hiragana-Katakana',
'NFD; [:Mn:] Remove; NFC' (strip all accent/combining marks).
Call ListTranslitIDs to enumerate all available transforms.
| Function | Description |
|---|
| NewTranslit(id) | Create a transliterator from an ICU transform ID string; returns a translit handle |
| FreeTranslit(t) | Release a transliterator handle |
| Transliterate(t, s) | Apply the transform to string s and return the result |
| ListTranslitIDs() | Return a List of all registered transform IDs available in the current ICU installation |
Text Boundary Detection
| Function | Description |
|---|
| NewWordBreaker(locale) | Create a word-boundary iterator for the given locale; aware of CJK, Thai, and other non-space-delimited scripts |
| NewSentenceBreaker(locale) | Create a sentence-boundary iterator; handles abbreviations, ellipses, and quotation marks correctly |
| NewLineBreaker(locale) | Create a line-break opportunity iterator for text layout and wrapping |
| FreeBreakIter(bi) | Release a break iterator handle |
| BreakText(bi, s) | Return a List of substrings split at each boundary position for the given text |
| CountWords(locale, s) | Convenience function returning the number of words in s using locale word-break rules |
Encoding Conversion
| Function | Description |
|---|
| NewConverter(fromEnc, toEnc) | Create a charset converter between two encoding names (e.g. 'windows-1252' to 'UTF-8') |
| FreeConverter(cv) | Release a converter handle |
| BytesToUTF8(cv, bytes) | Convert a byte array in the source encoding to a UTF-8 string using the converter |
| UTF8ToBytes(cv, s) | Convert a UTF-8 string to a byte array in the target encoding using the converter |
| DetectCharset(bytes) | Heuristically detect the charset of a raw byte array; returns a TCharsetMatch with Name and Confidence (0.0–1.0) |
Locale Information
| Function | Description |
|---|
| GetLocaleInfo(locale) | Return a TLocaleInfo record with DisplayName, Language, Country, Script, and Variant fields |
| ListLocales() | Return a List of all locale identifiers available in the current ICU data file |
| DefaultLocale() | Return the ICU default locale string for the current process (e.g. 'en_US') |
IDNA / Punycode
IDNA (Internationalized Domain Names in Applications) converts Unicode domain names to their ASCII-compatible
encoding for DNS lookup. Example: 'münchen.de' → 'xn--mnchen-3ya.de'.
| Function | Description |
|---|
| DomainToASCII(domain) | Convert a Unicode domain name to its ACE/Punycode form for DNS queries (e.g. 'münchen.de' → 'xn--mnchen-3ya.de') |
| DomainToUnicode(domain) | Convert an ACE/Punycode domain back to its Unicode display form (e.g. 'xn--mnchen-3ya.de' → 'münchen.de') |