Character encoding conversion for PascalAI. GNU libiconv converts text between 100+ legacy and modern encodings — UTF-8, GBK, Shift-JIS, CP1252, KOI8-R, Latin-1, Big5, and more.
ppm install libiconv
What is libiconv?
GNU libiconv is the reference implementation for character set conversion.
It covers 100+ encodings spanning every major script: Unicode (UTF-8/16/32), Western European,
Cyrillic, Chinese, Japanese, Korean, Arabic, Hebrew, Greek, Turkish, and Thai.
On Linux, iconv is built directly into glibc and requires no additional library.
The well-known iconv command-line tool is a thin wrapper over the same API this
package exposes.
Why encoding conversion is needed
Legacy systems — Windows desktops, old databases, email servers, mainframes —
store text in non-UTF-8 encodings. A file saved in GBK on a Chinese Windows machine, or a
CSV export using CP1252 from an old ERP, will display as mojibake unless converted to UTF-8
before processing. Convert to UTF-8 first; work in UTF-8 everywhere.
Append //TRANSLIT to the target encoding to substitute similar characters when
an exact mapping does not exist (e.g. pass 'ASCII//TRANSLIT' as ToEncoding
to approximate Unicode characters with their ASCII look-alikes). Append //IGNORE
to silently drop bytes that cannot be converted instead of raising an error. Both suffixes work
with any encoding name: 'CP1252//TRANSLIT', 'GBK//IGNORE', etc.
Types
Type
Description
TIconvCD
Conversion descriptor handle returned by OpenCD. Represents an open conversion context between two fixed encodings. Must be closed with CloseCD when no longer needed.
API Reference
Open / Close
Function
Description
OpenCD(ToEncoding, FromEncoding)
Open a conversion descriptor. ToEncoding and FromEncoding are encoding name strings such as 'UTF-8', 'GBK', or 'ISO-8859-1'. Append //TRANSLIT or //IGNORE to ToEncoding for fallback behavior. Returns a TIconvCD handle.
CloseCD(CD)
Release a conversion descriptor and free all associated resources.
Convert
Function
Description
Convert(CD, Input)
Convert Input using an already-open TIconvCD. Efficient for repeated conversions between the same encoding pair — the descriptor is reused without re-opening.
ConvertString(Input, FromEnc, ToEnc)
One-shot conversion without managing a descriptor. Opens, converts, and closes internally. Convenient for infrequent conversions where performance is not critical.
Common Conversions (convenience)
Function
Description
ToUTF8(Input, FromEncoding)
Convert from any encoding to UTF-8.
FromUTF8(Input, ToEncoding)
Convert from UTF-8 to any encoding. Supports //TRANSLIT and //IGNORE suffixes.
GBKToUTF8(Input)
Convert GBK (Chinese GB2312/GBK, Windows code page 936) to UTF-8.
UTF8ToGBK(Input)
Convert UTF-8 to GBK.
SJISToUTF8(Input)
Convert Shift-JIS (Japanese Windows, CP932) to UTF-8.
CP1252ToUTF8(Input)
Convert CP1252 (Windows Western European / Latin-1 superset) to UTF-8.
KOI8RToUTF8(Input)
Convert KOI8-R (legacy Russian encoding) to UTF-8.
Latin1ToUTF8(Input)
Convert ISO-8859-1 (Latin-1) to UTF-8.
Big5ToUTF8(Input)
Convert Big5 (Traditional Chinese, Taiwan/Hong Kong) to UTF-8.
Detection
Function
Description
DetectEncoding(Data)
Heuristically detect the encoding of a raw byte sequence. Returns an encoding name string, or 'UTF-8' when detection is inconclusive. Not guaranteed to be correct — use only when the encoding is truly unknown.
IsValidUTF8(Data)
Check whether a byte sequence is well-formed UTF-8. Returns Boolean. Useful for routing: if the input is already valid UTF-8, skip conversion entirely.
File
Procedure
Description
ConvertFile(InPath, OutPath, FromEnc, ToEnc)
Read the file at InPath, convert its encoding from FromEnc to ToEnc, and write the result to OutPath. Handles large files without loading the entire content into memory.
Info
Function
Description
ListEncodings()
Return an array of string containing all encoding names supported by the installed libiconv.
LibIconvVersion()
Return the libiconv library version string (e.g. '1.17').
Code Example
uses libiconv;
var
CD: TIconvCD;
GBKText, UTF8Text: string;
begin{ Read a GBK-encoded file (common in China) }
GBKText := ReadFile('/data/chinese_data.txt'); { raw bytes }{ Convert to UTF-8 for processing }
UTF8Text := GBKToUTF8(GBKText);
WriteLn(UTF8Text);
{ Or use a CD for repeated conversions (more efficient) }
CD := OpenCD('UTF-8', 'GBK');
UTF8Text := Convert(CD, GBKText);
CloseCD(CD);
{ Convert UTF-8 back to CP1252 for a Windows app }WriteLn(FromUTF8('Héllo Wörld', 'CP1252//TRANSLIT'));
end.
Common Use Cases
Scraping old websites that serve content in Shift-JIS (Japan) or GBK (China) without proper Content-Type headers.
Importing legacy database exports (.csv, .sql) generated by pre-Unicode systems using CP1252, CP1251, or ISO-8859-x.
Reading text files copied from Windows machines that use CP1252 by default instead of UTF-8.
Processing email attachments where MIME headers declare a legacy charset like ISO-2022-JP or KOI8-R.
Install
ppm install libiconv
Built into glibc on Linux — no install needed. brew install libiconv on macOS.
Package Info
Version1.0.0
Typeclib
CategoryEncoding
Encodings100+
API
OpenCD / CloseCD
Convert(CD, Input)
ConvertString
ToUTF8 / FromUTF8
GBKToUTF8 / UTF8ToGBK
SJISToUTF8
CP1252ToUTF8
KOI8RToUTF8
Latin1ToUTF8
Big5ToUTF8
DetectEncoding
IsValidUTF8
ConvertFile
ListEncodings
LibIconvVersion
Related packages
For full Unicode normalization, locale-aware collation, and script transliteration see
libicu. For regex with
Unicode category support see pcre2.