Unicode Normalization Mark Davis

Normalization Uniqueness two equivalent strings have precisely the same normalized form Fast binary comparison, accurate digital signatures Recommended for XML, JavaScript and other standards

Canonical Equivalence Fundamental equivalence Indistinguishable to users, when correctly rendered Includes Combining sequences Hangul Singletons Ω C¸Ç

Compatibility Equivalence Formatting differences Font variants ( ) Breaking differences (-) Cursive forms ( ) Circled ( ) Width, size, rotated ( ) Super/subscripts ( ) Squared characters ( ) Fractions ( ) Others ( dž ) fi kg

UTR #15: Unicode Normalization Forms Form DCanonical Decomposition Form KDCompatibility Decomposition Form C Form D + Canonical Composition Form KC Form KD + Canonical Composition

Normalization Requirement Uniqueness: two equivalent strings will have precisely the same normalized form If two strings x and y are canonical equivalents, then C(x) = C(y) D(x) = D(y) If two strings are compatibility equivalents, then KC(x) = KC(y) KD(x) = KD(y)

Affected Characters None of the forms affect text with only ASCII characters (U+0000 to U+007F) None of the forms generate compability characters that were not in the source text. Both KD and KC replace compatibility characters. Both D and C maintain compatibility characters.

Cautions: Decomposition Requires decomposition mappings from the Unicode Character Database Those decomposition mappings must be applied recursively The string must be put into canonical order Either Canonical or Compatibility

Cautions: Composition Decomposition required first! Then canonical composition Composition data: fixed at Unicode Some characters are excluded from composition Form C and Form KC can still have combining characters! Required for Indic, Arabic, Hebrew, &c.

Caution: Both C & D All normalization forms are not closed under string concatenation. Example: NFC/D "…a̰ " + " ̀…" Not Norm. "…à̰ …" NFC "…à̰ …" NFD "…a ̰̀ …" Exceptions easy to test for

Composition Process 1. Decompose (D or KD) 2. Combine unblocked characters with the previous starter, if possible*

Composition Exclusions Script Specifics + ̣ Futures: G + ̣ G ̣ Singletons* Ω Ω Non-starter sequences* ̈ + ́ ̈́

Legacy Encoding Legacy text is normalized if it maps 1:1 to normalized Unicode text Legacy sets: Prenormalized: e.g. ISO Normalizable: e.g. ISO 2022 (ISO 5426/ISO /…) Unnormalizable: e.g. ISO 5426

Programming Identifiers Closed under all Normalization Forms, if minor changes incorporated Modified syntax: identifier := start ( start | extend )* start := [{Lu}{Ll}{Lt}{Lm}{Lo}{Nl}] - irregulars – combining_like extend := [{Mn}{Mc}{Nd}{Pc}{Cf}] - irregulars + combining_like + mid_dot (Almost) closed under Case Mappings see SpecialCasing.txt

Resources Reference version on Unicode Site Production Version ICU: C/C++ and Java Versions Open Source, with IBM Public License Free commercial use and distribution: Not Viral! Panel Later today Other companies also providing: ask!

Normalization Uniqueness: two equivalent strings have precisely the same normalized form Fast binary comparison, accurate digital signatures Recommended for XML, JavaScript and other standards

Q & A

Backup Slides

Definition: Starter S is a starter = Canonical class of zero in the Unicode Character Database Can start a composition Examples: Starters: Spacing marks, some non-spacing a, ق Θ Non-starters: most non-spacing marks ̀, ̊ ̽ ̥

Definition: Blocked C is blocked from S There is some character B between S and C, and either B is a starter or B has the same canonical class as C Examples ABC – B blocks C from A A ̀̊ – ̀ blocks ̊ from A Ḁ̊ –̥ doesnt block ̊ from A

Testing Conformance: Canonical For all Unicode characters X C(X) = C(D(X) D(X), C(X) in canonical order CDMNo CDM X = D(X) X = C(X) X D(X) No characters in D(X) have CDM X Exclusions X C(D(X)X = C(D(X)

Unicode Normalization Introduction Normalization forms Design goals Specification Excluded characters Versions Legacy encodings Applications

Characters and Encoding Forms Å A ° C5 AbstractEncoded 212B F A Serialized B DB80DC A C5 UTF-16BE UTF-8 C3 E284 F3B080 61CC8A 85 AB