Unicode Support in ICU for Java Doug Felt Globalization Center of Competency, San Jose, CA

2 Overview What is ICU4J? ICU and the JDK, a brief history Benefits and tradeoffs of ICU4J Features of ICU4J Performance of ICU4J Using ICU4J Conclusion and References

3 What is ICU4J? Internationalization Library –Sister project of ICU (C/C++) –Open-source, non-viral license –Sponsored by IBM Unicode Standard compliant, up-to-date 100% Pure Java Enhances and extends JDK functionality Over five years of continuous development

4 ICU and Java, a History Started with Java 1.1 internationalization –Much code contributed by IBM/Taligent –IBM provided support, bug fixes, enhancements Became open-source project in 2000 –ICU4C code started with port from Java Continued contributions to Java since then –TextLayout, OpenType layout, Normalization

5 Collaboration with Java Teams We continue to work with Java internationalization, graphics2D teams We participate in Java expert groups (e.g. JSR 204, Supplementary Support) Differences –perspectives (conformance, features versus size) –processes (open source versus corporate/JSR) –timetable (twice a year versus every two years)

6 Benefits Fully implements current standards –Unicode collation, normalization, break iteration –Updated more frequently than Java Full CLDR data Improved performance Open source, open license, customizable Compatible with ICU C/C++ libraries and data Runs on JDK 1.4 –Get supplementary support without moving to 1.5

7 Tradeoffs Not built-in, unlike Java i18n support Some API differences –But generally a superset of the Java API –Some differences unavoidable due to class restrictions –Rule syntax differs to varying degrees Data differences –ICU4J uses its own CLDR data, not the JVM’s data Size –Can trim ICU4J, but it will always be larger than 0K

8 Features of ICU4J Collation Normalization Break Iteration UnicodeSet and Transforms Character Properties Locale data Other –Calendars, Formatters, IDNA, StringPrep, IMEs

9 Collation Full UCA (Unicode Collation Algorithm) –Java does not implement UCA collation Locale data –Over 60 tailorings for locale-specific collation –Variants: Pinyin, stroke, traditional, etc. Performance –sorting: 2 to 20 times faster –sort key generation: 1.5 to 4 times faster –sort key length: 2/3 to 1/4 the length of Java sort keys

10 Normalization Java does not provide normalization APIs –Java uses ICU’s implementation internally –Useful for searching, string equivalence, simplifying processing of text Full implementation of Unicode standard –NFC, NFD, NFKC, NFKD –Also provides FCD ‘quick check’ for optimization

11 Break Iteration Fully conforms to Unicode specifications –supplementary characters, Hangul Tags –e.g., “what kind of word was this” Title case iteration Rule-based, dictionary-based for Thai

12 Unicode Set and Transforms UnicodeSet –collections of characters based on properties –logical set operations, flexible –“[[:mark:]&[\u0600-\u067f]]” Transliterator –general transformations, with chaining and editing –converts between scripts, e.g. Greek/Latin, Devanagari/Gujarati –rule-based, rules for common conversions supplied\ UScriptRun

13 Character Properties All Unicode character properties –over 80, Java provides access to about 10 All defined code points Current with latest Unicode release –ICU4J 3.0 uses Unicode data Fast access to character data

14 Locale Data Standard data, included with ICU4J –CLDR (Common Locale Data Repository) –Ensures same data is available everywhere –Can share resource data with ICU4C applications More locales, more kinds of data –~230 locales, compared to ~130 for Java –Can modularize to include only the data you need RFC3066bis support (language_script_region) –e.g., zh_Hans, zh_Hant –keywords (orthogonal variants)

15 Performance of ICU4J Instantiation times are comparable –Common instantiate and reuse model –ICU4J and Java both use caches to limit impact Collation performance faster –faster sorting, smaller sort keys Performance is difficult to measure –JVM makes a difference –ICU4J performs well in spot tests –Use a scenario that matters to you to test

16 Property Data Timings JVMICU4JJava(J-I)/I Sun ns/op101 ns/op13% Sun 1.5.0b2117 ns/op102 ns/op-13% IBM ns/op66 ns/op32% 1.13MHz PIII, Win2K Nanoseconds/operation for character property access (getType, toLowerCase, getDirectionality) on three JVMs.

17 Sizes of ICU4J Full jar file: 2,700K Modular builds for common subsets –normalizer: 420K –collator: 1,400K –calendar: 1,300K –break iterator: 1,300K –basic properties: 500K –full properties: 1,200K –formatting: 2,200K –transforms: 1,500K

18 Using ICU4J Jar file, just add to class path –Or roll into your distribution, it’s Open Source! –Modular builds help you to trim ICU4J’s code –Data can be trimmed to further reduce size Parallel APIs –APIs on parallel classes are generally a superset –Change import (one line change) or change class name –Some differences unavoidable (our supplementary support for Java 1.4 can’t add API to String)

19 Code Examples (1) import com.ibm.icu.text.BreakIterator; BreakIterator b = BreakIterator.getWordInstance(); b.setText(text); for (int pos = b.first(); pos != BreakIterator.DONE; pos = b.next()) { doSomething(pos); }

20 Code Examples (2) import com.ibm.icu.lang.UCharacter; int cp, pos = 0; while (pos < text.length()) { cp = UCharacter.codePointAt(text, pos); if (UCharacter.getType(cp) == UCharacter.SURROGATE) return true; pos += UCharacter.charCount(cp); }

21 Code Examples (3) import com.ibm.icu.util.ULocale; import com.ibm.icu.text.Collator; import java.util.Arrays; ULocale ulocale = new Collator col = Collator.getInstance(ulocale); String[] list =... Arrays.sort(list, col);

22 Conclusion ICU4J is not for you if –you have tight size constraints –you require the Java runtime behavior ICU4J is for you if –you need full compliance with current standards –you need current or additional locale and property data –you need customizability –you need features missing from Java (normalization) –you need additional performance

23 References ICU4J – Java – – Unicode, CLDR – –