April 25, 2005 Globalization Features in Whidbeys CLR Michael Kaplan Technical Lead Globalization Infrastructure, Fonts and Tools Microsoft Windows International Division

April 25, 2005 Customized Cultures and Regions CultureAndRegionInfoBuilder class CultureAndRegionInfoBuilder class Create an override to an existing culture Create an override to an existing culture Create based on an existing culture Create based on an existing culture Create from scratch Create from scratch Must be an administrator to register Must be an administrator to register Can register the file on multiple machines Can register the file on multiple machines

April 25, 2005 CultureAndRegionInfoBuilder sample CultureAndRegionInfoBuilder carib = new CultureAndRegionInfoBuilder(de-DE-MineMine, CultureAndRegionModifiers.None); // load up all of the existing data for German and for Germany.... carib.LoadDataFromCultureInfo(new CultureInfo(de-DE", false)); carib.LoadDataFromRegionInfo(new RegionInfo(de); // Change a property carib.ThreeLetterISORegionName = ZZZ; // Register the culture on the machine carib.Register(); // Use the new culture CultureInfo ci = new CultureInfo(de-DE-MineMine);

April 25, 2005 CaRIB serialization with LDML Locale Data Markup Language Locale Data Markup Language Described in UTS#35 at Described in UTS#35 at CaRIB objects can be saved as LDML files CaRIB objects can be saved as LDML files Data can be loaded from LDML files Data can be loaded from LDML files CaRIB will do its best with files it did not create CaRIB will do its best with files it did not create

April 25, 2005 LDML Sample string file1 = Path.GetTempFileName(); string file1 = Path.GetTempFileName();File.Delete(file1); CultureInfo ci = new CultureInfo("ar-EG"); RegionInfo ri = new RegionInfo("de-DE"); CultureAndRegionInfoBuilder carib = new CultureAndRegionInfoBuilder("x-en-US-Pepsi", CultureAndRegionModifiers.None); carib.LoadDataFromCultureInfo(ci);carib.LoadDataFromRegionInfo(ri);carib.Save(file1); carib = CultureAndRegionInfoBuilder.CreateFromLdml(file1); carib.Register();

April 25, 2005 When Windows knows more than.NET As of XPSP2, there are 25 new locales in Windows: As of XPSP2, there are 25 new locales in Windows: Bengali - India Bengali - India Croatian - Bosnia and Herzegovina Croatian - Bosnia and Herzegovina Bosnian - Bosnia and Herzegovina Bosnian - Bosnia and Herzegovina Serbian - Bosnia and Herzegovina (Latin) Serbian - Bosnia and Herzegovina (Latin) Serbian - Bosnia and Herzegovina (Cyrillic) Serbian - Bosnia and Herzegovina (Cyrillic) Welsh - United Kingdom (more info in English, in Welsh) Welsh - United Kingdom (more info in English, in Welsh) Maori - New Zealand Maori - New Zealand Malayalam - India Malayalam - India Maltese - Malta Maltese - Malta Quechua - Bolivia Quechua - Bolivia Quechua - Ecuador Quechua - Ecuador Quechua - Peru Quechua - Peru Setswana / Tswana - South Africa Setswana / Tswana - South Africa isiXhosa / Xhosa - South Africa isiXhosa / Xhosa - South Africa isiZulu / Zulu - South Africa isiZulu / Zulu - South Africa Sesotho sa Leboa / Northern Sotho - South Africa Sesotho sa Leboa / Northern Sotho - South Africa Northern Sami - Norway Northern Sami - Norway Northern Sami - Sweden Northern Sami - Sweden Northern Sami - Finland Northern Sami - Finland Lule Sami - Norway Lule Sami - Norway Lule Sami - Sweden Lule Sami - Sweden Southern Sami - Norway Southern Sami - Norway Southern Sami - Sweden Southern Sami - Sweden Skolt Sami - Finland Skolt Sami - Finland Inari Sami - Finland Inari Sami - Finland There will be more in future service packs There will be more in future service packs In Longhorn, there will be 75 or more new locales In Longhorn, there will be 75 or more new locales

April 25, 2005 Windows-only Cultures The solution: Windows-only cultures! The solution: Windows-only cultures! Synthesizes a CultureInfo object when Windows supports a locale that the.NET Framework does not know how to create itself Synthesizes a CultureInfo object when Windows supports a locale that the.NET Framework does not know how to create itself

April 25, 2005 Windows only culture test foreach(CultureInfo culture in CultureInfo.GetCultures(CultureTypes.WindowsOnlyCultures)) { Console.WriteLine(ci.Name); Console.WriteLine(ci.Name);} // New cultures on XP SP2 include: // mt-MT, bs-BA-Latn, smn-FI, smj-NO, smj-SE, sms-FI, sma-NO, // sma-SE, quz-BO, quz-EC, quz-PE, ml-IN, bn-IN, cy-GB, and more

April 25, 2005 Special CultureInfo support for SQL Server 2005 (Yukon) SQL Server locale semantics: SQL Server locale semantics: One setting for UI and formatting One setting for UI and formatting Another setting for collation/encoding Another setting for collation/encoding.NET/Windows semantics.NET/Windows semantics One setting for UI One setting for UI Another setting for formatting/collation Another setting for formatting/collation Solution Solution Special GetCultureInfo override that takes two CultureInfo names for the two SQL Server settings Special GetCultureInfo override that takes two CultureInfo names for the two SQL Server settings

April 25, 2005 How Yukon uses this support Microsoft.ReportingServices.Diagnostics.Localization Microsoft.ReportingServices.Diagnostics.Localization CatalogCulture CatalogCulture ClientPrimaryCulture ClientPrimaryCulture DefaultReportServerCulture DefaultReportServerCulture FallbackUICulture FallbackUICulture InstalledCultureNames InstalledCultureNames ReportParameterCulture ReportParameterCulture SqlCulture SqlCulture

April 25, 2005 New locale properties/methods TextInfo TextInfo CultureName CultureName LCID LCID CompareInfo CompareInfo Name Name DateTimeFormatInfo DateTimeFormatInfo ShortestDayNames ShortestDayNames MonthGenitiveNames MonthGenitiveNames AbbreviatedMonthGenitiveNames AbbreviatedMonthGenitiveNames NumberFormatInfo NumberFormatInfo NativeDigits NativeDigits DigitSubstitution DigitSubstitution CultureInfo CultureInfo IsCustomCulture IsCustomCulture IetfLanguageTag IetfLanguageTag CultureTypes CultureTypes GetCultureInfo() GetCultureInfo() GetCultureInfoByIetfLanguageTag() GetCultureInfoByIetfLanguageTag() RegionInfo RegionInfo GeoId GeoId NativeName NativeName CurrencyEnglishName CurrencyEnglishName (Can now create via full culture names) (Can now create via full culture names)

April 25, 2005 Updates to encodings Now built into the BCL Now built into the BCL Improved performance Improved performance more flexibility more flexibility consistent results across supported platforms consistent results across supported platforms Encoding enumeration API Encoding enumeration API UTF-32 support (little endian and big endian) UTF-32 support (little endian and big endian) UTF-16 big endian support UTF-16 big endian support Encoding/decoding fallbacks Encoding/decoding fallbacks Exception Exception Replacement Replacement Best fit Best fit Custom Custom

April 25, 2005 public class NumericEntitiesFallback : EncoderFallback { public class NumericEntitiesFallback : EncoderFallback { public override EncoderFallbackBuffer CreateFallbackBuffer() { public override EncoderFallbackBuffer CreateFallbackBuffer() { return new NEFallbackBuffer(); return new NEFallbackBuffer(); } public override int MaxCharCount { public override int MaxCharCount { get { get { return 8; return 8; } } } public class NEFallbackBuffer : EncoderFallbackBuffer { public class NEFallbackBuffer : EncoderFallbackBuffer { // Store our default string // Store our default string private String strEntity; private String strEntity; int fallbackCount = -1; int fallbackCount = -1; int fallbackIndex = 0; int fallbackIndex = 0; // Fallback Methods // Fallback Methods public override bool Fallback(char charUnknown, int index) { public override bool Fallback(char charUnknown, int index) { // If we had a buffer already we're being recursive, throw, // If we had a buffer already we're being recursive, throw, // it's probably at the suspect character in our array. // it's probably at the suspect character in our array. if (fallbackCount >= 0) if (fallbackCount >= 0) ThrowLastCharRecursive(unchecked((int)charUnknown)); ThrowLastCharRecursive(unchecked((int)charUnknown)); // Go ahead and get our fallback // Go ahead and get our fallback strEntity = String.Format("&#{0};", (int)charUnknown); strEntity = String.Format("&#{0};", (int)charUnknown); fallbackCount = strEntity.Length; fallbackCount = strEntity.Length; fallbackIndex = 0; fallbackIndex = 0; return fallbackCount != 0; return fallbackCount != 0; } public override bool Fallback(char charUnknownHigh, char charUnknownLow, int index) { public override bool Fallback(char charUnknownHigh, char charUnknownLow, int index) { // Double check input surrogate pair // Double check input surrogate pair if (!Char.IsHighSurrogate(charUnknownHigh)) if (!Char.IsHighSurrogate(charUnknownHigh)) throw new ArgumentOutOfRangeException("charUnknownHigh", throw new ArgumentOutOfRangeException("charUnknownHigh", supposed to be between 0xD800 and 0xDBFF"); supposed to be between 0xD800 and 0xDBFF"); if (!Char.IsLowSurrogate(charUnknownLow)) if (!Char.IsLowSurrogate(charUnknownLow)) throw new ArgumentOutOfRangeException("CharUnknownLow", throw new ArgumentOutOfRangeException("CharUnknownLow", supposed to be between 0xD800 and 0xDBFF"); supposed to be between 0xD800 and 0xDBFF"); // If we had a buffer already we're being recursive, throw, it's // If we had a buffer already we're being recursive, throw, it's // probably at the suspect character in our array. // probably at the suspect character in our array. if (fallbackCount >= 0) if (fallbackCount >= 0) ThrowLastCharRecursive(Char.ConvertToUtf32(charUnknownHigh, charUnknownLow)); ThrowLastCharRecursive(Char.ConvertToUtf32(charUnknownHigh, charUnknownLow)); // Go ahead and get our fallback // Go ahead and get our fallback strEntity = String.Format("&#{0};", Char.ConvertToUtf32(charUnknownHigh, charUnknownLow)); strEntity = String.Format("&#{0};", Char.ConvertToUtf32(charUnknownHigh, charUnknownLow)); fallbackCount = strEntity.Length; fallbackCount = strEntity.Length; fallbackIndex = 0; fallbackIndex = 0; return fallbackCount != 0; return fallbackCount != 0; } public override char GetNextChar() { public override char GetNextChar() { // We want it to get < 0 because == 0 means that the current/last // We want it to get < 0 because == 0 means that the current/last // character is a fallback and we need to detect recursion. We // character is a fallback and we need to detect recursion. We // could have a flag but we already have this counter. // could have a flag but we already have this counter. fallbackCount--; fallbackCount--; // Do we have anything left? 0 is now last fallback char, negative // Do we have anything left? 0 is now last fallback char, negative // is nothing left // is nothing left if (fallbackCount < 0) if (fallbackCount < 0) return (char)0; return (char)0; // Need to get it out of the buffer. // Need to get it out of the buffer. return strEntity[fallbackIndex++]; return strEntity[fallbackIndex++];} public override bool MovePrevious() { fallbackCount++; fallbackIndex--; fallbackCount++; fallbackIndex--; return true; return true;} public override int Remaining { get { get { return (fallbackCount < 0) ? 0 : fallbackCount; return (fallbackCount < 0) ? 0 : fallbackCount; }} // private helper methods private void ThrowLastCharRecursive(int charRecursive) { // Throw it, using our complete character // Throw it, using our complete character throw new ArgumentException( throw new ArgumentException( String.Format("Last character \\u{0:4X} was a recursive fallback", String.Format("Last character \\u{0:4X} was a recursive fallback", charRecursive), "chars"); charRecursive), "chars"); } }

April 25, 2005 Collation Improvements OrdinalIgnoreCase OrdinalIgnoreCase Same results as ToUpper/Ordinal Same results as ToUpper/Ordinal Matches OS file system results Matches OS file system results Correct Serbian collation Correct Serbian collation Fixed in Windows XPSP2 Fixed in Windows XPSP2 Customer reported (MSDN Feedback Center) Customer reported (MSDN Feedback Center) Better handling of ignored/ignorable characters Better handling of ignored/ignorable characters IndexOf/LastIndexOf/IsPrefix/IsSuffix IndexOf/LastIndexOf/IsPrefix/IsSuffix StartsWith/EndsWith, too StartsWith/EndsWith, too

April 25, 2005 OrdinalIgnoreCase sample string strTest1 = "IamAString"; string strTest2 = "STRING"; if(strTest1.EndsWith(strTest2, StringComparison.OrdinalIgnoreCase)) { Console.WriteLine(Successful test!); Console.WriteLine(Successful test!);};

April 25, 2005 Unicode normalization Described in UAX#15 at Described in UAX#15 at String.IsNormalized() String.IsNormalized(NormalizationForm normalizationForm) String.IsNormalized() String.IsNormalized(NormalizationForm normalizationForm) String.Normalize() String.Normalize(NormalizationForm normalizationForm) String.Normalize() String.Normalize(NormalizationForm normalizationForm) NormalizationForm enumeration NormalizationForm enumeration FormC, FormD, FormKC, FormKD FormC, FormD, FormKC, FormKD (U+00f5 U+0068 U+0302 U+00b5 U+00a8) LATIN SMALL LETTER O WITH TILDE; LATIN SMALL LETTER H; COMBINING CIRCUMFLEX ACCENT; MICRO SIGN; DIAERESIS õĥµ¨ (U+00f5 U+0068 U+0302 U+00b5 U+00a8) LATIN SMALL LETTER O WITH TILDE; LATIN SMALL LETTER H; COMBINING CIRCUMFLEX ACCENT; MICRO SIGN; DIAERESIS FormC: (U+00f5 U+0125 U+00b5 U+00a8) FormC: õĥµ¨ (U+00f5 U+0125 U+00b5 U+00a8) FormD: (U+006f U+0303 U+0068 U+0302 U+00b5 U+00a8) FormD: õĥµ¨ (U+006f U+0303 U+0068 U+0302 U+00b5 U+00a8) FormKC: (U+00f5 U+0125 U+03bc U+0020 U+0308) FormKC: õĥμ ̈ (U+00f5 U+0125 U+03bc U+0020 U+0308) FormKD: (U+006f U+0303 U+0068 U+0302 U+03bc U+0020 U+0308) FormKD: õĥμ ̈ (U+006f U+0303 U+0068 U+0302 U+03bc U+0020 U+0308) In collation, õĥµ¨ õĥµ¨ õĥμ ̈ õĥμ ̈

April 25, 2005 namespace àáâãäå { using System; using System.Text; using System.Globalization; class àáâãäa ̊ class àáâãäa ̊ { [STAThread] [STAThread] static void Main(string[] args) { static void Main(string[] args) { àáâãäå(); àáa ̂ ãa ̈ a ̊ (); àáâãäå(); àáâãäå(); àáa ̂ ãäå(); àáa ̂ ãäå(); àáa ̂ ãa ̈ å(); àáâãäå(); àáa ̂ ãa ̈ a ̊ (); àáâãäå(); àáâãäå(); àáa ̂ ãäå(); àáa ̂ ãäå(); àáa ̂ ãa ̈ å(); } static void àáa ̂ ãäå(string àáâãa ̈ a ̊ ) { static void àáa ̂ ãäå(string àáâãa ̈ a ̊ ) { StringBuilder àáâãäa ̊ = new StringBuilder(); StringBuilder àáâãäa ̊ = new StringBuilder(); StringInfo àáâãäå = new StringInfo(àáâãa ̈ a ̊ ); StringInfo àáâãäå = new StringInfo(àáâãa ̈ a ̊ ); àáâãäa ̊.Append(àáâãa ̈ a ̊.Normalize(NormalizationForm.FormC)); àáâãäa ̊.Append(àáâãa ̈ a ̊.Normalize(NormalizationForm.FormC)); àáâãäa ̊.Append(": "); àáâãäa ̊.Append(": "); for(int àáâãa ̈ å=0; àáâãa ̈ å < àáâãäå.LengthInTextElements; àáâãa ̈ å++) { for(int àáâãa ̈ å=0; àáâãa ̈ å < àáâãäå.LengthInTextElements; àáâãa ̈ å++) { string àáa ̂ ãa ̈ å = àáâãäå.SubstringByTextElements(àáâãa ̈ å, 1); string àáa ̂ ãa ̈ å = àáâãäå.SubstringByTextElements(àáâãa ̈ å, 1); if(àáa ̂ ãa ̈ å.IsNormalized(NormalizationForm.FormC)) { if(àáa ̂ ãa ̈ å.IsNormalized(NormalizationForm.FormC)) { àáâãäa ̊.Append("C"); àáâãäa ̊.Append("C"); } else if(àáa ̂ ãa ̈ å.IsNormalized(NormalizationForm.FormD)) { } else if(àáa ̂ ãa ̈ å.IsNormalized(NormalizationForm.FormD)) { àáâãäa ̊.Append("D"); àáâãäa ̊.Append("D"); } else { } else { àáâãäa ̊.Append("_"); àáâãäa ̊.Append("_"); } } Console.WriteLine(àáâãäa ̊.ToString()); Console.WriteLine(àáâãäa ̊.ToString()); return; return; } static void àáâãäå() { static void àáâãäå() { àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); } static void àáa ̂ ãa ̈ a ̊ () { static void àáa ̂ ãa ̈ a ̊ () { àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãa ̈ a ̊ "); àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãa ̈ a ̊ "); } static void àáâãäå() { static void àáâãäå() { àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); } static void àáâãäå() { static void àáâãäå() { àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); àáâãäa ̊.àáa ̂ ãäå("àáâãäå"); } static void àáa ̂ ãäå() { static void àáa ̂ ãäå() { àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãäå"); àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãäå"); } static void àáa ̂ ãäå() { static void àáa ̂ ãäå() { àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãäå"); àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãäå"); } static void àáa ̂ ãa ̈ å() { static void àáa ̂ ãa ̈ å() { àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãa ̈ å"); àáâãäa ̊.àáa ̂ ãäå("àáa ̂ ãa ̈ å"); } }}

April 25, 2005 IDN Mapping APIs IdnMapping class IdnMapping class Based on three RFCs (standard based on Unicode 3.2) Based on three RFCs (standard based on Unicode 3.2) Internationalizing Domain Names in Applications (IDNA) Internationalizing Domain Names in Applications (IDNA) Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN) Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN) Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA) Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA) \u5B89\u5BA4\u5948\u7F8E\u6075-with-SUPER-MONKEYS becomes xn---with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n \u5B89\u5BA4\u5948\u7F8E\u6075-with-SUPER-MONKEYS becomes xn---with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n Properties Properties AllowUnassigned (allows new Unicode characters) AllowUnassigned (allows new Unicode characters) UseStd3AsciiRules (more like DNS rules) UseStd3AsciiRules (more like DNS rules) Methods Methods GetAscii - Gets ASCII (Punycode) version of the string GetAscii - Gets ASCII (Punycode) version of the string GetUnicode - Gets Unicode version of the string, normalized and limited to IDNA characters. GetUnicode - Gets Unicode version of the string, normalized and limited to IDNA characters.

April 25, 2005 Unicode property information New CharUnicodeInfo class New CharUnicodeInfo class Extends methods on Char Extends methods on Char Offical data from the Unicode Character Database at Offical data from the Unicode Character Database at IsWhiteSpace IsWhiteSpace GetNumericValue GetNumericValue GetDigitValue GetDigitValue GetDecimalDigitValue GetDecimalDigitValue GetUnicodeCategory GetUnicodeCategory GetBidiCategory GetBidiCategory

April 25, 2005 New text element support in the StringInfo class StringInfo ctor that takes a string StringInfo ctor that takes a string StringInfo.String StringInfo.String StringInfo.LengthInTextElements StringInfo.LengthInTextElements StringInfo.SubstringByTextElements() StringInfo.SubstringByTextElements() Both use ParseCombiningCharacters() to get their results Both use ParseCombiningCharacters() to get their results

April 25, 2005 New StringInfo props/methods sample StringInfo si = New StringInfo("A\u0300\u0301\u0300e\u0300\u0301\u0300); Console.WriteLine(si.LengthInTextElements); // Length is two for(int ich = 0; ich < si.LengthInTextElements; ich++) { Console.WriteLine(si.SubstringByTextElements(ich, 1); Console.WriteLine(si.SubstringByTextElements(ich, 1);}

April 25, 2005 New supplementary character support in lots of methods New signature -- (String s, int index) New signature -- (String s, int index) IsControl, IsDigit, IsLetter, IsLetterOrDigit, IsLower, IsNumber, IsPunctuation, IsSeparator, IsSurrogate, IsSymbol, IsUpper, IsWhiteSpace, GetUnicodeCategory, GetNumericValue, IsHighSurrogate, IsLowSurrogate, IsSurrogatePair IsControl, IsDigit, IsLetter, IsLetterOrDigit, IsLower, IsNumber, IsPunctuation, IsSeparator, IsSurrogate, IsSymbol, IsUpper, IsWhiteSpace, GetUnicodeCategory, GetNumericValue, IsHighSurrogate, IsLowSurrogate, IsSurrogatePair ConvertToUtf32, ConvertFromUtf32 methods ConvertToUtf32, ConvertFromUtf32 methods

April 25, 2005 References MSDN Magazine Article MSDN Magazine Article Make the.NET World a Friendlier Place with the Many Faces of the CultureInfo Class March Make the.NET World a Friendlier Place with the Many Faces of the CultureInfo Class March SQL Server Books Online SQL Server Books Online International Considerations for SQL Server a8-a8ef-bc134502b4e0.asp International Considerations for SQL Server a8-a8ef-bc134502b4e0.asp a8-a8ef-bc134502b4e0.asp a8-a8ef-bc134502b4e0.asp My Blog My Blog Some other blogs for intl support in Whidbey Some other blogs for intl support in Whidbey Other useful sites Other useful sites

April 25, 2005 Globalization Features in Whidbeys CLR Questions