Ahead of Time Dynamic Translation PreJit/NGEN by any other name George Bosworth Microsoft MRE04 March 21, 2004

2 Dynamic Translation What is it? – a compiler (not) What does it do? – produce good fast code (hopefully) When does it run? – Before the code is executed. i.e ahead of time. How much before is the subject of this talk

3 Outline Code Generation and execution in the Common Language Runtime (CLR) PreJit –What is it, why do it, at what cost –What is it really, and what is hard vs easy Simple example –What the resultant code could look like

4 Common Language Runtime Frameworks Class loader and layout IL to native code compilers GC, stack walk, code manager Security Execution Support Base Classes

5 Runtime Control Flow Just in Time Class Loader IL to native code compiler CPU Security System Code Managers Managed Native Code Assembly First call to method First reference to type Execution Support

6 Execution Model VBVC...Script IL Native Code Econo-JIT Compiler Standard JIT Compiler Native Code PreJit time Code Gen Common Language Runtime

7 Compiling IL To Native Econo JIT –Generates unoptimized native code –Code can be discarded and regenerated Standard JIT –Generates optimized native code –Includes verification of IL code PreJit code generation –Done at install time –Reduces start-up time –Native code has version checks and reverts to runtime JIT if they fail

8 Assemblies Unit of deployment –One or more files, independent of packaging –Self-describing via metadata (manifest) Versioning –Captured by compiler –Policy per-application as well as per-machine Security boundary –Assemblies are granted permissions –Methods can demand proof that a permission has been granted to entire call chain Mediate type import and export –Types named relative to assembly

9 What Happens at Compile Time? Compiler reads any imported metadata and builds internal symbol table Compiler resolves method overloading using language- specific type matching rules –Compiler adds any required coercions or casts –Compiler selects and records a precise member reference Compiler provides object layout information through its choice of –Layout technique –Non-static fields –Virtual methods, with new slot vs use existing slot Compiler emits metadata by merging incoming metadata with metadata it has generated Compiler emits IL with metadata tokens

10 What Happens at Class Load Time?, I A type reference occurs and is resolved either to an existing type or to an assembly reference. The assembly is loaded and some validation checks occur. The assembly is asked to resolve the type reference and a module is located –It is at this point that the security system runs inheritance permission checks

11 What Happens at Class Load Time?, II If no managed native code is available, the IL module is loaded and preliminary validation checks occur. The particular class is then validated and the CLR creates in-memory data structures representing the class, including thunks for use when a method is called the first time. When a method on the class is called the first time, the code is JITted (and verified if needed).

12 What Happens at JIT Time? If a member reference appears in the code, the type is first loaded (see below), which will also load classes from which it inherits. If the type does not itself have the member, the inheritance chain is searched to see if it can be resolved higher up in the class hierarchy. The member access check, including a possible security check, occurs at this time. The JITter may insert calls to security code for declarative security checks in addition to those specified using the imperative security mechanism. The code runs.

13 Unmanaged (Traditional) CodeDEPLOYMENT app. directory, Win32 directory public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } CompilerDEVELOPMENT public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } Source code PE header + EH Table + Native code (incl. object layout) EXECUTION Native code EH table Services Exception Exception

14 Managed Code Execution PEVerify NGENDEPLOYMENT GAC, app. directory, download cache public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } CompilerDEVELOPMENT public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } Source code Assembly PE header + MSIL + Metadata + EH Table EXECUTION Assembly info Module + Class list Policy Manager Host Policy <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissio ns and forms the root of the code group tree."> <IMembershipCondition clas s="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Class Loader Granted permissions Vtable + Class info JIT + verification Native code + GC table CLR Services GC GC Exception Exception Class init Class init Security Security (assembly) (class) (method) Assembly Loader Evidence Permission request

15 PreJit: What Is It? An MSIL-to-Native Compiler (Not) Pre-compile, pre-load, pre-layout –Validate at execution time May factor in processor type, other installed assemblies, etc. –If invalid, use the normal execution path Goal: faster start-up –Because of smaller start-up working set Usually as part of assembly installation ngen.exe, ships in Visual Studio ®.Net

16 What Happens at PreJit Time? Record version information about any dependents Classes are laid out Internal runtime data structures allocated Generate managed native code, verifying as needed Emit a specially formed PE file (.exe,.dll) in the code cache and associate it with the input IL assembly

17 NGen Code Creation PEVerify NGEN DEPLOYMENT GAC, app. directory, download cache public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } CompilerDEVELOPMENT public static void Main(String[] args ) { String usr; FileStream f; StreamWriter w; try { usr=Environment.GetEnvironmentVariable("USERNAME"); f=new FileStream(C:\\test.txt",FileMode.Create); w=new StreamWriter(f); w.WriteLine(usr); w.Close(); } catch (Exception e){ Console.WriteLine("Exception:"+e.ToString()); } Source code Assembly PE header + MSIL + Metadata + EH Table EXECUTION Assembly info Module + Class list Policy Manager Host Policy <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissio ns and forms the root of the code group tree."> <IMembershipCondition clas s="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Class Loader Granted permissions Vtable + Class info JIT + verification Native code + GC table CLR Services GC GC Exception Exception Class init Class init Security Security (assembly) (class) (method) Assembly Loader Evidence Permission request

18 EXECUTION Assembly info Module + Class list NGen Code Execution DEPLOYMENT GAC, app. directory, download cache Prebuilt(checkVersion) Policy Manager Host Policy <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="Nothing" Name="All_Code" Description="Code group grants no permissio ns and forms the root of the code group tree."> <IMembershipCondition clas s="AllMembershipCondition" version="1"/> <CodeGroup class="UnionCodeGroup" version="1" PermissionSetName="FullTrust" Class Loader Vtable + Class info JIT + verification Native code + GC table CLR Services GC GC Exception Exception Class init Class init Security Security (assembly) (class) (method) Assembly Loader Permission request Granted permissions Precomputed permissions (check) Prebuilt Vtable + Class info Fixups / Relocs (or JIT)

19 Deployment Model Shipping and installation –Compile IL in build lab –Ship IL –PreJit during installation (optional) –Possible JIT during execution Patching –Deliver new IL image –Repair native images (all depending assemblies)

20 Why Add Complexity of PreJit? Performance –Remove expensive post-deployment work from app startup Signature validation Cross-component validity checks Type safety verification Optimization (distribution -> execution) –Code generation –Data structures –Packing –Disk-based images allow sharing data/code across processes and across machine runs

21 Performance Gains Warm startup time –Up to 6x faster Private pages –Reduced by as much as 2/3 Startup working set –Reduced by as much as 1/2 Basis –Whidbey (the upcoming release of the CLR) –Reference WinForm app with 100s of controls Caveats: your mileage may very and these are very preliminary based on pre-release builds and hence subject to change

22 Costs Disk space –Typical prejit image is ~2-2.5x IL size –Stored in Native Image Cache (alongside GAC) Installation time –Typically ~3s per MB of IL –40s to prejit Framework assemblies Steady state performance –Slowdown due to indirections in code –ASP.net V1: PreJit 7-12% slower Whidbey: PreJit 2-5% slower –Bytemark Whidbey: same performance –Future: PreJit will have better performance Recompilation

23 Scenarios Critical: libraries –Loaded into multiple processes Warm startup is the norm Maximize shared pages –Used by OS like components Important: large apps –Large percentage of code is app code –Large impact on machine –Benefit is primarily startup time Important: shared apps –Terminal server –Shared pages are key

24 Scenarios Questionable: small single instance apps and addins –~1MB working set improvement –~15% startup improvement –Relatively small amount of compilation overhead –Likely to be numerous –Recommendation: do not ngen unless heavily used Not helpful: transient code –Reflection emit –Run and (probably) throw away code –Frequently updated apps (relative to execution) –Code under development (edit-compile-debug) Assumption: some processes will be jitting some assemblies

25 PreJit Architectural Background IL was designed as CLR deployment format –Verifiable type safety –Small –Self describing –Portable PreJit image was designed as internal format –Designed for execution speed Working set rather than disk size Data structures paged off of disk & used directly –Highly dependent on CLR implementation Data structure snapshots Complex codegen requirements High rate of churn over time –Specialized for particular scenario

26 Specialization and Recompilation PreJit images are specialized translations of an IL assembly for a particular environment –Processor –Version of CLR –Versions of assembly dependencies –Development scenario (debug, profile, IBC instrument) –Other factors Processor features Security policy Hosting hooks (SQL, IE,???) AppDomain shareable (state access) Profiling data Different environments require different prejit images –Different machines may have different environments –Servicing can change CLR or assembly dependencies –Environmental changes can invalidate prejit images –App-level configuration can change requirements across apps Recompilation is needed to adjust available prejit images over time to adapt to a changing environment

27 Client code generation motivation IL Distribution –Verifiable (limited trust) –Small –Self describing Specialized native code –Reach Processor CLR Implementation –CLR Version –Platform (assemblies) Version Recompilation Brittleness okay Rebuilt as needed

28 Why is specialization important? Innovation –Managed APIs and implementation –CLR implementation Compatibility –Application needs to deploy on multiple OSs –OS update must retain application compatibility Plus performance (of course)

29 Framework evolution & recompilation We want to be able to recompile application native code as frameworks (platform assemblies) evolve Flexible compatible update rules –Add public stuff –Change any non-public stuff –Rules based on semantics – no implementation driven constraints Rearrange and add fields to public structures & classes Rearrange and add virtual methods Insert into inheritance heirarchy –Non-compatible changes are possible and handled relatively gracefully Versionability is a fundamental assumption when designing managed APIs –Implementation inheritance Each layer up the hierarchy may add fields, methods, virtuals All are rolled into single flat structure All can version independently from each other and from client –Typesafe extensibility Public fields on a structure or class Set can be extended as needed Example: delegates on form, properties on window –Fine grained, version-resilient component interaction Properties Can protect access and future evolution with a function call but still get efficiency –Raises the bar for API quality Uniform mechanisms with language support Patterns can be enforced with tools

30 CLR evolution & recompilation We want to be able to recompile native code as CLR evolves Much of runtime implementation is baked into code –Code sequences Helper calls, Dispatch sequences, Global variable access –Runtime data structures Highly specialized, Slow to build from IL, Likely to change over time –Invariants Garbage collection, Lazy initialization Recent Changes –Interface/Virtual dispatch –Working set tuning –Single file loading –Generics –And on and on… Future potential –More data structure tuning –Phoenix The CLR is a great vehicle for delivering improvements into Windows

31 Future Directions Phoenix –Changes prejit equation Higher compile cost Better perf for prejit vs JIT –More cross-assembly analysis Client compilation optimizations –Profile data feedback –Active base address management Enable some additional specialization conditionally –Allow developer to dial conservative behavior vs. aggressive optimizations

32 Why its Not So Easy let me count the ways

33 Issue: Compilation Overhead Installation cost Recompilation cost –Cascaded low level changes have very high cost Image churn can have secondary costs (OS optimizations, virus scan) Goals: –Ensure compilation performance is well tuned –Minimize recompilation Minimize factors which cause recompilation Allow limited assembly servicing without cascading recompilation –Ensure that most low-level servicing falls in this limited category –Be smart about deferring compilation work –However, some compilation cost is fundamental

34 Issue: Managed Watson Dumps Native code and CLR data structures are generated on user machine, not available in any shipped dlls Much of this data is required to do a stack trace IL Native mapping info is typically not available unless debug code is in use –Source mapping is described in terms of IL Goals: –Leverage repeatability of codegen to optimize ngen image contents out of process dump Result is faster dump collection and smaller dumps –Also leverage repeatability to regenerate IL native mapping info post mortem

35 Cross-assembly optimizations These optimizations expose internal implementation details to other assemblies code generation Examples –Method inlining –Constant value inlining –Persisted instance data created at compile time –Class constructors –Struct enregistration Likely to be much more in Phoenix

36 NGen Fixup Blobs Descriptive reference to something in another module Blobs are grouped by referenced module? –Module level table describes ranges of blobs –Each table entry has an assembly ref token + file ref token Could roughly follow metadata signature byte encoding? –Byte sized tags with fixed following values –Variable length encoded numbers & tokens Treat blob is a type signature? –However, scope is in destination assembly –This avoids symbolic lookups, however introduces fragility if token numbers change Other possible blob formats –MODULE (no args) –TYPE + type signature –METHOD + + containing type signature –FIELD + + containing type signature –STRING + –SIGNATURE + Blobs are read-only? & interned within a module? Typically described with an RVA? Details

37 NGen Fixup Tables Set of tables which provide fixup cells for code and data to reference Header describes ranges of tables? Table entry initially contains tagged RVA of fixup blob? Table index implies type of fixup –Handle (runtime data structure) –Method call address –Static field address –Varargs signature –Others? –Side effects? Which ones? (e.g. type initializations?) Entries are fixed up lazily? –All fixups are idempotent?, tag allows atomic update –Eager update is too expensive? –Fixup can fail? And they never end

38 But when it works it can work well An example (take with a grain of salt)

39 Example Assembly A (a.cs) public class A { static public int a() {... } } Assembly B (b.cs) public class InheritFromA : A { static public int a1() {... } } public class B { public static void b() { A a = new A(); A.a(); InheritFromA.a1(); }

40 Example: Naive b() has 4 fixups & requires fixup guarantee before execution { A a = new A(); mov ecx,[ ] // Fixup on A's method table pointer call // Call to new d mov ecx,eax f mov eax,[ ] // Fixup on A.ctor call dword ptr [eax] // Call to A.ctor A.a(); mov eax,[ ] // fixup on A.a d call dword ptr [eax] // Call to A.a InheritFromA.a1(); f call dword ptr [20005cb8] // Indirect call to InheritFromA.a1 // (InheritFromA may need one time init) } ret

41 Example: In the Limit b() has no fixups and can be called directly { A a = new A(); a mov ecx,0x6ab95b98 // Note no fixup on A's method table pointer f call // call to new mov ecx,eax call a!A..ctor() // Note direct call to A.ctor A.a(); b call a!A.a() // Note direct call to A.a InheritFromA.a1(); call b!InheritFromA.a1() // Note direct call to // InheritFromA.a1 (InheritFromA // is pre-inited) } ret

42 In Conclusion Ahead of Time Dynamic Translation (PreJit) What is it? – a compiler (not) What does it do? – produce good fast code (hopefully) When does it run? – Before the code is executed. i.e ahead of time, (just a bit earlier than a JIT).

43 Bottom Line All that is left is compiler style optimizations Prejit makes sense in some scenarios, but not all