Presentation is loading. Please wait.

Presentation is loading. Please wait.

Scala Next SF Scala meetup Dec 8 th, 2011. 2 Scala Today.

Similar presentations

Presentation on theme: "Scala Next SF Scala meetup Dec 8 th, 2011. 2 Scala Today."— Presentation transcript:

1 Scala Next SF Scala meetup Dec 8 th, 2011

2 2 Scala Today

3 3 Some adoption vectors: Web platforms Trading platforms Financial modeling Simulation Fast to first product, scalable afterwards

4 Github vs. Stack Overflow RedMonk: Revisiting the Dataists Programming Language Rankings 4 Typesafe Confidential

5 Commercial Adoption Scala jobs tripled in last year Now at estimated 100,000 developers 5 Typesafe Confidential

6 6 Scala 2.8: (Only 17 months ago!) New collections Package objects Context bounds Better implicits...

7 7 Scala 2.9: Parallel collections DelayedInit and App Faster REPL Progress on IDEs: Eclipse, IntelliJ, Neatbeans, ENSIME Better docs Lots of bug fixes

8 Parallel Collections Use Java 7 Fork Join framework Split work by number of Processors Each Thread has a work queue that is split exponentially. Largest on end of queue Granularity balance against scheduling overhead On completion threads work steals from end of other thread queues 8

9 9... and its usage import java.util.ArrayList;... Person[] people; Person[] minors; Person[] adults; { ArrayList minorsList = new ArrayList (); ArrayList adultsList = new ArrayList (); for (int i = 0; i < people.length; i++) (people[i].age < 18 ? minorsList : adultsList).add(people[i]); minors = minorsList.toArray(people); adults = adultsList.toArray(people); }... in Java:... in Scala: val people: Array[Person] val (minors, adults) = people partition (_.age < 18) A simple pattern match An infix method call A function value

10 10 Going Parallel ?... in Java:... in Scala: val people: Array[Person] val (minors, adults) = people.par partition (_.age < 18)

11 General Collection Hierarchy GenTraversable GenIterable GenSeq Traversable Iterable Seq ParIterable ParSeq 11 Remove this layer in 2.10?

12 Going Distributed Can we get the power of parallel collections to work on 10000s of computers? Hot technologies: MapReduce (Googles and Hadoop) But not everything is easy to fit into that mold Sometimes 100s of map-reduce steps are needed. Distributed collections retain most operations, provide a powerful frontend for MapReduce computations. Scalas uniform collection model is designed to also accommodate parallel and distributed. Projects at Google (Cascade), Berkeley (Spark), EPFL. 12

13 13 Scala next: Eclipse IDE Play web framework 2.0 Akka 2.0 Scala 2.10

14 14 Scala Eclipse IDE Now in RC2 Final expected before the end of the year.

15 Goals reliable (no crashes/lock ups) responsive (never wait when typing) work with large projects/files –Scala compiler (80k LOC), LOC/file –advanced use of the type system: path-dependent types, self-types, mix-ins

16 Features Keep it simple –highlight errors as you type –completions (including implicits) –hyperlinking –project builder (+ dependent projects) Support mixed Java-Scala projects –all features should work between Java/Scala sources JUnit Test Runner should pick up tests More stuff based on external libraries –(some) refactoring, code formatter, mark occurrences, structured selections, show inferred semi-colons

17 Features (3) based on external libraries –(some) refactoring –code formatter –mark occurrences –structured selections –show inferred semi-colons

18 @jonifreeman Joni Freeman Latest Scala Eclipse plugin works surprisingly well! Even manages our mixed Java/Scala project. Kudos to the team! Eduardo Sorribas The latest beta of the Scala IDE for eclipse is much better. I'm starting to like Janne Hietamäki After years of misery, the Eclipse Scala plugin actually seems to work quite well.

19 Architecture Use the full-blown Scala compiler for: –interactive error highlight, completion, hyperlinking –turning Scala symbols into Java model elements Weave the JDT compiler when it needs help –JDT was NOT meant to be extended

20 Why rely on scalac? –reuse (type-checker == 1-2 person years) –consistency –compiler plugins Why not? –SPEED –(very) tight dependency on the Scala version

21 Presentation Compiler asynchronous interruptible targeted stop after type-checking



24 Result is communicated through a SyncVar

25 All compiler activity happens on PC thread compile loaded files when work queue is empty (in the background) Check work queue when type checker reaches safe-points in the AST Drop everything when a file is changed (AskReload)

26 26 Implementation

27 1 type-checker run / instance --> 100s of type-check runs / minute –memory leaks –side-effects/state –out-of-order and targeted type-checking needed to improve the compiler –2.9.x, 2.10 (trunk) –what about 2.8? 2.8.2, SNAPSHOT

28 New: Play Framework 2.0 Play Framework is an open source web application framework, inspired by Ruby on Rails, for Java and Scala Play Framework 2.0 retains full Java support while moving to a Scala core and builds on key pieces of the Typesafe Stack, including Akka middleware and SBT Play will be integrated in TypeSafe stack 2.0 Typesafe will contribute to development and provide commercial support and maintenance.

29 Roadmap Typesafe Stack 1.0 Typesafe Stack 1.1 Typesafe Stack 2.0 Typesafe Stack 2.x May 2011Oct 2011Q1 2012Q Scala Akka 1.1 Scala Akka 1.2 Scala 2.9.x Akka 2.0 Play 2.0 Scala 2.10 Akka 2.x Play 2.x Slick (DB)

30 30 Scala 2.10: 1.New reflection framework 2.Reification 3.type Dynamic 4.More IDE improvements: find- references, debugger, worksheet. 5.Faster builds 6.SIPs: string interpolation, simpler implicits. ETA: Early 2012.

31 New in Scala 2.10: Dynamic Type Dynamic bridges the gap between static and dynamic typing. Method calls get translated to applyDynamic Great for interfacing with dynamic languages (e.g. JavaScript) 31 class JS extends Dynamic { def applyDynamic(methName: String, args: Any*): Any = { println("apply dynamic "+methName+args.mkString("(", ",", ")")) } val x = new JS // x.applyDynamic(foo, 1) // x.applyDynamic(bar)

32 Proposed for Scala 2.10: SIP 11: String interpolation Idea: Instead of Bob is + n + years old write: sBob is $n years old which gets translated to new StringContext(Bob is, years old).s(n) Here, s is a library-defined method for string interpolation. 32

33 This can be generalized to other string processors besides s : xml ${linktext} scala scala.concurrent.transaction.withinTransaction { (implicit currentTransaction: Transaction) => $expr } 33

34 Proposed for Scala 2.10: SIP 12: Uncluttering control Should be able to write: if x < 0 then –x else x while x > 0 do { println(x); x -= 1 } for x <- xs do println(x) for x <- xs yield x * x 34

35 Proposed for Scala 2.10: SIP 13: Implicit classes Variation: to class def to get speed of extension methods. 35

36 New in Scala 2.10: Reflection Previously: Needed to use Java reflection, no runtime info available on Scalas types. Now you can do: 36

37 (Bare-Bones) Reflection in Java 37 Want to know whether type A conforms to B? Write your own Java compiler! Why not add some meaningful operations? Need to write essential parts of a compiler (hard). Need to ensure that both compilers agree (almost impossible).

38 How to do Better? Problem is managing dependencies between compiler and reflection. Time to look at DI again. 38 Dependency Injection Idea: Avoid hard dependencies to specific classes. Instead of calling specific classes with new, have someone else do the wiring.

39 Using Guice for Dependency Injection 39 (Example by Jan Kriesten)

40 ... plus some Boilerplate 40

41 Dependency Injection in Scala 41 Components are classes or traits Requirements are abstract values Wiring by implementing requirement values But what about cyclic dependencies?

42 The Cake Pattern 42 Requirements are types of this Components are traits Wiring by mixin composition

43 Cake Pattern in the Compiler The Scala compiler uses the cake pattern for everything Heres a schema: (In reality there are about ~20 slices in the cake.) 43

44 Towards Better Reflection Can we unify the core parts of the compiler and reflection? Compiler Reflection Different requirements: Error diagnostics, file access, classpath handling - but we are close! 44

45 Compiler Architecture 45 reflect.internal.Universe nsc.Global (scalac) reflect.runtime.Mirror Problem: This exposes way too much detail!

46 Complete Reflection Architecture Cleaned-up facade: Full implementation: 46 reflect.internal.Universe nsc.Global (scalac) reflect.runtime.Mirror reflect.api.Universe / reflect.mirror

47 How to Make a Facade 47 The Facade The Implementation Interfaces are not enough!

48 Conclusion Scala is a very regular language when it comes to composition: 1.Everything can be nested: –classes, methods, objects, types 2.Everything can be abstract: –methods, values, types 3.The type of this can be declared freely, can thus express dependencies 4.This gives great flexibility for SW architecture, allows us to attack previously unsolvable problems. 48

49 Going further: Parallel DSLs Mid term, research project: How do we keep tomorrows computers loaded? –How to find and deal with threads in an application? –Parallel collections and actors are necessary but not sufficient for this. Our bet for the mid term future: parallel embedded DSLs. –Find parallelism in domains: physics simulation, machine learning, statistics,... Joint work with Kunle Olukuton, Pat Stanford. EPFL side funded by ERC. 49

50 EPFL / Stanford Research Domain Embedding Language (Scala) Virtual Worlds Personal Robotics Data informatics Data informatics Scientific Engineering Scientific Engineering Physics (Liszt) Scripting Probabilistic (RandomT) Machine Learning (OptiML) Rendering Parallel Runtime (Delite, Sequoia, GRAMPS) Dynamic Domain Spec. Opt. Locality Aware Scheduling Staging Polymorphic Embedding Applications Domain Specific Languages Heterogeneous Hardware DSL Infrastructure Task & Data Parallelism Hardware Architecture OOO Cores SIMD Cores Threaded Cores Specialized Cores Static Domain Specific Opt. Programmable Hierarchies Programmable Hierarchies Scalable Coherence Scalable Coherence Isolation & Atomicity On-chip Networks On-chip Networks Pervasive Monitoring 50

51 Fuel injection Transition Thermal Turbulence Combustion Example: Liszt - A DSL for Physics Simulation Mesh-based Numeric Simulation Huge domains –millions of cells Example: Unstructured Reynolds-averaged Navier Stokes (RANS) solver 51

52 Liszt as Virtualized Scala val // calculating scalar convection (Liszt) val Flux = new Field[Cell,Float] val Phi = new Field[Cell,Float] val cell_volume = new Field[Cell,Float] val deltat = untilconverged { for(f <- interior_faces) { val flux = calc_flux(f) Flux(inside(f)) -= flux Flux(outside(f)) += flux } for(f <- inlet_faces) { Flux(outside(f)) += calc_boundary_flux(f) } for(c <- cells(mesh)) { Phi(c) += deltat * Flux(c) /cell_volume(c) } for(f <- faces(mesh)) Flux(f) = 0.f } AST Hardware DSL Library Optimisers Generators … … Schedulers GPU, Multi-Core, etc 52

53 53 Follow us on

Download ppt "Scala Next SF Scala meetup Dec 8 th, 2011. 2 Scala Today."

Similar presentations

Ads by Google