1. ©1998-2003, Michael Aivazis DANSE Software Architecture Challenges and opportunities for the next generation of data analysis software Michael Aivazis California Institute of Technology DANSE Software Workshop September 3-8, 2003

2. ©1998-2003, Michael Aivazis 2 Outline Ontology Introduction to the architectural elements components and their parts data transport: ports and pipes Implementation issues analysis, modeling, simulation software engineering collaborative development distributed computing user interfaces graphics Status

3. ©1998-2003, Michael Aivazis 3 Ontology DANSE is a software architecture: a specification of the organization of the software system a description of the crucial structural elements and their interfaces a specification for the possible collaborations of these elements a strategy for the composition of structural and behavioral elements DANSE is multi-layered flexibility complexity management robustness under evolutionary pressures Contemplate the disconnect between a remote, parallel computation and your ability to control it from your laptop application-general application-specific framework computational engines

4. ©1998-2003, Michael Aivazis 4 User stereotypes Visiting scientist occasional user of prepackaged and specialized analysis tools Instrument specialist author of prepackaged specialized tools Expert scientist prospective author/reviewer of PRL paper Analysis expert author of analysis, modeling or simulation software Software integrator responsible for extending software with new technology Framework maintainer responsible for maintaining and extending the DANSE infrastructure

5. ©1998-2003, Michael Aivazis 5 Example dataflow diagram NeXusReader Selector Bckgrnd Selector Energy NeXusWriter times instrument info raw counts filename time interval energy bins filename A picture that’s worth a thousand questions…

6. ©1998-2003, Michael Aivazis 6 Component Component schematic input ports output ports properties component core name control

7. ©1998-2003, Michael Aivazis 7 Component anatomy Core: encapsulation of computational engines middleware that manages the interaction between the framework and codes written in low level languages Harness: an intermediary between a component’s core and the external world framework services: control port deployment core services: deployment launching teardown

8. ©1998-2003, Michael Aivazis 8 Component core Three tier encapsulation of access to computational engines engine bindings facility implementation by extending abstract framework services Cores enable the lowest integration level available suitable for integrating large codes that interact with one another by exchanging complex data structures UI: text editor facility bindings custom code core

9. ©1998-2003, Michael Aivazis 9 Computational engines Normal engine life cycle: deployment staging, instantiation, static initialization, dynamic initialization, resource allocation launching input delivery, execution control, hauling of output teardown resource de-allocation, archiving, execution statistics Exceptional events core dumps, resource allocation failures diagnostics: errors, warnings, informational messages monitoring: debugging information, self consistency checks Distributed computing Parallel processing

10. ©1998-2003, Michael Aivazis 10 Flexibility through the use of scripting Scripting enables us to organize large numbers of user tunable parameters allow the runtime environment to discover new capabilities without the need for recompilation or relinking compose computations at runtime The interpretive environment: Python is a modern object oriented language robust, portable, mature, well supported, well documented easily extensible rapid application development has been extended to support for parallel programming has no measurable impact on either performance or scalability

11. ©1998-2003, Michael Aivazis 11 Pyre architecture component bindings library extension component bindings custom code core facility framework facility component bindings custom code service requirement implementation package The integration framework is a set of co-operating abstract services FORTRAN/C/C++ python

12. ©1998-2003, Michael Aivazis 12 Component harness The harness collects and delivers user configurable parameters interacts with the data transport mechanisms guides the core through the various stages of its lifecycle provides monitoring services Parallelism and distributed computing are achieved by specialized harness implementations The harness enables the second level of integration adding constraints makes code interaction more predictable provides complete support for an application generic interface

13. ©1998-2003, Michael Aivazis 13 Data transport data pipe input port output port

14. ©1998-2003, Michael Aivazis 14 Ports and pipes Ports further enable the physical decoupling of components by encapsulating data exchange Runtime connectivity implies a two stage negotiation process when the connection is first established, the io ports exchange abstract descriptions of their requirements appropriate encoding and decoding takes place during data flow Pipes are data transport mechanism chosen for efficiency intra-process or inter-process components need not be aware of the location of their neighbors Standardized data types obviate the need for a complicated runtime typing system meta-data in a format that is easy to parse (XML) tables histograms

15. ©1998-2003, Michael Aivazis 15 Component implementation strategy Write engine custom code, third party libraries modularize by providing explicit support for life cycle management implement handling of exceptional events Construct python bindings select entry points to expose Integrate into framework construct object oriented veneer extend and leverage framework services Cast as a component provide object that implements component interface describe user configurable parameters provide meta data that specify the IO port characteristics code custom conversions from standard data streams into lower level data structures All steps are well localized!

16. ©1998-2003, Michael Aivazis 16 Application-general user interface The natural interface good correspondence between logical and physical descriptions of the computation Multiple views of the computation data flow control flow deployment of distributed components Dynamic interface generation from component supplied specifications High end graphics data plotting in two and three dimensions three dimensional visualizations

17. ©1998-2003, Michael Aivazis 17 Data flow paradigm appears natural usability problems are focused on knowledge of what is possible used by many commercial and open source tools Improvements decouple UI from diagram logic interface use OpenGL! collaborative interesting and relevant research diagram logic thin, reusable component scripting multi-layered control development can use existing solutions as a guide of what not to do many modules already available in pyre enable distributed programming Target for prototype: early 2004 Visual Programming Environment

18. ©1998-2003, Michael Aivazis 18 User interface prototypes - I

19. ©1998-2003, Michael Aivazis 19 User interface prototypes - II

20. ©1998-2003, Michael Aivazis 20 Distributed computing Enabled by design careful orchestration of the interactions between the computational engines and their environment Issues authentication security remote deployment, control and monitoring Two pronged implementation strategy use currently available, robust (but primitive) technologies to extend the framework initial authentication and deployment based on ssh authentication and security using pyre services User interfaces are harder to write correctly!

21. ©1998-2003, Michael Aivazis 21 Application-specific interfaces User interface should be simple, easy to use treat the data analysis process at the right level of abstraction have reasonable defaults for most choices Forms or wizards? As users get more familiar they become more demanding transparent access to the underlying application-generic interface access to the scripts?

22. ©1998-2003, Michael Aivazis 22 Status Leveraging pyre: large number of existing services component cores integration strategy well understood component harnesses support for user configurable properties is complete data transport User interfaces two application-generic prototypes under construction Missing pieces engines for analysis, simulation and modeling application-specific UI tests, of all sorts… documentation training materials