A PPARC funded project Astronomical services: situated software vs. commodity software Guy Rixon, AstroGrid/AVO/IVOA Building Service Based Grids - GGF11

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 2. What is “situated software”? Situated software is: written for/by a specific, limited group of users; therefore specialized, not general; not a commodity; written for a particular environment; therefore optimized, not portable or scalable. Named by Clay Shirky ( Astronomical computing traditionally runs on situated software

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 3. User-programmers in astronomy Rough estimates for the astro population: ~10 4 professional astronomers At least one project per researcher ~10 3 projects generating custom code > 100 FTE of user programming  Includes scripting of “building-block” mini-apps < 50 FTE of professional developers Major resistance to change User programmers naturally produce situated software

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 4. Cost savings of situated software Wide user-base Fully general Fully situated Wide scope Effort x3 Effort x3 (Adapted from Brookes, “The Mythical Man-Month”) Suitable for commodity service Non-portable (situated) Highly-portable Effort x3 Suitable for submission to job- manager service

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 5. Data-complexity rules code Computing for observational astronomy is dominated by data complexity Data volume is a secondary problem  CPU power is less problematic Some data need specialized treatment Not uniform across all astronomy fields E.g. photometric calibration varies between wavebands Fit of s/w to data limits quality of science Specialized treatment leads to situated software

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 6. Design conflict for astronomy grids Astronomy “eco- system” depends on situated software Grids depend on commodity software But the Virtual Observatory is supposed to enhance astronomy by introducing a grid! Have to fix this conflict when factoring the VObs into services

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 7. Simple strategy #1 Desktop Data-grid node Local storage Application File store Data files in Results out Fails: not enough bandwidth; some data in RDBMS

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 8. Simple strategy #2 Desktop Compute-grid node Application Grid client Job-manager service Data, app. in Results out Fails: situated apps not portable; problem not CPU- bound; bandwidth limits

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 9. Composite strategy Don’t commoditize whole applications Commoditize building-block services Searchable archives as services Publish new algorithms as situated services Connect services with data grid Support scripting Commoditize interfaces, not implementations Prefer description to standardization Succeeds (so far…)

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 10. Searchable archives as services Desktop Data centre Application Archive Web-service Data extract out Query (ADQL) Archive Commodity service with plug-in i/f (e.g. Publishers’ AstroGrid Library) Situated search engine

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 11. New algorithms as situated services Application Desktop Data centre 1 Archive Web-service Data 1 Data centre 2 Algorithm Web-service Data 1 Data 2 Written, owned and operated only by data centre 2  situated Problem: can user programmers handle SOAP + attachments? Written specifically for service in question

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 12. Services connected by data grid Application Desktop Data centre 1 Archive Web-service Data 1 Data centre 2 Algorithm Web-service Data 2 VoSpace Web-service Storage node Data 1 Control Storage manager Situated Commodity

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 13. Established algorithms as commodities Application Desktop Data centre 1 Archive Web-service Data 1 Algorithm Web-service Data 2 VoSpace Web-service Storage node Control Made portable outside data centre of origin Many suitable services in grid; application picks closest one to data Registry Look up services ( n/view/IVOA/IvoaResReg)

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 14. Workflow as scripting Workflow engine Grid node Data centre 1 Archive Web-service Data 1 Algorithm Web-service Data 2 VoSpace Web-service Storage node Control Application Desktop Situated scripting: not portable to desktop; optimized for VObs Script Report Could use commodity app to control workflow

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 15. Service description: CEA Workflow engine Grid node Data centre Algorithm Data 1 CEA Web-service Data 2 VoSpace Web-service Storage node Control Application Desktop CEA metadata tells engine how to use CEA for a particular algorithm Script Report Registry CEA-Service metadata Situated 100% commodity

GGF11. Astronomical services: situated s/w vs. commodity s/w; slide 16. Summary Astronomy runs on situated software The Virtual Observatory enhances astronomy by introducing commodity software Service Oriented Architecture may allow us the best of both worlds