1. S. Veseli - SAM Project Status SAMGrid Developments – Part I Siniša Veseli CD/D0CA

2. S. Veseli - SAM Project Status Completed Projects DB Schema Redesign Python API Batch Adapter D0 Online dCache TDP Monitoring Tools Data Dimensions Grammar (Rob K.)

3. S. Veseli - SAM Project Status Ongoing Projects DB Server C++ API Configuration Management Test Harness Grid-JIM Developments (Rob K.) Chains and Links (Rob K.) CDF DFC Migration (Rob K.) CDF (H Stream) Reprocessing (Rob K.) Station Developments (Rob K.) Linux Deployment of Centralized Services (Rob K.) Subproject Assessments (Rob K.)

4. S. Veseli - SAM Project Status DB Schema Redesign Provided support for the unified (D0/CDF) DB schema Wyatt and Adam were driving the efforts, many others have been involved Description of schema changes can be found on http://www-d0.fnal.gov/~lyon/samSchema/ http://www-d0.fnal.gov/~lyon/samSchema/ Completed in May 2003, few minor changes introduced later One of the major reasons for the DB Server rewrite

5. S. Veseli - SAM Project Status Python API Most of the work done by Lauri Replacement for the old sam_user code Benefits: – Real Python API – Self-documenting interfaces – Maintainable code – Improved performance Status: – Deployed in production since June 2003 – Maintenance load is very low – Ongoing work on replacing the functionality relevant to file metadata

6. S. Veseli - SAM Project Status Batch Adapter Replacement for the old station batch adapters Features: – Flexible/configurable, supports any batch system – Non-restrictive (i.e., does not require specific batch system configuration) – Support for multiple user processes, grid job submission interface Status: – Deployed in production since June 2003 – Low maintenance load Possible future work: – ability to export/import adapter configuration in XML format

7. S. Veseli - SAM Project Status D0 Online dCache TDP (Tapeless Data Path) Work done by Lauri/Siniša Provides functionality of storing files via dCache write pools for the D0 online system: – Reduce resource contention (tape drives) – Files are immediately available for use, even before they are stored on tape D0 requirement: ability to choose between encp and dccp for different streams – this led to sam_cp redesign

8. S. Veseli - SAM Project Status D0 Online dCache TDP New sam_cp features: – Refined interface: ability to perform tasks before and after invoking of the actual file transfer – New file transfer protocol management: ability to plug in an arbitrary algorithm for determining which protocol to choose Status: – Used by D0 Online in production since November 2003 – Operations: load problems with online machines Possible future sam_cp work: – XML-based configuration – Server which handles handshakes between sites

9. S. Veseli - SAM Project Status Monitoring Tools Excellent monitoring tools developed in the last year by Adam and Robert SAMTV: shows what is happening with stations and projects ( http://d0db-dev.fnal.gov/sam/samTV/samTV.html ) http://d0db-dev.fnal.gov/sam/samTV/samTV.html CAB monitoring: automatically reconfigure the CAB station depending on the problems in the cluster

10. S. Veseli - SAM Project Status DB Server Work started in June 2003 after agreement on the new DB schema was reached Developers involved: Lauri, Carmenita, Steve, Siniša – none of us worked full time the project We’ve got lots of help from DBAs (Julie, Anil): – mini-SAM developer databases Benefits: – Code base and the number of IDL interfaces were significantly reduced and cleaned up – New CORBA infrastructure: wrapper classes around IDL structs and methods, exception handling – lead to code reuse, easier development, more maintainable code

11. S. Veseli - SAM Project Status DB Server Benefits: – Removal of non-atomic methods – Multithreading capability – Replaced fnorb with omniorb: better performance, less bugs – New file metadata classes: more flexibility, easier maintenance – New set of administrative interfaces – Merging of the MC and Farm Request systems – Improved autodestinations

12. S. Veseli - SAM Project Status DB Server Status: – Basic functionality is ready (metadata classes, station/project interfaces) – We are getting ready to upgrade the D0 development DB schema, start running test harness – To be completed: MC/Farm Request system merge, autodestinations, client code Deployment plan: – minimize effect on users, do not force immediate client upgrades – introduce new servers and upgrade clients gradually as they become ready

13. S. Veseli - SAM Project Status DB Server Milestones: – Early March: upgrade D0 integration DB to the latest schema, start integration testing – Early April: upgrade D0 production DB to the latest schema, start deploying new db servers - CDF will follow Next project assessment: mid-March

14. S. Veseli - SAM Project Status C++ API Provide unified support for both D0 and CDF SAM C++ clients – Code reuse, common CORBA infrastructure – Faster development, lower maintenance load New features: – File deliveries via CORBA callbacks – SAMTV Ready! (i.e., has complete monitoring event infrastructure)

15. S. Veseli - SAM Project Status C++ API Planned features: – Support for the multiple naming services – Ability to influence client behavior at runtime (e.g., direct the client to stop requesting files) – Support for staging files – Ability to define datasets, start projects (needed for CDF) Status: – Initial version released, D0 C++ client (sam_manager) using it (sam_manager code reduced to 30% of its original size) – CDF (Fedor) is looking into adapting AC++ to use the API Mid-March milestone: add CDF-requested functionality Next project assessment: mid-March

16. S. Veseli - SAM Project Status Configuration Management Goal: simplify existing (complex) installation/configuration, eliminate code compatibility problems, etc. Work done by Art, recently joined by Adam on the D0 side Status: – Station installation scripts ready, used by CDF in production – Adapting procedure for D0 in progress – We need similar scripts for client code – We have to revisit upgrade/downgrade procedures

17. S. Veseli - SAM Project Status Configuration Management Late February milestone: both experiments should be using the scripts for their standard station installations Next project assessment: mid-March

18. S. Veseli - SAM Project Status Test Harness Old test harness: – Has not been developed in couple of years, did not keep up with the SAM code development – Hard to adapt for CDF usage Work on the new test framework started in December 2003 (Matthew, Stefan, Adam) Key features: – Easier, XML-based configuration – Written in python only, uses new API, easier to maintain – Cleaner output, easier monitoring – More flexibility in creating/adding tests

19. S. Veseli - SAM Project Status Test Harness Status: – Basic infrastructure completed – Work on creating tests is in progress Mid-March milestone: demonstrate basic functionality (tests involving file delivery/storage) Next project assessment: early April

20. S. Veseli - SAM Project Status Concluding Remarks To be continued by Rob…