1. New logging and notification system for the Offline DropBox Martynas Mitrulevičius Thursday, 26 April 2012 AlCaDB/PdmV meeting Vilnius University, Lithuania

2. Outline What is Offline DropBox Motivation Internship goals Authentication SQLite upload Meta data upload Meta data structure Upload workflow Upload script Offline DropBox DB schema Online DropBox DB schema Workflow in Online DropBox References April 26, 2012Martynas Mitrulevičius2

3. What is Offline DropBox The Offline DropBox service: is implemented using web application and supports the automatic export of calibration constants. simplifies tasks for end-users in case they need to upload calibration resulting from an Offline analysis into the condition databases. is used to facilitate the automation of alignment and calibration workflows. April 26, 2012Martynas Mitrulevičius3

4. Motivation The current application is working reliably and smoothly, allowing for calibration constants to be exported into production DB, but some consolidation from the development and the operation team is needed, and some features were also requested by end-users In the Offline part of the DropBox the notification system for the exportation of calibration constants is not implemented. In the online process, only the logging coming from the CMS Framework libraries (known as PopCon monitoring) is implemented. o Tails of the log file are exported for checking by experts o http://cms-conddb.cern.ch/popcon April 26, 2012Martynas Mitrulevičius4

5. Internship goals Get acquainted to technologies and frameworks used at CERN and in the CMS experiment. Create PdmV database. Create Offline DropBox server with new logging and notification system. o Create a Web page allowing for calibration dataset upload. o Create CLI for the automatic exportation of calibration constants. April 26, 2012Martynas Mitrulevičius5

6. Authentication April 26, 2012Martynas Mitrulevičius6 Authentication using CERN username and password Only for members of cms-cond-dropbox CERN Authentication SOAP WebServices for user authentication

7. SQLite upload April 26, 2012Martynas Mitrulevičius7 Necessary tables in SQLite file: ORA_DB ORA_NAMING_SERVICE ORA_C_COND_IOVSEQUENCE ORA_C_COND_IOVSEQU_A0

8. Meta data upload April 26, 2012Martynas Mitrulevičius8

9. Meta data structure destDB connectionstring tag tagname inputtag inputtagname since begin Timetype timeSpec IOVCheck checker DuplicateTagHLT hltTag DuplicateTagEXPRESS expressTag DuplicateTagPROMPT promptTag DuplicateTagPCL pclTag usertext userlog checksum sqlitechecksum username user April 26, 2012Martynas Mitrulevičius9

10. Upload workflow April 26, 2012Martynas Mitrulevičius10 Server tasks: 1.Authenticates user. 2.Gets SQLite file. 3.Checks if SQLite is not corrupted and has required content. 4.Prepares pre-filled form. 5.Gets and validates metadata from the user. 6.Creates Tarball from SQLite and meta data. 7.Moves the file into Offline DropBox repository and adds new record to DB.

11. Upload script Command: opython dropbox_upl.py -u username -s sqlitefile.db -t metadata.txt How it works: o Server authenticates the user. o Script uploads SQLite file. o Server checks if the file is not corrupted and has required content. o Server returns the form. o Script reads meta data file, fills the form and upload it. o Moves file into Offline DropBox repository and acknowledges the file transfer to the user and to the notification system. April 26, 2012Martynas Mitrulevičius11

12. Offline DropBox DB schema April 26, 2012Martynas Mitrulevičius12

13. Online DropBox DB schema April 26, 2012Martynas Mitrulevičius13

14. Workflow in Online DropBox 1.Downloads Tarball file and extract it. 2.Checks the SQLite file checksum. 3.Moves to the DropBox directory. 4.Process the file. 5.Exports calibration constants with synchronization if needed. 6.Duplicates payloads if requested. All these steps are logged in the logging schema. April 26, 2012Martynas Mitrulevičius14

15. References Upload file: o http://m-dropbox.cern.ch/uploadSQLiteFile http://m-dropbox.cern.ch/uploadSQLiteFile Get tarball list: o http://m-dropbox.cern.ch/getTarList http://m-dropbox.cern.ch/getTarList Get tarball file: o http://m-dropbox.cern.ch/getTarFile?file=FILENAME http://m-dropbox.cern.ch/getTarFile?file=FILENAME Project files: o https://svnweb.cern.ch/cern/wsvn/cmsconddbweb/Offlin eDropBox https://svnweb.cern.ch/cern/wsvn/cmsconddbweb/Offlin eDropBox Documentation: o https://twiki.cern.ch/twiki/bin/edit/CMS/OfflineDropBox https://twiki.cern.ch/twiki/bin/edit/CMS/OfflineDropBox April 26, 2012Martynas Mitrulevičius15

16. Thank you April 26, 2012Martynas Mitrulevičius16