1. CCP4(i) Database Development Wanjuan (Wendy) Yang CCP4 Annual developer’s meeting March 28, 2006 York

2. Outline of the talk  Background of the project  System Architecture & components functionalities  Choice of technologies  Current status & immediate plan

3. Background of the project  As part of BIOXHIT: Expand current CCP4i job database and include extended tracking and storage of crystallographic data Aim to separate the project database handling from main CCP4i process, allow access to project database from non-CCP4i applications. Expand scope of CCP4i database to store richer dataset.  Useful for: Users of CCP4i Automated structure determination pipelines: to record the progress/steps/result within pipelines Non-CCP4i applications that want to use the database: sharing data between different applications

4. System Architecture DB Handler Project DB SQL db Client API CCP4i Data Visualiser Other databases e.g. PIMS XIA Client API HAPPy Client API Coot CCP4 MG Other CCP4 & Non-CCP4 Applications Project DB Database.def

5. Components functionalities  DB handler: Runs in server mode: one handler per user; allow many clients connect at the same time. Can be invoked from clients or start separately. Shut down procedure: automatically shutdown when there is no clients; clients send shutdown message. Broadcasting: when database changes state, handler notifies clients. Can deal with different types of database through DB APIs: e.g. i. CCP4i project database ii. SQL database

6. Components functionalities  Project Database: project history tracking data: store steps/jobs taken within software pipeline knowledge base data: common crystallographic data items used within the software pipeline

7. Components functionalities  Client API: A list of methods/commands for interacting with the handler. Hides details of socket communications Implemented in different programming languages to support different applications.  Visualisation tool: for viewing data and analysing data

8. Choice of technologies  Client-Server architecture: socket communications between server and clients.  XML used as messaging technology: Allow system being language independent Request example: NewProject ProjectName ProjectDirectory Response example: ok 3

9. Choice of technologies  Database backend choices: We provide multiple databases backend: Embedded database: we are using SQLite. Features:  simple to use: a small C library. Self-contained, embedded SQL database engine  portable: single files on the file system.  zero-configuration: no set-up or administration needed  fast random access to the data flat files: Def file backend implemented in Python. Replicates current CCP4i.  Client APIs implemented in: Tcl API to support CCP4i and Tcl-based applications. Python API to support python-based automation projects.

10. Current Status & immediate plans  Python DB handler has been implemented. Next step is to integrate in CCP4i with.def file backend. Allow other applications to make use of database.def data.  Currently developing SQL schema for “rich database” backend Consists of knowledge base (i.e. crystallographic data) and tracking (project history) Being developed in conjunction with HAPPy and XIA projects. Several iterations will be required to stabilise schema and API. Next step is to make Client APIs available to HAPPy and XIA.  Begin to develop visualisation tool to integrate project tracking data

11. Acknowledgements  Peter Briggs  Graeme Winter  Charles Ballard  Daniel Rolfe  Steve Ness  Many others …