1. Integration and development within the CCPN framework Expanded from presentation, Paris Oct 2006

2. Overview ARIA 2 integration DASH integration CLOUDS integration

3. Integration options ■Integration requires lossless exchange of data with the CCPN data standard : ●Direct API access. Application interacts directly with CCPN data structures. Easiest for new applications. ●In-memory conversion. Application loads CCPN data and converts them to its own data structures in memory. ●Format conversion. Application input/output files are converted to data standard via file conversion. Less well integrated than the alternatives ■Interoperability, or combining applications into a pipeline, requires further coordination.

4. Integration options 2 The integration options in this presentation will work for applications in any language that has a CCPN implementation. The CCPN applications are written mainly in Python, as are the utility function libraries and GUI widgets presented here. Only Python applications can be developed ‘inside the CCPN framework’ in the manner presented here. Applications in other languages will have only the generated APIs and I/O libraries available.

5. Application Custom conversion CCPN Interface Schemes Proprietary Memory Formatted File Format Converter CCPN XML/SQL Proprietary Model CCPN Data Model CCPN XML/SQL CCPN Data Model CcpNmr Functions CCPN XML/SQL Application Via FormatConverter In-memory conversion Direct API access

6. Development inside the CCPN framework ■CcpNmr Macros ●Small home-use Python functions ■Additions to function library ●Functions incorporated in software release ●Community sharing ■Embedded options ●Extension to CcpNmr application ■Stand-alone applications ●Built on CCPN libraries and API

7. CcpNmr Analysis Function Library ■Assignment ■Constraints ■Data Analysis ■Experiments & Spectra ■Peaks ■Structure ■Spectrum Windows assignResToDim(peakDim, resonance) Assign a resonance to peak dimension Checks: Any atoms are of a valid molecule Isotopes match dimension Shift is within tolerances Whether the aliasing is changed Creates: Covalent links between resonances Peak annotation label Updated chemical shift value Data Model objects

8. Ccp Stand-Alone AppTemplate ■ Menu System ■ Project handling ● New ● Load ● Save ● Backup ■ Popup template ● Widgets ● Geometry ● Plumbing

9. CcpNmr Graphical Widgets ■ A library for any developer to use ColorListPulldownMenuScrolledMatrixLabelFrameCheckButtonButtonLabelEntryButtonList

10. CcpNmr Mega Widgets ■ Build them into your own code! ● ScrolledMatrix ● ScrolledGraph ● StructureFrame

11. Example Applications ■ARIA 2 ●Structure calculation by iterative assignment of ambiguous restraints ●In memory data model to data model conversion: Python API code ●Constraints, Violations, Structures, Molecules ■DASH ●Predict dihedral angles from chemical shifts ●GUI based on application template ●Write input and read output of C program ●Collate reference chemical shifts with structure: FormatConverter ●Molecules, Dihedral constraints ■CLOUDS ●Assignment free structure determinations ●Python and C modules: Python-C interface ●Interpret proprietary files

12. Overview ARIA 2 integration DASH integration CLOUDS integration

13. ARIA2-CCPN interface Custom conversion ARIA Data Model CCPN Data Model CCPN XML/SQL Application Python-Python in-memory conversion ARIA XML ■ARIA imports ●Peak Lists ●Constraints ■Distance ■Dihedral ■RDC ●Sequences ●Chemical shifts ■ARIA exports ●Peak Assignments ●Constraints ●Violations ●Structures

14. ARIA functions & CCPN functions ARIA Data Model CCPN Data Model CCPN XML/SQL Python ARIA XML ARIA2-CCPN interface

15. CCPN object selection

16. CCPN conversion in ARIA2 def getAriaChemicalShift(self, ccpn_object): from Datum import ChemicalShift from Datum import ChemicalShift if ccpn_object is None: if ccpn_object is None: val = None val = None err = None err = None else: else: val = ccpn_object.value val = ccpn_object.value if hasattr(ccpn_object, 'valueError'): if hasattr(ccpn_object, 'valueError'): err = ccpn_object.valueError err = ccpn_object.valueError else: else: err = ccpn_object.error err = ccpn_object.error return ChemicalShift(val, err) return ChemicalShift(val, err) CCPN Nmr.Shift ARIA ChemicalShift

17. ARIA 2 functions in CCPN ■Bundled with CCPN distribution ■This side of the interface maintained by CCPN ■Gives CcpNmr Applications access to ARIA projects importAria2RunData(dirName, constraintSet=None, project=None, loadStructures=False) project=None, loadStructures=False)getAria2ObjectsFromPickle(fileName) getChainFromAria2(restraints, project) getStructuresFromAria2Dir(dirName, chain) getConstraintsFromAria2(ariaRestriaints, chain, constraintSet) getConstraintFromAria2(ariaRestraint, constraintList) getFixedResonanceFromAria2Atom(ariaAtom, chain, constraintSet) getAria2AtomSetPairs(ariaRestraint)

18. ARIA 2 conversion in CCPN def getViolationFromAria2(ariaViolation, constraint, violationList): violation = None if ariaViolation.isViolated(): calcDist = ariaViolation.getAverageDistance().getValue() calcError = ariaViolation.getAverageDistance().getError() fracViols = max(0.0,min(ariaViolation.getDegreeOfViolation(),1.0)) violValue = ariaViolation.getUpperBoundViolation().getValue() violation = violationList.newViolation(violation=violValue, calcValue=calcDist, calcValueError=calcError, calcValue=calcDist, calcValueError=calcError, constraint=constraint, fractionViolated=fracViols) return violation constraint=constraint, fractionViolated=fracViols) return violation ARIA Violation CCPN NmrConstraints.Violation

19. ARIA 2 calling CCPN functions import ccpnmr.analysis.ConstraintBasic as CB self.message('CCPN export: Accessing data model...') ccpn_project = self.open_ccpn_data_model(project) export = 1 chain = CB.getChainfromAria2(restraints, ccpn_project) if not chain: self.warning(StandardError, self.warning(StandardError, 'CCPN export: No molecular system found.') 'CCPN export: No molecular system found.') return return structures = CB.getStructuresFromAria2Dir(path, chain) if not structures: self.warning('CCPN export: Unable to load any structures self.warning('CCPN export: Unable to load any structures from iteration directory %s' % path) from iteration directory %s' % path)else: self.message('CCPN export: PDB files exported.') self.message('CCPN export: PDB files exported.') struct_gen = CB.makeStructureGeneration(structures, constraintSet) struct_gen = CB.makeStructureGeneration(structures, constraintSet)

20. Overview ARIA 2 integration DASH integration CLOUDS integration

21. DASH Overview ■Predicts peptide dihedral angles from chemical shifts ●Similar to TALOS ■Uses CCPN Python API directly in code ●No real CCPN conversion needed ■Scientific C code separate from CCPN DASH Python Tkinter GUI DASH CInput FileOutput File CCPN XML DASH database

22. DASH Overview ■Imports ●Chemical shifts ●Reference chemical shifts ●Reference structures ●Molecular sequence BMRB chemical shifts NMR Star format PDB protein structures PDB format DASH Python DASH C Chemical shifts Molecular sequence Dihedral angle constraints ■Exports ●Dihedral angle constraints ■Values ■Errors CcpNmr FormatConverter & Analysis functions

23. CCPN API use in DASH phiResonances = (fixedResonances[0], fixedResonances[1], fixedResonances[2], fixedResonances[3]) fixedResonances[2], fixedResonances[3]) phiConstraint = constraintList.newDihedralConstraint(resonances=phiResonances) psiResonances = (fixedResonances[1], fixedResonances[2], fixedResonances[3], fixedResonances[4]) fixedResonances[3], fixedResonances[4]) psiConstraint = constraintList.newDihedralConstraint(resonances=psiResonances) # make items DASHres = self.DASHmol.getResidue(i + first_resNum) phi = DASHres.pred_phi[DASHres.index] psi = DASHres.pred_psi[DASHres.index] phi_limit = DASHres.sd_phi[DASHres.index] * 2 psi_limit = DASHres.sd_psi[DASHres.index] * 2 phiConstraint.newDihedralConstraintItem(targetValue=phi, upperLimit =phi+phi_limit, upperLimit =phi+phi_limit, lowerLimit =phi-phi_limit, lowerLimit =phi-phi_limit, error=phi_limit) error=phi_limit) psiConstraint.newDihedralConstraintItem(targetValue=psi, upperLimit =psi+psi_limit, upperLimit =psi+psi_limit, lowerLimit =psi-psi_limit, lowerLimit =psi-psi_limit, error=psi_limit) error=psi_limit)  angle prediction CCPN DihedralConstraint

24. DASH – Python GUI ■Uses the CcpNmr application template ■Uses the CcpNmr Python Tkinter graphics objects (widgets) ●AppTemplate provides menu ●ScrolledMatrix provides tables ●ScrolledCanvas used custom graphics

25. ■DASH Ramachandran plot used as basis for CcpNmr equivalent ●Development goes both ways! DASH – Python GUI

26. Overview ARIA 2 integration DASH integration CLOUDS integration

27. The CLOUDS Protocol ■Automated assignment & structure determination ●Miguel Llinas, Alex Grishaev, et al. ●Spatial distribution of anonymous resonances generated with NOEs ■Integrated within CCPN ●An Analysis module ●Data Model glues modules ●Functional platform ●Distribution network A network of distance constraints between anonymous atoms is sufficient to generate a low resolution protein structure. H H H H

28. The CLOUDS Protocol A family of Clouds

29. CcpNmr Clouds Spectra Distance Constraints Proton Clouds Chain Assignment Protein Structure Spin Systems NOE matrix Pick Peaks, Link Shifts & Combine Relaxation Matrix Optimisation Hydrogen Atom Molecular Dynamics Chain Fitting & Molecular Replacement HCloudsMD Full Structure Calculation LinkPeakLists BACUS MIDGE CloudThreader ARIA 2.1 ■Implementation of existing software ●Interface existing code ●Re-written code modules ●Embed in CcpNmr

30. CcpNmr Clouds: BACUS ■Preserve C code ■Generate input files ■Interpret output files ■Really simple GUI! ShiftList NOE PeakLists Resonances BACUS Assigned PeakLists Input FilesOutput Files

31. CcpNmr Clouds: HCloudsMD ■FORTRAN -> C ■Python wrapper ●Control distance constraints ●Cooling scheme parameters ●Dynamics engine Distance Constraints HCloudsMD Family of Clouds Parameters Resonances

32. CCPN Python using CLOUDS objects def runDynamicsProtocol(atomCoordList, noeConstrList, dynamicsEngine, noeForceField, coolingScheme, rp_force_const=1): noeForceField, coolingScheme, rp_force_const=1): for (i,temp_i,temp_f,ncooling,nsteps,tau,rp_scale) in coolingScheme: for (i,temp_i,temp_f,ncooling,nsteps,tau,rp_scale) in coolingScheme: tref = temp_i tref = temp_i rpf = rp_force_const * rp_scale rpf = rp_force_const * rp_scale if (ncooling > 1): if (ncooling > 1): dtemp = (temp_f - temp_i) / float(ncooling - 1) dtemp = (temp_f - temp_i) / float(ncooling - 1) else: else: dtemp = 0 dtemp = 0 dynamicsEngine.rp_force_const = rpf dynamicsEngine.rp_force_const = rpf dynamicsEngine.tref = tref dynamicsEngine.tref = tref dynamicsEngine.tau = tau dynamicsEngine.tau = tau dynamicsEngine.nsteps = nsteps dynamicsEngine.nsteps = nsteps for j in range(ncooling): for j in range(ncooling): dynamicsEngine.run(atomCoordList, noeConstrList, noeForceField) dynamicsEngine.run(atomCoordList, noeConstrList, noeForceField) dynamicsEngine.tref = dynamicsEngine.tref + dtemp dynamicsEngine.tref = dynamicsEngine.tref + dtemp return atomCoordList return atomCoordList

33. Python-C interface #include "python_util.h" static PyObject *init_Py_Dynamics(PyObject *self, PyObject *args, PyObject *keywds) { float rp_force_const = 1, beta = 10, rmin = 2.25, drzap = 2; float rp_force_const = 1, beta = 10, rmin = 2.25, drzap = 2; float tref = 1000, tau = 0.001, elapsed_time = 0; float tref = 1000, tau = 0.001, elapsed_time = 0; int nsteps = 1000, nprint = 3000; int nsteps = 1000, nprint = 3000; static char *kwlist[] = { "rp_force_const", "beta", "rmin", static char *kwlist[] = { "rp_force_const", "beta", "rmin", "drzap", "tref", "tau", "elapsed_time", "nsteps", "nprint", NULL }; if (!PyArg_ParseTupleAndKeywords(args, keywds, "|fffffffii", kwlist, if (!PyArg_ParseTupleAndKeywords(args, keywds, "|fffffffii", kwlist, &rp_force_const, &beta, &rmin, &drzap, &tref, &tau, &elapsed_time, &nsteps, &nprint)) RETURN_OBJ_ERROR("need arguments: [ rp_force_const, beta, rmin, drzap, tref, tau, elapsed_time, nsteps, nprint ]"); RETURN_OBJ_ERROR("need arguments: [ rp_force_const, beta, rmin, drzap, tref, tau, elapsed_time, nsteps, nprint ]"); return (PyObject *) new_py_dynamics(rp_force_const, beta, rmin, return (PyObject *) new_py_dynamics(rp_force_const, beta, rmin, drzap, tref, tau, elapsed_time, nsteps, nprint); }

34. Python callable functions in C static PyObject *run(PyObject *self, PyObject *args) { Py_Dynamics py_dynamics = (Py_Dynamics) self; Py_Dynamics py_dynamics = (Py_Dynamics) self; PyObject *atom_coord_list_obj, *noe_list_obj, *noe_force_obj; PyObject *atom_coord_list_obj, *noe_list_obj, *noe_force_obj; Py_Atom_coord_list py_atom_coord_list; Py_Atom_coord_list py_atom_coord_list; Py_Dist_constraint_list py_noe_list; Py_Dist_constraint_list py_noe_list; Py_Dist_force py_noe_force; Py_Dist_force py_noe_force; Atom_coord_list atom_coord_list; Atom_coord_list atom_coord_list; Dist_constraint_list noe_list; Dist_constraint_list noe_list; Line error_msg; Line error_msg; CcpnStatus status; CcpnStatus status; if (!PyArg_ParseTuple(args, "OOO", &atom_coord_list_obj, if (!PyArg_ParseTuple(args, "OOO", &atom_coord_list_obj, &noe_list_obj, &noe_force_obj)) RETURN_OBJ_ERROR("must have three arguments: atomCoordList, RETURN_OBJ_ERROR("must have three arguments: atomCoordList, noeList, noeForce"); noeList, noeForce"); … … py_atom_coord_list = (Py_Atom_coord_list) atom_coord_list_obj; py_atom_coord_list = (Py_Atom_coord_list) atom_coord_list_obj; py_noe_list = (Py_Dist_constraint_list) noe_list_obj; py_noe_list = (Py_Dist_constraint_list) noe_list_obj; py_noe_force = (Py_Dist_force) noe_force_obj; py_noe_force = (Py_Dist_force) noe_force_obj; atom_coord_list = get_atom_coord_list(py_atom_coord_list); atom_coord_list = get_atom_coord_list(py_atom_coord_list); noe_list = get_dist_constraint_list(py_noe_list); noe_list = get_dist_constraint_list(py_noe_list); status = run_dynamics(py_dynamics->dynamics, atom_coord_list, status = run_dynamics(py_dynamics->dynamics, atom_coord_list, noe_list, py_noe_force->dist_force,error_msg); noe_list, py_noe_force->dist_force,error_msg);

35. Popup Constructors and Notifiers ■Init ●Setup local variables ●Subclass popup window ■Body ●Arrange Graphical elements ●Set up Data Model notifiers ●Set initial state ■Update ●Process updated values ●Redraw widgets based on status ■Widget callback ●From entry, buttons etc ●User functions ●Data Model change Body Update Notifiers Widgets Data Model External Influence Initialisation User Influence Update Filter