1. Software for MAPS and MERLIN T.G. Perring ISIS Facility, Rutherford Appleton Laboratory

2. Outline Plan of talk: Overview Brief description of the present MAPS/Het/MARI software Challenges of MERLIN/ARCS How we plan to re-write the software Our aims from this meeting: Contribute to the plans for direction of ARCS software development Explore cooperation in software development –Experience + a working ARCS-like spectrometer –Fresh view of the problems of handling huge inelastic datasets

3. MAPS spectrometer Position sensitive detector array Specification: 20meV< E I < 2000meV l mod-chop = 10m l sam-det = 6m low angle bank: 3  -20  high angle bank:  60   hbar  /E I = 1- 5% (FWHH) 40,000 detector elements 2500 time channels  10 8 pixels  0.4GB datasets Background chopper Monochromating chopper Sample position

4. Overview Inelastic scattering from single crystals: Triple axis spectrometer: point-by-point serial operation –Few kB of data –Every group of users writes its own least-squares fitting algorithm from scratch –Ad-hoc cooperation only MAPS/MERLIN/ARCS: Parallel operation: collecting on 3D surface in 4D S(Q  space –MAPS: raw data: 10 8 pixels = 0.4GB corrected data: 1-2 x 10 7 = 0.1-0.2GB x 10 datasets MERLIN, ARCS: 1-2 x 10 7 = 0.1-0.2GB x 100+ datasets Need ‘Rietveld refinement’ for inelastic scattering –Groups cannot afford to write own software –Pointless anyway: e.g. in diffraction GSAS, CCSL are trusted –We should move to a similar model of trusted black boxes

5. Overview (Rietveld refinement cont.) Use entire data set to extract a limited number of parameters in a model –Magnetic coupling constants –Force constants –Full inversion of the data: if S(Q  in 4D Pragmatism: –In magnetism S(Q  (or  ''(Q  is the quantity to test –Qualitative features (are there antiferromagnetic fluctuations and if so, where ?) –Unknown processes distorting measure of goodness of fit –Rapid slicing and dicing, testing models on limited volumes of data, testing implications for other parts of the data volume, feeding back into operation of experiment seamless integration of simulation, visualisation and analysis programs More formal cooperation: Agreed data structures for instrument information, counts/ S(Q  Definitions for input/output of algorithms Capitalise on the investment we individually make in algorithms

6. Current software on MAPS, HET, MARI Planning experiment Tobyplot (reciprocal space viewing) Chop (resolution, flux) Data reduction HOMER Genie-II Visualization MSLICE GUI driven Matlab Analysis Tobyfit (single crystals) [Ad-hoc programs in Multi-frills MFIT MSCATT] Monitoring experiment Genie-II OpenGenie

7. Tobyplot Reciprocal space viewing Mostly important for 3D systems but very useful for 1D, 2D, to assess access in reciprocal space Ei=450 meVPsi=101.3° Fortran77 + PGPLOT VMS/Windows/Unix

8. CHOP Flux Resolution: At elastic position As function of Test flux/resolution compromises Fortran77 + PGPLOT VMS/Windows/Unix

9. Monitoring an Experiment GENIE-II I(t) for single spectrum rebinning, units conversion, integration Algebra on spectra [e.g. W1=(0.3W2 + W3)/W4] can call user-written FORTRAN algorithms Good for quick checks during and after experiments Still use today on MAPS VMS only OpenGENIE New generation Used on many ISIS instruments Windows, Unix, VMS Not the features of Matlab, IDL Is free, however

10. Data reduction HOMER (au. Ray Osborn) I(detno, t) corrects K f /k i, efficiency(k f ) S(detno,  ) WHITE_VAN (  solid angle of detectors) DIAG (  list of bad detectors) MONO_VAN (  absolute units conversion) Map file (detectors  workspaces) ASCII output file (or VMS binary) Encapsulates years of experience of the instruments Scaled very well even to MAPS $ homer/map=par:pix_981.map/mask=8900/van=8850 8900 100 -30 95 0.25

11. GUI interface  run info.  sample parameters  2D & 1D cuts 2D slices in (Q,  ) 1D cuts in (Q,  ) Imaging single crystal data on HET, MARI, MAPS, and IRIS (Radu Coldea (ISIS / Oak Ridge now Oxford) MSLICE:

12. Least squares fitting of resolution broadened cross-section models (Toby Perring, ISIS) TOBYFIT : Simultaneous fitting to many 2D or 1D data sets Text-based interface for entering:  instrument  sample parameters  cross-section parameters

13. MSLICE and TOBYFIT Integral part of the operation of the spectrometer - ‘Tertiary spectrometer’ MSLICE: Visualisation of 3D data in 2D slices, 1D cuts Can generate ‘backgrounds’ from selected parts of the data MATLAB as front end: GUI Graphics manipulating data structures, ad-hoc programming FORTRAN77 for speed of operation of algorithms PC with 1GB RAM, 500MHz+ necessary TOBYFIT: Fitting and simulation  test ideas by feeding S(detno,  ) to MSLICE Hold an experiment in a parameter file Fortran + PGPLOT Runs on VMS, Windows, UNIX Communicate via ASCII files: 1 for data, 1 for detector parameters other sample & instrument parameters

14. Challenges offered by future instrumentation Physics a function of 4 variables: C(x1,x2,x3,x4) Instrument gathers data on a 3D volume in that 4D space Data gathered on a fine non-Cartesian grid 0.1-0.2 Gbyte MAPS: usually ~10 settings in an experiment fine data on 3D surface, coarse in 4th dimension ARCS/MERLIN: 100+ settings fine in all 4 dimensions (scan Ei, or crystal orientation) 10-20GByte complete data set Data has low statistics - need techniques to pick out features in data will always need real-time slicing and dicing of data [too many ways of being led astray or being deceived] will be doing this after going back to home institution

15. Visualisation: a hierarchy of views View 4D data: ? How do that ? Define integration interval along any one dimension, and then: View 3D data: isosurfaces + slider control for intensity levels rotation and viewpoint binning along the three axes smoothing, image processing control Move a plane through the 3D volume: define integration interval along one of the remaining dimensions, and then: View 2D data: Contour plots, mountain plots, + slider controls for contouring levels interval scaling (linear, log, sqrt …) binning along the two axes Move a line across the plane, defining a thickness, and then View 1D data: overplotting, fine comparison book-keeping of titles of the plots... On raw counts, white beam files,… as well as S(Q,  )

16. Instrument resolution, modelling Must be able to simulate results of experiments and view results in same way as data (number crunching) Must perform on-line analysis (resolution-convoluted model fitting,multiple scattering) within framework of same package (even more number crunching) flexibility: fit on limited volume of data, simulate for whole dataset, slice- and-dice in same way as data to try out ideas User wants one-stop shop I(det,t) S exp (Q,w) * inst S calc (Q,w) Convolve with instrument Compare, fit Visualisation, algebra on 1,2,3,4D + Tobyplot MkII

17. Issues Number crunching - more than the typical user institute will have huge storage requirements (not just the raw data) data management a real problem - we already create hundreds of cuts thumbnails when click on file database functions (select by date, temperature, field, scan of a parameter…) history of analysis stored in file well-defined data structures needed ease interact seamlessly with other programs deconvolution, modelling user-written algorithms easy to write define appropriate methods and algebras (addition, subtraction, background generation, symmetrization…) not just GUIs: scripting must always be possible maximises flexibility

18. Our plans Existing programs need to rewritten F77, getting unmaintainable, monolithic,functionality insufficient, grown organically, written independently Define NeXus files to hold all processed files 1D,2D,3D,4D data + all relevant instrument information (detectors etc.) sufficient information for MCSTAS simulation include raw data files eventually Mirror the data structures in Fortran95 + methods Fortran95 for speedy algorithms [number crunching for visualisation (MSLICE-2), fitting (Tobyfit-2)] MATLAB for graphics, language for manipulation and scripting - the glue [+access to all features of MATLAB for ad-hoc manipulations] TGP, SMB + inst. Scientists + part effort from 2 post-docs (25-50% effort from each)

19. (plans cont.) About to start working with E-Science centre at Rutherford Laboratory funding to implement grid based applications for science within the laboratory demonstration projects: data portal to distributed data stores graphics processing and number-crunching ISIS projects chosen to focus the development for real applications aim to isolate user from where the work is done - no need for their own Beowulf cluster user will have a front end - in our case we want MATLAB but also web-based interface (oceanography, space science) Ensures that one version is maintained assumes high-speed networks [GLOBUS toolkit to isolate user from location of resources] ISIS: full effort of ISIS computing staff member + post-doc effort of E-Science centre