1. Prototyping the Next EPICS Archiver
Nikolay Malitsky
EPICS Collaboration Meeting
PSI, Switzerland
October 5, 2011

2. Content Background SciDB in Three Slides
Benchmark and Proposal of Channel Archiver Backend
Benchmark and Proposal of Channel Archiver Engine
Integrated Environment

3. Background Backends: original approach, RDBs New requirements:
ARCHIVER
Backends: original approach, RDBs
New requirements:
1 M channels and 1 M samples/s (BNL, SLAC, Diamond, PSI, …)
New data types (EPICS 4, DAQ, ITER, …)
New solutions: Protocol Buffers (SLAC), MSDPlus (Consorzio RFX ), Hypertable (INFN), SciDB (BNL), …

4. SciDB: Open Source Data Management and Analytics System
Consistent array-oriented model and formalism :
Generalization of the OLAP models
Natural and fundamental data type of scientific software (e.g. MATLAB)
Strong team of database experts lead by Mike Stonebraker
Wide range of domains represented by Science Advisory Board
Genomics
Astronomy
Environmental Observing Systems
Earth Science
Fusion
Remote Sensing
High Energy Physics
Atmospheric Sciences
Oceanography
Control System 4

5. SciDB Array-Oriented Model
CREATE ARRAY Example < a1:integer, a2:float>, a3:MyType> [Dim1=0:5, Dim2=0:4]
AQL: Array Query Language,
similar to SQL
AFL: Array Functional Language,
reminiscent of APL, etc. Including a
collection of operators, like join,
filter, slice, apply, and others
Other approaches:
Y. Zhao, et al., Array-Based Evaluation of Multi-Dimensional Queries in Object-Relational Database Systems, 1997
P. Baumann, et al., The Multidimensional Database System RasDaMan, 1998
Barrodale Company, DBXten DataBlade , (thanks to Lana Abadie for this reference)

6. SciDB Array-Oriented API
Chunk-based partitioning
Column-oriented partitioning

7. Benchmark of the SciDB 0.75 backend
May 17, 2011
Benchmark of the SciDB 0.75 backend
struct Complex {
double re;
double im; };
Scenario:
1000 channels
100 chunks/channel
1000 samples/chunk
1 sample : status(int16), severity (int16) , time stamp (int64), and Complex
Writing of 1000 channels: 127 s ( ~ 0.8 M samples/sec)
REGISTER_TYPE(complex, 0); // 0 – for variable size data …
Complex v;
v.re = 1.0*ic;
v. im = 1.0*(ie+1);
value.setData(&v, 16);
chunkIterator->writeItem(value);
Reading of 4 channels : 0.12 s ( ~ 3 M samples/sec)
Value& v = chunkIterator->getItem();
Complex* data = (Complex*) v.data();
Nice results, but the existing in-memory indexing approach does not scale
for historical data

8. Benchmark of the SciDB and Channel Archiver backends
CREATE ARRAY Ch001 < dbr: dbr_time_double> [ ts (epicsTimeStamp) = 0:*,1000, 0]
struct dbr_time_double{
dbr_short_t status; /* status of value */
dbr_short_t severity; /* severity of alarm */
epicsTimeStamp stamp; /* time stamp */
dbr_long_t RISC_pad; /* RISC alignment */
dbr_double_t value; /* current value */ };
Scenario:
1000 channels
chunks/channel
samples/chunk
1 sample : dbr_time_double
Chunk Size
SciDB
EPICS
100
1.6 M events /s
336 K events / s
1000
2.0 M events /s
454 K events /s
Writing
Chunk Size
SciDB
EPICS
100
6.7 M events /s
5 M events /s
1000
10 M events /s
Reading

9. SciDB - EPICS Driver SciDB API SciDB-EPICS EPICS backend
CHUNK SIZE
SciDB API
SciDB-EPICS
EPICS backend
Similar approach: Barrodale Company. Universal File Interface (again,
thanks to Lana Abadie for this reference )

10. Benchmark of the Channel Archiver Engine
K dbr_time_double/s
Engine
2-10 K 10 Hz
M. Kraimer, et al., EPICS Application Developer ‘s Guide, Ch 2.2 Example IOC Application, 2010
G.Manduchi, et al. New EPICS Channel Archiver based on MDSPlus Data System, Proc. IEEE RT, 2011

11. Chunk-based solution Buffer : 10s (Chunk size = 10 Hz * 10s = 100)
Herb Sutter, Writing a Generalized Concurrent Queue, Dr. Dobb’s, 2008 CA
Engine
10,000 channels PV
CircularBuffer PV
CircularBuffer
10,000 channels
Queue
Engine CA PV
CircularBuffer
09/30/ :43:33 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :43:42 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :43:52 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :44:03 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :44:12 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :44:22 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :44:32 Engine: writing done, channels: 10000, count: , time: e+00 s
09/30/ :44:42 Engine: writing done, channels: 10000, count: , time: e+00 s
Buffer : 10s (Chunk size = 10 Hz * 10s = 100)
Writing rate: 1 M / 3 s = 300 K samples/ s

12. A Bigger Picture Beamline Experiments EPICS 3 Control System EPICS 4
SciDB Node
Archiver
SciDB Node
SciDB
Node
SciDB Node
SciDB Node
EPICS 3 Driver
EPICS 3 Driver
HDF5 Driver
V 3
HDF5 Driver
HDF5 Driver
EPICS3
HDF5
HDF5
HDF5
Detectors
Magnets, BPMs, etc
Beamline Experiments
EPICS 3
Control System
Archiver
SciDB
Node
SciDB
Node
DAQ
EPICS 4
Infrastructure
Model Service
Another Service
V 4
EPICS 4 Driver
EPICS 4 Driver
HDF5 Driver
HDF5 Driver
V 4
EPICS4
HDF5
SciDB API
Magnets, BPMs, etc
Detectors

13. Thank You BNL : B. Dalesio, D. Dohan,
Diamond Light Source : J. Rowland
INFN : M. Giacchini
SciDB: J. Becla, P. Brown
SLAC: M. Shankar
Stony Brook University: Y. Kulinich