1. Roy Williams Andrew Drake, Matthew Graham, Ashish Mahabal California Institute of Technology Skyalert and Event Processing

2. This talk defines these  VOEvent : Int’l standard representation of data for astronomical transient as XML  VOEventNet : A Protocol for pushing VOEvents, and its nodes  Skyalert : Software for managing VOEvents: web, subscriptions, repository

3. VOEvent Key Concepts  IVORN – unique identifier for event  Instance of a Stream that also has IVORN  How is it resolved?  Who  author’s provenance  What  data model designed by author  May have URLs … permanence, calibration?  WhereWhen  targeting in spacetime  Citation  related events  “followup", "supersedes" or "retraction” 3

4. VOEventLib http://lib.skyalert.org/VOEventLib/  Python interface to VOEvent  Code built automatically from schema  With ‘utility’ package Vutil v = VOEvent.VOEvent(role='test', ivorn='ivo://testing#111') author = Author(contactName=['Donald Duck']) v.set_Who(Who(Author=author)) what = What() what.add_Param(Param(name='apple', value='123')) what.add_Param(Param(name='orange', value='124')) v.set_What(what) s = stringVOEvent(v) print s

5. VOEventNet Key Concepts  Broker  Broadcaster of received events  May know which subscriber wants what  Subscriber  Connects to Broker  Long-term connection  Publisher  Scientifically responsible  Works with a broker  May sign events with PGP 5

6. VOEventNet a network of independent nodes exchanging VOEvents with the VOEvent Transport Protocol (VTP) LOFAR and 4piSky running Comet NASA events running GCN/TAN DC3.com running Dakota Skyalert running Comet CBAT AAVSO CRTS1,2,3 CSS NEO

7. VOEventNet defined http://arxiv.org/abs/ 1206.0236

8. query subscribe relay annotate translate registry publish repository VOEventNet Patterns

9. Query VOEventNet External annotator Translate Publish repository Skyalert Open sourced VOEventNet node Push messages Push actions Web interface Web services Many types of events can be homogenized and interoperated Supported by US National Science Foundation OCI-0915473

10. Skyalert Annotation Annotation can be: Newer observations or calibration Archive search for matches Machine-learning ‘decision’ Follow-up observation Comment / Circular VOEventNet Publish Annotation How can annotations be queued and clouded in case of high event rate?

11. Two lists of events betelgeuse.ligo.caltech.edu All ‘events’ from five high-energy missions including ‘utility’ events Received by VTP skyalert.org ‘role=observation’ (astrophysics) events only XMPP and email translations

12. Skyalert Stream Registry... will be VO registry http://www.skyalert.org/streams/ 12

13. Skyalert Key Concepts  Stream  Event semantics defined by author  What does Burst_SOD mean in the Fermi stream?  Like a data dictionary, a schema, common metadata  Portfolio  Event + associated data  As collection of VOEvents with the observation first  Query  Define what is ‘interesting’  For Past or Future events  Language, as XML, SQL tables (flex and fixed), KW dict, triple store?? 13

14. Concept: Stream as Event Template Event author defines parameters and meaning in advance. streamevent made by person in daytime made by robot system at night 14

15. SWIFT portfolio Notices from the satelliteCirculars from the humans 15

16. Event Databases Technology and Language Evaluations  Stardog  SELECT ?s WHERE {?s voe:hasParam ?p. ?p voe:hasName "…"^^xsd:string. ?p voe:hasValue ?v. FILTER (?v < …)}  MySQL fixed: with a separate table for each stream  select id from … where … < …  MongoDB (NoSQL): ivorn, param name, value  db.test.find({"param":…, "value":{"$lt":…}})  MySQL flex: with a params table  select id from params where name = "…" and value < …  eXist native XML  for $event in /voe:VOEvent where $event/What//Param[@name = '...']/@value {$event/@ivorn} Speed

17. Skyalert web services from django.utils import simplejson import urllib sk = http://skyalert.orghttp://skyalert.org # CRTS and CRTSCircular["First"]["eventClass"] == "Supernova" query = "285" jtableurl = "%s/events/jtable/%s/" % (sk, query) while 1: jsontable = urllib.urlopen(jtableurl).read() pytable = simplejson.loads(jsontable) list = pytable["list"] print "Found %d portfolios" % len(list) for pf in list: ivorn = pf[1] dict = {} dict['ivorn'] = ivorn dict['JSON'] = 'on' params = urllib.urlencode(dict) resolveurl = "%s/resolve/" % sk jsondetail = urllib.urlopen(resolveurl, params).read() pydetail = simplejson.loads(jsondetail) # ivorn of the primary event of the portfolio should be here crtsevent = pydetail["portfolio"][ivorn] mag = crtsevent["FirstDetectionparams"]["magnitude"] tok = ivorn.split('#') localivorn = tok[1] date = crtsevent[""]["ISOtime"] print localivorn, date, mag jtableurl = jsonresult["next"] if jtableurl == "finished": break 1107030071174105491 2011-07-03T08:36:01 18.935400 1107030091124153498 2011-07-03T08:01:30 18.756399 1107031180794128821 2011-07-03T04:39:47 20.169800 1107031070764142675 2011-07-03T04:07:39 18.166100 1107021180094118167 2011-07-02T10:46:06 18.601400 1107021041264109182 2011-07-02T09:31:06 15.879600 1107011150104107901 2011-07-01T10:47:36 18.279699 1107011120104119909 2011-07-01T10:46:15 17.874800 1107010040834139996 2011-07-01T06:17:32 18.483601 1107011040674116917 2011-07-01T04:13:08 19.630899 Magnitudes of CRTS supernovae 17

18.  Who are the subscribers?  What size telescope / Data mining / Citizens  Is it the same as the “GCN followup” gang?  How can we set up a clearinghouse of information?  How to build “simulated event stream” for LSST?  So prototypes can be tested  Remember average rate and burst rate  What are annotation services for LSST?  Same as “common queries”  What is the security requirement?  In testing, commissioning, observing  What language for alerts and history queries  Triples, SQL, noSQL, XML,...