1. First Indico Workshop HTTP API Adrian Mönnich 29-27 May 2013 CERN

2. API Uhh… what’s an API and why do we need one? An application programming interface (API) is a protocol intended to be used as an interface by software components to communicate with each other. Wikipedia

3. Before the api export.py Not powerful at all: Only events inside categories XML, RSS, iCal or HTML  Accessing it requires an XML parser Protection? Not really  Must be restricted to trusted machines by IP Reusable code? None / export.py?fid=2l12&date=today&days=1000&of=xml

4. And the developers said… …let there be a proper API… …and they created one

5. Design ideas How to make the API awesome? Handle authentication in a clean and secure way (API Keys, OAuth2) Various security levels available (API keys, HMAC signatures, …) Let developers write code for the actual task Serialize Python data structures as JSON, XML, … Provide utilities for common things (relative dates, pagination)

6. Using the API Before we go into details, look at how it works Indico Administrators: API Modes: Control how strict the API key requirements are At CERN: We always require an API key, but a signature is only needed to access non-public information Persistency: Lowers security a little bit (one leaked URL is valid forever) Allows people to use e.g. a RSS reader (cannot sign requests)

7. Using the API Before we go into details, look at how it works Indico Users: Creating an API key is easy – just one click! Keys can be reset at any time, e.g. in case of a (suspected) leak Last request including IP address and full path is displayed Persistence can be toggled (if enabled globally)

8. Using the API Before we go into details, look at how it works The same data – available as XML, JSON and sometimes additional more formats (iCal, RSS, HTML) / export/categ/4751.[json|xml|html]

9. Using the API Before we go into details, look at how it works What it we need a signed request? /export/categ/123.json?limit=10 1. Add the API key to the params … &ak=api-key 2. Add the current timestamp to the params … &timestamp=1234567890 3. Sort the query string params alphabetically… ?ak=api- key&limit=10&timestamp=123… 4. Merge path and the sorted query string /export/categ/123.json?ak=api-key&limit=10&timestamp=1234567890 5. Create a HMAC-SHA1 signature of this string using the secret key as the key. 6. Append the hex-encoded signature to the query string: …&signature=xxx

10. EXTENDING THE API Creating a custom API endpoint Let’s keep it simple: We want an API that returns a range of numbers or characters. This does not access any indico-specific data but demonstrates some of the utilities available when using the API as a developer. / export/num/1-10.json / export/char/a-z.json

11. EXTENDING THE API

12. Relevant classes HTTPAPIHook : Hooks into the /export/ URL subtree, handles arguments IteratedDataFetcher : Provides pagination, sorting, relative dates, etc. IFossil : Describes how an object is converted to primitives (dict, list, int, str, bool) that can be serialized (JSON, XML, …) later

13. EXTENDING THE API

14. The hook, Pt. 1: Metadata @HTTPAPIHook.register class RangeHook(HTTPAPIHook): TYPES = ('num', 'char') RE = r'(?P [0-9]+|[a-z])-(?P [0-9]+|[a-z])' DEFAULT_DETAIL = 'simple' MAX_RECORDS = { 'simple': 10, 'palindrome': 5 }

15. EXTENDING THE API The hook, Pt. 2: Arguments Perform additional validation if necessary Use self._pathParams for arguments from the path regex Use self._queryParams to access arguments in the query string def _getParams(self): super(RangeHook, self)._getParams() self._start = self._pathParams['start'] self._end = self._pathParams['end']

16. EXTENDING THE API The hook, Pt. 3: The export methods Perform actions/validations specific to the export type Must return an iterator, usually provided by an IteratedDataFetcher def export_num(self, aw): try: start = int(self._start) end = int(self._end) except ValueError: raise HTTPAPIError('Invalid value', 400) return RangeFetcher(aw, self).numbers(start, end) def export_char(self, aw): if len(self._start) != 1 or len(self._end) != 1: raise HTTPAPIError('Invalid character', 400) return RangeFetcher(aw, self).chars(self._start, self._end)

17. EXTENDING THE API

18. The fetcher, Pt. 1: Metadata Remember DEFAULT_DETAIL and MAX_RECORDS ? Here we specify how objects are fossilized in those detail levels. More about fossilization later, let’s get some data first! class RangeFetcher(IteratedDataFetcher): DETAIL_INTERFACES = { 'simple': IDummyFossil, 'palindrome': IDummyPalindromeFossil }

19. EXTENDING THE API The fetcher, Pt. 2: Python iterator magic Remember: We call these methods from our export methods. In this case we just iterate over numbers/characters in the given range. DummyObject ? You will see soon… def numbers(self, start, end): iterable = xrange(int(start), int(end) + 1) iterable = itertools.imap(str, iterable) iterable = itertools.imap(DummyObject, iterable) return self._process(iterable) def chars(self, start, end): iterable = itertools.imap(chr, xrange(ord(start), ord(end) + 1)) # Besides that first line everything is the same like in numbers()

20. EXTENDING THE API The Dummy Object Usually we deal with custom objects in Indico. Because of historical reasons it is common to have getter methods (yes, we know it is ugly and unpythonic). The fossilization system is built around this  thin wrapper providing a getter for our primitive string values. class DummyObject(object): def __init__(self, value): self.value = value def getValue(self): return self.value

21. EXTENDING THE API

22. Fossils IDummyFossil is really simple: It fossilizes the getValue() return value of the actual object to a field named value. For the palindrome detail level we specify a custom callable to generate the value on the fly. The field name is determined automatically again. class IDummyFossil(IFossil): def getValue(self): pass class IDummyPalindromeFossil(IDummyFossil): def getPalindrome(self): pass getPalindrome.produce = lambda x: x.value + x.value[::-1]

23. EXTENDING THE API Does it work? / export/num/128-256.json?detail=palindrome / export/num/128-256.json / export/num/1-1.xml

24. EXTENDING THE API Yes! Did you notice the pagination? / export/num/128-256.json?detail=palindrome / export/num/128-256.json / export/num/1-1.xml

25. What else The API is even more powerful! Caching (unless disabled by the user per-request) POST (to modify things, e.g. booking a room) Additional URL prefixes besides /export/ (e.g. /api/ ) Many things (like the export_* method name) can be changed easily when subclassing HTTPAPIHook Reading the existing code is the best way to get used to it!

26. EXTENDING THE API

27. Another example, using POST @HTTPAPIHook.register class AnnouncementHook(HTTPAPIHook): PREFIX = 'api' TYPES = ('announcement',) RE = r'set' GUEST_ALLOWED = False VALID_FORMATS = ('json', 'xml') COMMIT = True HTTP_POST = True def _getParams(self): super(AnnouncementHook, self)._getParams() self._message = get_query_parameter(self._queryParams, ['message'], '') def api_announcement(self, aw): am = getAnnoucementMgrInstance() am.setText(self._message) return {'message': self._message}

28. Adrian Mönnich Questions? https://github.com/ThiefMaster View the example code on GitHub: https://goo.gl/btdVc [Diff] https://goo.gl/wGJXG [Gist]