1. SPARK Search Engine

2. Martijn Harthoorn Programmer at Furore Implementer of the Search Engine of SPARK http://spark.furore.com/fhir/patient?http://spark.furore.com/fhir/patient?... The work after the question mark. Who am I?

3. The place of Search RESTService Storage Index & Search MongoDB Spark

4. Paradigm FHIR client should be easy. FHIR server needs to solve the complex issues. Search Search has some…

5. First there was Storage Search Then there was Search

6. Connectathon To test a client – you must have a tested server To test a server – you must have a tested client “One fool can ask more questions than seven wise men can answer”

7. Connectathon “But what if you are wrong?”

8. History Version 1. -A Generics based implementation -On top of the FHIR data model. -Programmed per search parameter programming. -No meta data available yet. -No indexing. -Slow.

9. History Version 2. -Data Model independent, -Meta data not available - manually added -Lucene.NET as indexer (Index in Lucene, Database in Mongo) -Fast -Standardised all parameter specifics into standard “modifiers”. -All Code based on search parameter types. -Joins are client side

10. History Version 3. -Modified to store the Lucene index in Mongo -Index storage unreliable. -Never saw light of day

11. History Version 4. CURRENT -Index storage to a dedicated Mongo collection -Build expression tree from parameters -Chained parameters have full functionality (modifiers, operators) -Joins are client side

12. Indexing Why indexing?

13. Why indexing http://spark.furore.com/fhir/patient?provider.name:partial=Health

14. Why indexing http://spark.furore.com/fhir/patient?provider.name:partial=Health

15. Indexing. HOW-TO You DO want A de-serialized data to an object with all values strongly typed. You DON’T want to spend time analyzing and interpreting JSON and/or XML. 1.Harvest the Resource 2.Determine data type 3.Groom your data 4.Store data in Index

16. Indexing. 1. Harvesting Resource: Patient Search parameter: family Searches for the family name and prefix of every HumanName that is registered with a Patient. Usage: http://spark.furore.com/fhir/patient?family=White

17. Indexing. 1. Harvesting Patient List Name (HumanName) Family Prefix Given Suffix Resource: Patient Search parameter: family Using the Visitor pattern Path from Meta data: "patient.Name.Prefix" "patient.Name.Family"

18. Indexing. 2. Determine data type > patient (Patient) > Name (HumanName) > LastName (string) Data type: string Search parameter type: string Selected indexing method: -Single value – as string -More values – as string array

19. Indexing. 2. Determine data type > patient (Patient) > Gender (Coding) > Coding (List ) > Code (CodeableConcept) Data type: Code Search parameter type: Token Selected Indexing method: Store in an array each codeable concept -System (uri) -Code (string) -Display (string)

20. Indexing. 3. Groom your data -Remove dashes, dots, slashes from dates etc. -If you implement a like search from the left side, you might want to split names at the dash in to multiple hits.

21. Indexing. 4. Store in the index FieldValue Resource"Patient" Local IDpatient/1 Level0 Family["LaVaughn", "Robinson", "Obama"] Given"Michelle" Gender[ { System: “…”, Code: “..”, Display: “..” }, … … * Level The patient is not a contained resource (level 0) * Family In Mongo you can store an array that can be searched like a normal string.

22. Future Version 5. NEXT -All parameters based on FHIR data types? -Joins using Mongo Map-Reduce?

23. Complexity So what is the issue?

24. Complexity Include & Chained parameters -Joining over references return multiple resource types -Client side (not in Mongo database) joins

25. Complexity Transactions -FHIR has bulk POST -Split between Indexing and storage

26. Complexity Multiple types Some properties do not have a fixed type. Example: observation.value Can be a: -CodeableConcept -String -Quantity (number + unit)