SPARK Search Engine

Martijn Harthoorn Programmer at Furore Implementer of the Search Engine of SPARK The work after the question mark. Who am I?

The place of Search RESTService Storage Index & Search MongoDB Spark

Paradigm FHIR client should be easy. FHIR server needs to solve the complex issues. Search Search has some…

First there was Storage Search Then there was Search

Connectathon To test a client – you must have a tested server To test a server – you must have a tested client “One fool can ask more questions than seven wise men can answer”

Connectathon “But what if you are wrong?”

History Version 1. -A Generics based implementation -On top of the FHIR data model. -Programmed per search parameter programming. -No meta data available yet. -No indexing. -Slow.

History Version 2. -Data Model independent, -Meta data not available - manually added -Lucene.NET as indexer (Index in Lucene, Database in Mongo) -Fast -Standardised all parameter specifics into standard “modifiers”. -All Code based on search parameter types. -Joins are client side

History Version 3. -Modified to store the Lucene index in Mongo -Index storage unreliable. -Never saw light of day

History Version 4. CURRENT -Index storage to a dedicated Mongo collection -Build expression tree from parameters -Chained parameters have full functionality (modifiers, operators) -Joins are client side

Indexing Why indexing?

Why indexing

Why indexing

Indexing. HOW-TO You DO want A de-serialized data to an object with all values strongly typed. You DON’T want to spend time analyzing and interpreting JSON and/or XML. 1.Harvest the Resource 2.Determine data type 3.Groom your data 4.Store data in Index

Indexing. 1. Harvesting Resource: Patient Search parameter: family Searches for the family name and prefix of every HumanName that is registered with a Patient. Usage:

Indexing. 1. Harvesting Patient List Name (HumanName) Family Prefix Given Suffix Resource: Patient Search parameter: family Using the Visitor pattern Path from Meta data: "patient.Name.Prefix" "patient.Name.Family"

Indexing. 2. Determine data type > patient (Patient) > Name (HumanName) > LastName (string) Data type: string Search parameter type: string Selected indexing method: -Single value – as string -More values – as string array

Indexing. 2. Determine data type > patient (Patient) > Gender (Coding) > Coding (List ) > Code (CodeableConcept) Data type: Code Search parameter type: Token Selected Indexing method: Store in an array each codeable concept -System (uri) -Code (string) -Display (string)

Indexing. 3. Groom your data -Remove dashes, dots, slashes from dates etc. -If you implement a like search from the left side, you might want to split names at the dash in to multiple hits.

Indexing. 4. Store in the index FieldValue Resource"Patient" Local IDpatient/1 Level0 Family["LaVaughn", "Robinson", "Obama"] Given"Michelle" Gender[ { System: “…”, Code: “..”, Display: “..” }, … … * Level The patient is not a contained resource (level 0) * Family In Mongo you can store an array that can be searched like a normal string.

Future Version 5. NEXT -All parameters based on FHIR data types? -Joins using Mongo Map-Reduce?

Complexity So what is the issue?

Complexity Include & Chained parameters -Joining over references return multiple resource types -Client side (not in Mongo database) joins

Complexity Transactions -FHIR has bulk POST -Split between Indexing and storage

Complexity Multiple types Some properties do not have a fixed type. Example: observation.value Can be a: -CodeableConcept -String -Quantity (number + unit)