1. BabyTalk: Generating English Summaries of Clinical Data
Ehud Reiter
Univ of Aberdeen, CS Dept

2. Structure Background: data-to-text Babytalk project
Results of first evaluation
Current work

3. What is data-to-text
Goal: generate English summaries of non-linguistic data
Numerical weather predictions
Medical records
Statistics
Etc

4. Simple Example: Weather Forecasts
Input: numerical weather predictions
From supercomputer running a numerical weather simulation
Output: textual weather forecast
We’ve developed several systems
Two used commercially (oil rig, road gritting)
Users prefer some gen texts to human texts!
Demo of pollen system on our webpage
So have others (FoG, MultiMeteo, …)

5. Pollen forecasts
Grass pollen levels for Tuesday have decreased from the high levels of yesterday with values of around 4 to 5 across most parts of the country. However, in South Eastern areas, pollen levels will be high with values of 6.

6. Other data-text apps Medical: to-be-discussed
Assistive technology: help blind people access statistical data
Financial: summarise stock-market data
Education: Summarise assessment results, help write stories
Engineering: Sum. gas-turbine data
Etc

7. Why is data-to-text useful
The world is drowning in data
NLP researchers talk about problems of too much text, but data problems are worse
Texts are at least read by someone (writer)
Most data is automatically collected and never looked at by a human

8. Data overload Sensor recording 2 bytes/second Simulations
170KB/day
63MB/year
Millions of sensors in hospitals, jet engines, …
Simulations
Weather: 30MB for one day in one UK county, from one model
Climate models: petabytes of data
Too much data, need better tools for utilising!

9. Decision Support Data often used for decision support
Medical: help doctors make decisions
Weather: helps staff on offshore oil rigs plan their operations
Engineering: help plan maintenance
Etc
Often under time pressure
Make a decision in 3 min, here is 30MB of data to help you

10. Using data for decision support
Alarming
Trigger alarm if value exceeds threshold
Or other such simple rule
Works, doesn’t get full value from data
Visualisation
Show data to experts visually
People like this, unclear how much it helps, especially when massive amount of data

11. Using data for decision support
Knowledge-based systems
Feed data into an expert system which makes recommendations based on it
Can work in some contexts, but problems
Domain experts dislike being told what to do
Often key data not available to KBS
Can be brittle, fragile

12. Data-text for decision support
Idea: use KBS, NLP tech to generate a short text summary of a data set
Intermediate between KBS and visualisation
Use domain reasoning to highlight key info, infer causal links, add background know
But stick to describing data, don’t tell experts what to do!

13. Data-text for decision support
vs alarms: deeper info
vs visualisation
Just key facts, not everything
Supplemented with causal links, etc
vs KBS
More acceptable to users
More robust, since not useless if missing some key data or knowledge

14. Data-text for decision support
Above is still somewhat speculative
But people in many domains are interested in exploring the concept to see if it works
Esp since current situation is so bad!
Of course other uses of data-to-text
Assistive technology, education

15. Language and World How does language relate to the world?
Data-to-text is a great way of exploring this
The real reason I got into this…

16. BabyTalk
Goal: Summarise clinical data about premature babies in neonatal ICU
Input: sensor data; records of actions/observations by medical staff
Output: multi-para texts, summarise
BT45: 45 mins data, for doctors (completed)
BT-Nurse: 12 hrs data, for nurses
BT-Family: 24 hrs data, for parents
BT-Clan: 24 hrs data, for other friends, family
Bt-Doc: several hrs data, for doctors

17. Neonatal ICU

18. Peripheral Temperature (TP)
Baby Monitoring
SpO2 (SO,HS)
ECG (HR)
Peripheral Temperature (TP)
Core Temperature (TC)
Transcutaneous Probe
(CO,OX)
Arterial Line
(Blood Pressure)

19. Input: Sensor Data

20. Input: Action Records FullDescriptor Time
SETTING;VENTILATOR;FiO2 (36%)
10.30
MEDICATION;Morphine
10.44
ACTION;CARE;TURN/CHANGE POSITION;SUPINE
ACTION;RESPIRATION;HAND-BAG BABY
SETTING;VENTILATOR;FiO2 (60%)
10.47
ACTION;RESPIRATION;INTUBATE

21. BT45 texts Human corpus text
At 1046 the baby is turned for re-intubation and re-intubation is complete by 1100 the baby being bagged with 60% oxygen between tubes. During the re-intubation there have been some significant bradycardias down to 60/min, but the sats have remained OK. The mean BP has varied between 23 and 56, but has now settled at 30. The central temperature has fallen to 36.1°C and the peripheral temperature to 33.7°C. The baby has needed up to 80% oxygen to keep the sats up.
Computer-generated text
By 11:00 the baby had been hand-bagged a number of times causing 2 successive bradycardias. She was successfully re-intubated after 2 attempts. The baby was sucked out twice.
At 11:02 FIO2 was raised to 79%.

22. Babytalk architecture
Signal analysis: patterns, trends
Data interpretation: based on medical knowledge (like expert sys)
Doc planning: select and structure events to be mentioned
Microplanning: choose words, syntactic structures, referring exp
Realisation: generate actual text

23. Signal Analysis Detect trends, patterns, events, etc Detect artefacts
Blood oxygen levels increasing
Downward spike in heart rate
Detect artefacts
Changes due to sensor problems
Plenty of algorithms exist for this
Will not further discuss here

24. Data Abstraction Detect higher-level events in the data
Sequence of bradycardias (downward spikes in HR)
Determine medical importance
Bradycardia more important if simultaneous desaturation (downward spike in SO)
Medical KBS

25. Data Abs: Links Between Events
Infer links between events
Blood O2 falls, therefore O2 level in incubator is increased
HR up because baby is being handled
Morphine given as part of the intubation procedure
Very imp, much of value added of text
Helps readers build good mental model of what is happening to the baby

26. Document Planning First NLP stage Decide what events to mention
Decide how these are ordered and organised

27. Content Determination
First approach: Include most medically important events
Also include moderately important events which are linked to very important events
Doesn’t always work

28. Problem: Continuity Omitting intermediate events confuses readers
Example: TcPO2 suddenly decreased to 8.1. SaO2 increased to 92. TcPO2 suddenly decreased to 9.3
There is a gradual rise in TcPO2 between the sudden falls
This is less important medically
But important for reader’s comprehension

29. Document Structure How do we order/group events
By time
By medical importance
By body subsystem (eg, respiration)
Initially focused on time, but users want more emphasis on subsystem
Eg, first a “scene” about respiration, then a “scene” about thermoregulation
Not constant shifting between two

30. Doc Planning: Narrative
High-level analysis: need to do a better job of generating a “story” from the data
Link events together
Include events needed for story progression even if not important
“Scene” structure
Qualitative observation by users

31. Microplannig Second NLP stage
Choose words and syntactic structure to express information
Aggregation
Reference

32. Challenge: Time Need to communicate temporal info
Enough so that readers can interpret the data
Not too much, text becomes unreadable
Imagine story with “At John left home. At he met Mary in the pub. At 10.39…”

33. Tenses Use Reichenbach model Usually worked, sometimes failed
Speech time: time of report being read
Event time: time of event being described
Reference time: determined using a salience model
Similar to resolving anaphoric reference
Usually worked, sometimes failed
Need better model for reference time

34. What does event time mean?
Sometimes explicit time given for event
Supposed to be start time of event, sometimes misinterpreted
Ex:”After three attempts, at a peripheral venous line was inserted successfully.”
13.53 refers to time of first (failed) attempt
Start of LINE-INSERT-ATTEMPTS event
Readers interpret as time of final (succ) attempt
Need better linguistic model of time
Linguistic temporal ontology (Moens Steedman)?

35. Lexical Choice
Need mechanism to map domain events (instances in a Protégé ontology) to linguistic structures
Use JESS rules
Lexical info from Verbnet, NIH lexicon
Engineering challenge
Relate to Sheffield work on NLG/ontologies

36. Vague language Human texts are full of vague language
Ex: There is a momentary bradycardia
What does “momentary” mean?
Our models of this are very crude, need to be improved!

37. Realisation Last NLG stage Generate actual text, once choices made
Use Aberdeen simplenlg package
Will not further discuss here

38. BT45 Evaluation
Showed 35 medical professionals 24 scenarios in 3 conditions (8 of each)
Visualisation of medical data
Textual summary (manually written)
Textual summary (from BT45)
Asked to make a treatment decision
Limited to 3 minutes
Measured correctness (against gold stan)
Off-ward, using historical data
So no other knowledge about baby

39. Free-text comments
Comments were not solicited, but were recorded if made
Most important were
Better layout (eg, bullet lists)
Continuity (as mentioned before)

40. Decision-Support results
No sig difference in time taken
Avg decision-quality (scale -1 to 1)
Human texts: 0.39
Computer texts: 0.34
Visualisation: 0.33
Human sig better than comp, visual
No sig diff comp, visual

41. Results by subject type
Analysis by type of subjects
Human texts especially good for junior nurses (ie, least experienced subjects)

42. Results by scenario Each scenario had a main target action
8 different ones
Computer texts as good as human texts for five of these; worse for three
No action, manage temperature, monitor equipment
These relate to specific problems in the system, which can be fixed

43. Target Actions with Poor Perf
No action: Needs high-level summary, not blow-by-blow event description
Manage Temperature: Two temp channels, need to describe together
Monitor equipment: Need to mention (not ignore) sensor artefacts

44. Summary
Good performance with human texts shows textual presentation is effective
Also seen in previous study
Babytalk as good as visualisation, could make better by addressing above issues
Even now giving users BabyTalk text as supplement to visualisations could help

45. Current Work BT-Nurse: shift summaries for nurses
Use live data from current babies
Evaluate on ward, using babies that subjects (nurses) actually looking after
Focus on info relevant to nurse shift planning, not real-time decision support
Longer time period (12 hrs)
Need more sensor abstraction
Longer texts (multi-page)

46. Current Work BT-Family: information for parents
Estimate how stressed parents are, use this to control content, phrasing
High stress means less content
Relate to Sheffield work on personality??
Express information in language which parents can understand, not medicalese

47. Current Work BT-Clan: Information for friends, family
Social networking perspective: encourage useful support, minimise hassle of dealing with numerous inquiries
Parents decide what to tell people
Intentional deceit: if granny is frail, don’t tell her bad news
Info about parents as well as baby

48. Research agenda Detecting complex events in the data
Integration with medical guidelines
Better use of vague language
Better stories
Role of text in interactive multimodal information presentation system
Try in domain of assisted living