1. Reading for Fun – Official History
VistA*/U.S. Department of Veterans Affairs national-scale HIS
Steven H. Brown, Michael J. Lincoln, Peter J. Groen, Robert M. Kolodner
International Journal of Medical Informatics 69 (2003) 135/156
VistA Document Library (VDL)
www4.va.gov/vdl

2. The VA Data Lifecycle - Internals, Data Flows, and Business Intelligence
(With Pharmacy Additions)
Richard Pham, PharmD
Enterprise Architect
OI&T Corporate Data Warehouse – Architecture

3. The Health Care System Is MUCH MORE COMPLICATED Than Other Business Processes (~7% of VistA)
Steve Anderson and Dan Hardan conversation about how ridiculously complicated BI would work from ANY medical setting.

5. Each Process Is A World Unto Itself

6. DHCP/VistA/CPRS
VistA – Veterans Health Information Systems and Technology Architecture - Refers both to the architecture and the database which the architecture supports
DHCP - Decentralized Hospital Computer Program – The DOS (Unix-like) system where many of VistA’s non-clinical entries take place
CPRS - Computerized Provider Record System – A user-friendly GUI providing access to clinical order entry functions

7. Objectives
The main objective is to understand the data lifecycle of VA’s VistA/CPRS and the user experience of VistA/CPRS
A high-level overview of VistA Internals
Learn about data structures and outputs in VistA
Learn where data enters and travels throughout the VA
Try to make sense of data resources within the VA and how they are accessed

8. The VA Data Lifecycle

9. The VA Data Lifecycle

10. Core Patient Care Functionality
VistA is first and foremost an Electronic Medical Record. The architecture design supports veteran health care.

11. Core Patient Care Functionality
VistA Internals
DHCP
CPRS

12. VistA Internals 101 MUMPS Server and Operating System Kernel
“Three Wise Men (Managers)”
TaskMan
MailMan
FileMan
Modules

13. Massachusetts General Hospital Utility Multi-Programming System (MUMPS or M)
My definition in English: M is a programming language designed for hierarchical databases that is convenient for medical applications or anything else where speed and data storage upkeep are a problem and programmer intelligence/organization is not
My technical definition: M is a Turing-complete, low and high-level, imperative, machine-compiled (no longer interpreted) programming language utilizing a hierarchical global array file structure
Used commonly in healthcare and financial industry settings
Turing-Complete: GOTO, IF/THEN, WHILE, Peano Arithmetic
Low-level – DSM exposes VAX/VMS architecture
High-level – Programming in the kernel environment is hardware and OS-agnostic
Machine-compiled – All modern implementations are compiled, not interpreted (see compiler choices in the Kernel documentation)

14. Structure of The Veterans Administration Data Efforts (Late 1970s)
VHA Ancestor
Department of Medicine and Surgery (DMAS)
OI&T Ancestor
Office of Data Management & Telecommunications (ODM&T)
VHA-OI Ancestor
Computer Assisted System Staff (CASS)

15. Comparing The Two Offices
CASS
ODM&T
Decentralized design philosophy
Rapid, agile development
SME-involved development
Centralized design philosophy
Bureaucratic, process-focused development
Development without SME’s

16. Highlights of ODM&T Development
Took 6 years to deploy APPLES Pharmacy at 10 sites
A 1980 paper detailing ODM&T’s transactional patient treatment file (PTF) system promised an interactive national solution by 1990.
Navigating the mandated 17 steps between system specification and deployment alone is said to have required at least 3 years.

17. Beginnings of DHCP
There were subject matter experts that believed that they could put out useful applications faster than the ODM&T sloth
Development of the testing and principles was done unofficially throughout the late 1970s
The Second

18. Original DHCP Design Principles
A commitment to rapid prototype development
All use ANSI MUMPS
Modular Design
Actively Maintained Data Dictionary
Code Sharing/Portability
Involve the SME’s

19. DHCP Kernel
Functions as both an operating system for VistA applications and an M virtual machine
Kernel shields DHCP modules from needing to know hardware and OS configurations on the server
Isolates M to the ANSI standard (1995)
Provides a toolbox of standard functions for most programmers

20. VistA to Relational Database Terminology
VistA (Example)
Relational Database (Example)
Namespace (VHAFRE)
Database (VA Fresno)
“Package” – Not hardcoded
Schema (RxOutpatient)
File (50.68 – VA PRODUCT)
Table (NationalDrug)
Field (.01 – NAME)
Column (DrugNameWithDose)
Domain (cardinal/decimal, setofcodes, freetext/wordprocessing)
Field Type (numeric, boolean, varchar)
Internal Entry Number (IEN or .001)
~Key (9722)
Record
Tuple/Row (ISOSORBIDE MONONITRATE 120MG TAB,SA)

21. MUMPS Classic Database
One Data Type
String (Text)
Other types
Cardinal Numbers
Float Numbers
$H Dates
One Data Storage Type
Multidimensional Array aka Globals
Dynamic (duck) typing
The logical database of a GT.M process consists of one or more global variable name spaces, each consisting of unlimited number of global variables. For each global variable name space, a global directory maps global variables to the database files where they actually reside. An unlimited number of global variables can fit within one database file; a global variable must fit in one database file.
A database file consists of up to 224M (276,168,704) database blocks. A database block is a multiple of 512 bytes, with a maximum size of 65,024 bytes. Commonly used block sizes are 4KB, 8KB and 16KB - so, with an 8KB block size, an individual global variable can grow to 1,792GB. A global variable node (global variable, subscripts plus value) must fit in one database block and each block has a 16 byte overhead. So, the largest node that will fit in a database with a 4KB block size is 4,080 bytes. A key (global variable plus subscripts) can be up to 255 bytes.
The database engine is daemonless and processes accessing the database operate with normal user and group ids - a process has access to a database file if and only if the ownership and permissions of that database file (plus any layered access control such as SELinux permits access). Each process has within its address space all the logic needed to manage the database, and processes cooperate with one another to manage database files. When a database file is journaled, updates are written to journal files before being written to database files, and in the event of a system crash, database files can be recovered from journal files.

22. VistA Data Organization
Namespace
File
Field
Record
654 (VAMC Reno)
File (GMR Vitals)
Field 0.1 (DATE/TIME VITALS TAKEN)
IEN-1, BP, 140/90
Most Files have an entry at the Field called “IEN” or “Internal Entry Number” as an identity key to mark the record as unique
The equivalent of Australian fishing

23. Upside of Using Globals
Faster - No joins
Faster – All parameter pointers built in
Faster – Direct and planned programmatic access to database (Look at SQL execution plans)
Less Data Storage Overhead and faster paging – If the data point does not exist in the array, there does not need to be a fixed point like in relational

24. Downside of Using Globals
No Intrinsic Structure and No Enforcement* - M believes whatever you put into the globals (most M programmers view this as an advantage while relational programmers have an MI)
ACID-compliance not mandated
(Il)logical data structures guaranteed – There are many interesting* ways that the M programmers modeled the data that does not make sense to later viewers
(PCMM, Lab, Pharmacy, anyone?)
“Like a puppy, M accepts your love and hate unconditionally and gives back unconditionally too…” - VistA programmer
3NF gives constraint to what you should build in a relational form.
If you did something wrong, well, M accepts it anyway.

25. MUMPS Quirks Whitespace (Space) matters
Requires knowledge of kernel and sometimes lower-level concepts
Programming Without Type or Structure Enforcement
VA programming standards and conventions
(Assembly is a better prerequisite than other languages)
(The VAX/VMS version of M will let you fool around with hardware calls, bypassing the OS)
(2 spaces after Q…)

27. The Three Wise Men (Managers)
TaskMan – The man(anger) that schedules tasks to the kernel
MailMan – The man(anger) that messages between the user, TaskMan, and any other two-way communication between packages
FileMan – The man(anager) that controls internal file (data structure) interactions

28. TaskMan TaskMan handles application processing:
Creation of application processing tasks
Scheduling these tasks
Monitoring health/statistics of these tasks
If kernel is the brain, then TaskMan is the body of the operation
If programming, NEVER EVER use the TaskMan global. This subverts TaskMan’s scheduling queue, and can cause a system memory leak. Use the calls instead…

29. MailMan
VistA needs a way to pass and receive data from the database to other areas
MailMan fulfills this function in the pre-TCP/IP days
“Electronic mail” doesn’t mean just
Practically any message between the database and anyone else (the end-user, another site, or application, etc.) can be moved this way
Gives programmers methods to both receive and return data to the database

30. FileMan
A higher-level method to access the VistA database without exposing a programmer interface
Mostly menu-driven
One can use limited programming
Serves as the model for all other modules that interact with the VistA database

31. ODM&T Initial Action Plan To DHCP Development (1980)
Ordered that development stop
Fired the developers
Removed the hardware
Cut the DMAS budget so it would never happen again…

32. The official history

33. Development Goes Underground
Developers that survived the ODM&T purge continued their work as a black project in DMAS
During 1980 and 1981, the survivors (Underground Railroad) continued work on developing modules for system integration

34. Modules Modules are programmed to interact with the VistA database
Most use Fileman as a model for programming

35. Some of the Many Modules
Medicine
Surgery
Dentistry
Nursing
Pharmacy
Laboratory
Care Management
Patient Care Encounters
ADT
Mental Health
EDIS
Oncology
Nutrition and Food Service
Imaging/PACS
Prosthetics
Not really in the scope of this presentation to cover each module .
Try the VistA Documentation Library: Or
VHA eHealth University (VeHU):

36. Acceptance and DHCP 1.0
Once there was a critical mass of packages that were shown to be useful, the tide turned and the project was blessed…
Initial testing done in
1.0 installation was in 1985
Most of the underlying packages can still be recognized by the original programmers
And if you’re awesome like Michael Distaso, then you can even get a class of API’s named after you.

37. Special Topic – The Pharmacy Package (File 50 Series)
Where are the files?
File 50 Series is the main line, though there are other places
How many files are there?
Look at screen capture on next side for File 50 series.
How many columns?

38. How Many File 50 Tables Are There? (1096 – 659 = 437 Tables!!)

39. How Many File 50 Series Columns Are There? (8526 – 5352 = 3172!!)

40. Further Information On The Background
For the VA Base M Training
For the VA Programming Standards and Conventions
For the VA Document Library

41. Computerized Patient Record System (CPRS)
A Real-Time Order Checking System that alerts clinicians during the ordering session that a possible problem could exist if the order is processed
A Notification System that immediately alerts clinicians about clinically significant events
A Patient Posting System, displayed on every CPRS screen, that alerts clinicians to issues related specifically to the patient, including crisis notes, warning, adverse reactions, and advance directives
The Clinical Reminder System, which allows caregivers to track and improve preventive health care for patients and ensure timely clinical interventions are initiated
Remote Data View functionality that allows clinicians to view a patient’s medical history from other VA facilities to ensure the clinician has access to all clinically relevant data available at VA facilities
CPRS DOES NOT STORE DATA!!!

42. CPRS Internals Written in Embarcadero Delphi (NOT in MUMPS)
Connects from the Graphic User Interface to the VistA database using a Remote Procedure Call (RPC) Broker
This Remote Procedure Call Broker translates instruction sets from other languages into M

43. Present State of VistA Large MUMPS database Many processes
Over 50+ Main Clinical Packages
Over 10,000 + Tables
Each medical center runs somewhere between 2-4 TB worth of data over 30 years (mostly imaging)
Many processes
300+ MB of running executable at any given time
Over 20,000 subroutines (VDL)
Many simultaneous users

44. The VA Data Lifecycle

45. National Analytic Systems
A list of systems that support policy, planning, and congressional needs
There are more extracts than this, but I have chosen the most common ones…

46. Deborah Richardson

47. Systems to Support Planning
Decision Support System (DSS)
Supports accounting and costing for the OIG, GAO, CBO, and other auditing agencies
Allocation Resource Center
Supports personnel and resource allocation at the medical center level
Workload capture, resource allocation
Basis for the VERA (VA’s Fund Control Point) Model

48. VistA DSS Data Feeds ADM Admissions CLI Clinic Visits
DEN Dental ECS Event Capture System
IVP Pharmacy: IV LAB Lab
LAR Lab Results MOV Patient Movement – Inpatient
MTL Mental Health Test NUR Nursing
PAS Patient Assessment PRE Pharmacy OP
PRO Prosthetics RAD Radiology/Nuclear Med
SUR Surgery TRT Patient Treatment Specialty
UDP Pharmacy Unit Dose IP
Deborah Richardson’s Slides
From VistA
SCHEDULE is primary purpose of slide
DSS processing month is 2 months behind -- give example
Handout will be provided with schedule of all extracts referenced in this briefing.
Can also be downloaded for the NDS website
May be transmitted by site at any time of the month, ideally
around the 25th of the month prior to the processing month.

49. Systems to Support Research
National Patient Care Database
An integrated set of data that captures a patient’s care encounter with the VA
Corporate Data Warehouse –
A near real-time accumulation of much of the same data
The result of the Health Data Repository process

50. NPCD Data Flow Diagram z900 NPCD DMI
VistA
MailMan
NPCD data is sent from the facilities to the AAC via MailMan messaging
Once a message reaches the AAC MailMan server,
It automatically moves to the Data Management
Interface System (DMI)
Acknowledgement message
MailMan
Message
NPCD and other applications retrieve their respective
data from DMI for use
Acknowledgement messages are sent to facilities
z900
Data extracted &
backed up nightly M-F
Data extracted
by application
Austin
MailMan
Server
Mention process similarities with DSS and other systems
Process originates with a VistA MailMan message
Austin MailMan server receives messages
Message is sent to DMI
Data goes into predefined tables or queues within DMI
Applications must retrieve the data -- DMI does not push data out to the apps.
HL7 data are sent to the NPCD Oracle Database
Data is backed up in DMI and within the application
Acknowledgement Reports (or OPC) sent daily using reverse process
Data Stream
DMI
NPCD
Acknowledgement message
Acknowledgement message
Data received
in DMI 24x7
HL7 data to Oracle DB

51. NPCD Processing Daily Data Loading SAS UNIX z900 (MAINFRAME) WINDOWS
Oracle on Unix
NPCD
UNIX
Master
Extract File
(MEF)
SAS
z900 (MAINFRAME)
VSSC/
KLF Menu
WINDOWS
National Patient Care Database Processing begins with daily data loads
Flat files are indexed and loaded into the Oracle database database daily
The Master Extract is a weekly process performed to get data out of the database and prepare it to go into the SAS data sets
SAS Data Sets are extracted approximately every two weeks from the NPCD database
Once data is in the SAS data sets it is available for analysis
SAS Data Sets are stored on the AAC mainframe by fiscal year
Are available for use by the KLF Menu Reporting Tool
Process takes 1 to 2 weeks depending on when data arrives at the AAC
All NPCD data processes are HUGE
As year progresses it takes longer to go through process
Another layer of complexity is added by data being routed through multiple environments including Unix, Mainframe, and Windows
Flat files are indexed and loaded into the database daily
DSS
data
extracted
Data is checked for duplicates bi-monthly
Data is extracted and filtered for reporting twice a month

52. Resources for Further Information
VA Information Resource Center (ViREC)
National Patient Care Database –
(Internal)
National Data Systems (NDS)
(Internal)

53. Secrets of the VA Data Universe
This was an extremely brief introduction to a complicated area
I have another presentation on the availability of databases in the VA and how to access them for operations and/or research

54. The VA Data Lifecycle

55. Regional Remote Data Processing Center Shadow Systems
A offsite backup process to ensure continuity of operations for VistA Patient Care

56. Regional Data Processing Centers (RDPCs)
Read only backup VistA systems are set up to take journaling files
When a record is written or altered to a local medical center’s VistA, a journal file with that entry is prepared and sent to a Regional Data Processing Center
This maintains an active backup in case the local medical center’s VistA goes down
VistA goes down
OpenVMS goes down

57. Regions and RDPCs Region I RDPC – Sacramento (SAC) and Denver (DEN)
Region IV RDPC – Philadelphia (PHI) and Brooklyn
VISNs are numbered from the Northeast to the Southwest
Regions are numbered from the West to the East

58. The Day VistA Died

59. Northern Cal Without VistA
The medical staff was forced to write discharge instructions and notes on paper.
The electronic lists of instructions and of medications were not available for the patients being discharged.
Patients being discharged could not be given follow-up appointments at the time of discharge.
The appointments had to be made later and the patient notified by phone.
There were delays in obtaining discharge medications and patients remained on the wards longer than would normally be required.
The nurses administered medications to the patients and used the paper Medication Administration Record, or MAR, to record the administration events.
Initial medication passes were interrupted and delayed until the paper copies of the (MAR) could be printed.

60. RDPC Denver and Brooklyn

61. The VA Data Lifecycle

62. Business Intelligence

63. Business Intelligence in the VA – Making the Data Work For Us
VistA has a wealth of clinical and administrative data available
In the past, giving a value-added, timely VistA dataset was hard
Querying the active system with minimal impact
Needed an interface between M and analyst languages (SAS, SQL, etc.)
Easy to read reports was hard to build

65. Corporate/Regional Data Warehouse
Takes a copy of the journal file that goes into the backup shadow system
Translated from the M array to a relational database format using Intersystems Cache’s class mapping program
Staged in a Feeder-Collector system for collection
Indexed and value-added columns produced and loaded to an VISN RDW Server

66. VHA Business Owners/SME’s Corporate Data Warehouse
CDW Governance
Communicates
Organizational Priorities
VHA Business Owners/SME’s
10N, OIA, VBA
CDW Governance Board
Sets and monitors domain, work priorities, and timelines for completion.
Organizes SMEs and Data Stewards
VHA-OI
Data Quality
OI&T
Corporate Data Warehouse
Provides Documentation and Clarification of
Business Logic

67. CDW Governance Is In VHA’s Hands
Ordered By VHA
Domain and Work Prioritization By CDW Governance Board
Chair – KLF (OIA)
Vice-Chair – Larry Mole (Public Health SHG)
Monitored and Accountable To VHA
Project management provided by John Quinn (National Data Systems) and KLF (OIA)
Supported By VHA
OI Data Quality
Business Owners
PBM’s Data Steward is Rob Silverman

68. “As the number of eyes goes up, the number of bugs goes down.”
Writing documentation about the business logic of the files and fields
Answering end user questions about the data
Data validation
Preferably before
Inpatient Pharmacy
ADR/Allergy Package

69. 1st category models are simple – V Health Factor
Source Mapping
FMFile
FMField
ResolveFld
DWTableName
DWFieldName
V HEALTH FACTORS
HEALTH FACTOR
HealthFactor
HealthFactorTypeIEN
0.01
HealthFactorType
PATIENT NAME
PatientIEN
EVENT DATE AND TIME
EventDateTime
VISIT
VisitVistaDate
VisitDateTime
LEVEL/SEVERITY
LevelSeverity
VisitIEN
ENCOUNTER PROVIDER
EncounterStaffIEN
COMMENTS
Comments

70. 2nd category models require transformation – Prescription
and 1st fill
Prescription Only
All Fills
Refill
Partial Fill
Data Warehouse
Fileman

71. 3rd category models not usable without transformation - PCMM

72. Levels of Data National – Corporate Data Warehouse (CDW)
Region – Regional Data Warehouse (RDW)
VISN – VISN Data Warehouse (VDW)
Medical Center – Local Data

73. Entities Who Produce Business Intelligence Products
National – VSSC, PSSG, DMDC, HEC, ARC, DSS, BIPL, OQP, PCS, PBM
Region – Regional BISL Teams
VISN – VISN Data Warehouse, VISN PBM
Local – DSS
Bolded are ones that have substantial resources in clinical business intelligence
PSSG handles much of the GIS and Statistical Demography for the VA
PSSG- Planning Systems Support Group
VSSC - VHA Support Service Center
DSS – Decision Support System
BIPL – OI&T Business Intelligence Product Line
OQP – Office of Quality and Performance (10Q)
PCS – Patient Care Services (10A)
PBM – Pharmacy Benefits Management Services

74. Data Access
VISN and Station Level – Contact Your VISN Database Manager
Regional/Corporate Access – Contact NDS for the 9957 Permissions

75. Operational Challenges of VistA
System Resources
$8 Billion investment over 20 years
New needs for new domains
MUMPS Programmers must be internally trained (and many of them are retiring or dying)
Communication with Other Systems
HIMISS compliance with data interchange
E-functions (billing, prescribing, verification)
Interagency Cooperation – DoD and NHIN
Business Intelligence
Closing the data lifecycle and bringing back clinical data for knowledge discovery

76. Why Is Pharmacy ALWAYS Picked As An IT Test Case Project?
Pharmacy is data savvy
Local data quality control from ADPACs
Federated data quality control from VISN PBM, CMOP, PBM SHG
Pharmacy is one of the few domains that have active business logic SME’s
Pharmacy did not contract institutional memory out the door
Pharmacy gets things done
Pharmacy has more technology success stories as a collective than any other PCS office (BCMA, Automation, Central Fills)
Pharmacy actively mines data

77. Acknowledgments Kernel – Jack Schram (Oakland OIFO)
SQLI – Ellen Zufall (SF IRMS)
RPC Broker and MUMPS coding – Perry Richmond (VISN 18 BI)
Regional Data Process – Vincent Bui and Ken Koenig (Region I SQL Back Office Team)

78. Acknowledgments OI&T Business Intelligence Product Line (BISL)
Jack Bates – Manager, OI&T BIPL
Stephen Anderson – Lead Data Architect
Mike Baker – Lead ETL Architect
Denver Griffith/Ken Fuchsel – Server Administrators
Dave Fackler
Ron Talmage
Dan Hardan, Jeff King, Jeff Price

79. Questions