Actuarial IQ (Information Quality) CAS Data Management Educational Materials Working Party

Actuarial IQ Introduction “IQ” stands for “Information Quality” Introduction to Data Quality and Data Management being written by the CAS Data Management Educational Materials Working Party Directed at actuarial analysts as much as actuarial data managers: what every actuary should know about data quality and data management

Objectives Introduce Working Party Paper “Actuarial IQ” Justify our care about information quality: industry horror stories Explain that information quality is not just about input data quality: depends on all stages: input process, analysis process, output process Suggest what can be done to improve information quality: (it starts with YOU!) Encourage actuaries to become data quality advocates

Why do we care? Information quality horror stories: Insolvencies Run-offs Mis-pricing Over-reserving (Loss of bonuses) GIRO working party on data quality survey: On average any consultant, company or reinsurance actuary: spends 26% of their time on data quality issues 32% of projects affected by data quality problems

Actuarial IQ Key Ideas Information Quality concerns not only bad data: information which is poorly processed or poorly presented is also of low quality Cleansing data helps: but is just a band aid There are plenty of things actuarial analysts can do to improve their IQ There are even more things actuarial data managers can do Actuaries are uniquely positioned to become information quality advocates CLEANSING: Good data management, not data cleansing, is the long term solution.

Why Actuaries? both: No one else is better information consumers and information providers have knowledge of data and high stakes in quality skillful and influential No one else is better

Working Party Publications Book reviews of data management and data quality texts in the Actuarial Review starting with the August 2006 edition These reviews are combined and compared in “Survey of Data Management and Data Quality Texts,” CAS Forum, Winter 2007, www.casact.org This presentation is based on our Upcoming paper: “Actuarial IQ (Information Quality)” to be published in the Winter 2008 edition of the CAS Forum

Data Quality vs Information Quality Information quality takes into account not just data quality but processing quality (and reporting quality) DATA + ANALYSIS = RESULTS (Dasu and Johnson)

What is Data Quality? Quality data is data that is appropriate for its purpose. Quality is a relative not absolute concept. Data for an annual rate study may not be appropriate for a class relativity analysis. Promising predictor variables in Predictive Modeling may not have been coded or processed with that purpose in mind.

Presentation of Results Step 0 Data Requirements Data Flow Step 1 Data Collection Information Quality involves all steps: Data Requirements Data Collection Transformations & Aggregations Actuarial Analysis Presentation of Results To improve Final Step: Making Decisions Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Principles on Data Quality: Perspectives Step 0 Data Requirements Principles on Data Quality: Perspectives Step 1 Data Collection ASB – ASOP 23 – “Data Quality” Step 2 Transformations Aggregations CAS Management Data and Information Committee: “White Paper on Data Quality” Step 3 Analysis Step 4 Presentation of Results Richard T. Watson “Data Management: Databases and Organization” Final Step Decisions

Presentation of Results Step 0 Data Requirements ASOP No. 23 Step 1 Data Collection Due consideration to the following: Appropriateness for intended purpose … Reasonableness and comprehensiveness … Any known, material limitations … The cost and feasibility of obtaining alternative data … The benefit to be gained from an alternative data set … Sampling methods … Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

White Paper on Data Quality Step 0 Data Requirements White Paper on Data Quality Step 1 Data Collection Evaluating data quality consists of examining data for: Validity Accuracy Reasonableness Completeness Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Watson Step 1 Data Collection 18 Dimensions of Data Quality: Many overlap with previously mentioned principles. Others describe ways of storing data e.g. Representational consistency, Precision Others go beyond data characteristics to processing and management e.g. Stewardship, Sharing, Timeliness, Interpretation Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Data Requirements Step 1 Data Collection Redman: “Manage Information Chain” establish management responsibilities describe information chart understand customer needs establish measurement system establish control and check performance identify improvement opportunities make improvements Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Data Requirements Step 1 Data Collection Data Quality Measurement: Quantify traditional aspects of quality data such as accuracy, consistency, uniqueness, timeliness and completeness using a score assigned by an expert Measure the consequences of data quality problems measure the number of times in a sample that data quality errors cause errors in analyses, and the severity of those errors Use measurement to motivate improvement Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Metadata Step 1 Data Collection Big help in describing Data Requirements – Metadata! Data that Describes the Data Key Data Management Tool Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Example – Marital Status Step 0 Data Requirements Example – Marital Status Step 1 Data Collection What is in the Marital Status Variable? Step 2 Transformations Aggregations Single? Step 3 Analysis Married? Polygamist? Step 4 Presentation of Results Final Step Decisions

Example: What is the Marital Status Variable? Step 0 Data Requirements Example: What is the Marital Status Variable? Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements What Is In It? Step 1 Data Collection Business Rules Data Processing Rules Report Compilation and Extraction Process Other Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements What Is In It? Step 1 Data Collection Business Rules Data Elements Definition of Field, e.g., How Claims are Defined How Exposure is Calculated Format of Field Valid Values and Interdependencies Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements What Is In It? Step 1 Data Collection Data Processing Rules How Database is Populated Sources of Data Handling of Missing Data Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements What Is In It? Step 1 Data Collection Report Compilation and Extraction Process How Data is Selected or Bypassed Fiscal Period Accounting Date for Transactions Actuarial Evaluation Date Calculations Mappings Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements What Is In It? Step 1 Data Collection Other Process Flow Documentation Versioning Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Why Actuaries Need Metadata? Step 0 Data Requirements Why Actuaries Need Metadata? Step 1 Data Collection Better Analysis Avoid Being Mis-Informed about Data Variable and What it Represents Did Anything Change During the Experience Period Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Example of Metadata Step 1 Data Collection Statistical Plans in P/C Industry General Reporting Requirements Data Element Definitions Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Data Collection Step 1 Data Collection Data supplier management Let suppliers know what you want Provide feedback to suppliers Balance the following Known issues with supplier Importance to the business Supplier willingness to experiment together Ease of meeting face to face Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Transformations and Aggregations Step 1 Data Collection In this step data is put into standardized structures and then combined into larger, more centralized data sets “Actuarial IQ” introduces two ways to improve IQ in this step: Exploratory Data Analysis (EDA) Data Audits Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

EDA: Data Preprocessing Step 0 Data Requirements EDA: Data Preprocessing Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis There is a fair bit of work involved before data can be analyzed: Pick the fields you want Clean the dataset. E.g. what to do with records that have missing values, invalid values or outliers? Merge data from different sources, e.g. claims and premiums files Assemble the data in a structure suitable for the analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Overview Step 1 Data Collection Typically the first step in analyzing data Purpose: Find outliers and errors Explore structure of the data Uses simple statistics and graphical techniques Examples for numeric data include histograms, descriptive statistics and frequency tables Step 2 Transformations Aggregations Step 3 Analysis Data fields are generally divided into numerical and categorical. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Histograms Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis The main variable of the technique is the definition of the classes (a.k.a. bins, windows) in which the observations will be grouped. What makes this graph suspicious is the scale on the bottom axis: why does it span centuries? Step 4 Presentation of Results Final Step Decisions

EDA: Descriptive Statistics Step 0 Data Requirements EDA: Descriptive Statistics Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis Basic statistics can also help identify outliers and the shape of the data. For example, this table was generated by Microsoft Excel’s Analysis ToolPak At the bottom, you can see the two largest and two smallest observations, which seem reasonable. The mean is slightly above the median, which also seems reasonable. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Categorical Data Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis Examples of categorical data are gender and state. Often there is no a priori relationship between values of categorical data, so this affects how we explore and understand them. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Data Cubes Step 1 Data Collection Usually frequency tables Example 1: search for missing gender values Step 2 Transformations Aggregations Step 3 Analysis Data cubes a.k.a. OLAP cubes, pivot tables. These frequency tables can be multivariate. These techniques can also be applied to numeric data. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Data Cubes Step 1 Data Collection Example 2: identify inconsistent coding of marital status Step 2 Transformations Aggregations Step 3 Analysis 1 in 7 observations is missing marital status. There is inconsistency in whether marital status is coded by number or letter. Codes 4 and D may not be valid, or may not be used as often as they should. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Missing Data Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis These tests can be combined for faster exploration of the data. Here percentiles have been added for numerical data. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements EDA: Summary Step 1 Data Collection Before data is analyzed, it must be gathered, cleaned and integrated Techniques from Exploratory Data Analysis can be used to explore the data, and to detect missing values, invalid values and outliers We have briefly covered histograms, descriptive statistics and frequency tables Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Data Audits Step 1 Data Collection ASOP No. 23 does not require actuaries to audit, but good to understand Main Idea: compare the data intended for use to its original source, e.g., policy applications or notices of loss Top-Down: check that totals from one source match the totals from a reliable source Bottom-Up: follow a sample of input records through all the processing to the final report Step 2 Transformations Aggregations Step 3 Analysis INTRO: whereas EDA cleans a dataset, auditing influences the processes that generate the data. CONCLUSION: Data quality audits are a tool to both assess and monitor data quality. Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Analysis Quality Step 1 Data Collection Models Results Step 2 Transformations Aggregations Data Step 3 Analysis On its way to results data can be: Rejected wrong Format Underutilized wrong Model Distorted wrong Settings Step 4 Presentation of Results Final Step Decisions Analysis is a crucial component in the overall process quality

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Model Design quality Implementation quality Testing and Documentation Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Model Design quality Model Selection and Validation Parameters Estimation Verification Step 2 Transformations Aggregations Step 3 Analysis Did I use the right model ? Did I use the model right ? Step 4 Presentation of Results Model Performance Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Model Performance Models predict observable events. Outcomes can be compared to predictions leading to… Model’s Improvements Model’s Recalibration Model’s Rejection leading to… higher process quality. Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Model Design quality Implementation quality Testing and Documentation Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Implementation quality Programming languages: C++, VBA, SQL many books on good design patterns Formulae in a Spreadsheet - also programming no books on good design patterns Need good software design to simplify: Usage Testing Modifications / Improvements Recovery Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions (side benefit)

Presentation of Results Step 0 Data Requirements Model Quality Results do not belong to the template either Step 1 Data Collection Implementation quality Separation of data and algorithms Step 2 Transformations Aggregations data results algorithm(s) Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Each Step on its own tab Step 1 Data Collection Implementation quality Layering simplifies Navigation optimizes Workflow shortens Learning Curve Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Model Design quality Implementation quality Testing and Documentation Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Step 1 Data Collection Testing and Documentation Validation black-box treatment: comparing results with correct ones… Verification inside-the-box treatment: checking formulae… Should be integral part of development Should be performed by outsiders Should be well-documented Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality Database “Documenter” or “Diagram Builder” External Data Source definition Comments as Structured Attributes Excel’s “CTRL ~” Displays formulae texts Step 1 Data Collection Testing and Documentation Self-documenting features Database “Documenter” Excel’s named ranges and expressions External Data Range definitions Structured comments Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality One can link Document Properties to Spreadsheet Cells Step 1 Data Collection Testing and Documentation Version management Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Model Quality “Smart diagrams” can be automated Step 1 Data Collection Testing and Documentation Documenting Workflow Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Presentation Quality Step 1 Data Collection Data and Results Presentation caveats Data can be Mislabeled Mismatched Overlooked Misinterpreted Step 2 Transformations Aggregations We fight it with Unambiguous labeling Consistent calculations Attention grabbing tools Visualization and other explanatory techniques Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Presentation Quality Step 1 Data Collection Interpretation: Option’s 4 outcome can be anywhere from -20 to 109 million Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Presentation Quality Step 1 Data Collection Interpretation: Option’s 4 outcome is most probably 28 million with a spread around expected value Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Expected Spread Final Step Decisions

Presentation of Results Step 0 Data Requirements Decisions, Decisions Step 1 Data Collection It is a great feeling to know that YOU play a part (an important part!) in the Decision Making process The main goal of the fight for Information Quality is to provide solid foundation for Quality Decision Making Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Actuarial Data Management Step 0 Data Requirements Actuarial Data Management Step 1 Data Collection Step 2 Transformations Aggregations Step 3 Analysis Bridge between data requirements, data collection, data transformation and aggregation, and data usage Step 4 Presentation of Results Final Step Decisions

Critical Data Management Issues Step 0 Data Requirements Critical Data Management Issues Step 1 Data Collection Appropriateness of the collected data elements for the related analyses Quality of the collected statistical experience for the related analyses Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Data must be fit for the intended business use: Even high quality data when repurposed may result in lessened data quality Data should be obtained from the authoritative and appropriate source: Data should flow from underlying business processes – example, expecting claim adjusters to create injury diagnoses Know your data sources and their data quality and data management processes Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Common data elements must have a single documented definition and be supported by documented business rules: B.I.: business intelligence, bodily injury, business interruption, ... Incurred Loss: net as to deductible, net as to reinsurance, loss and expense, … Metadata must be readily available to all authorized users of the data: Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Data standards are key building blocks of DQ. Industry standards must be consulted and reviewed before a new data element is created: Common Insurance Terminology (i.e., provision vs. reserve; what is a claim) Coverage and Forms (i.e., motor vs. auto insurance) Process Standards: Application Forms, Report of Injury or Claim, Licensing, etc. Solvency Standards – greatly impacting actuaries – Solvency II, RBC Data Exchange/Reporting Standards – external sources vs. internal data Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Data Quality Standards – industry DQ tools and report cards Data Element and Code List Definitions Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Benefits of Industry Data Standards Step 0 Data Requirements Benefits of Industry Data Standards Step 1 Data Collection Submissions Insurance Carriers Regulatory Compliance Broker/Insurer Step 2 Transformations Aggregations Reinsurer Ins/Reinsurer Regulatory Authorities Claims STRAIGHT THROUGH PROCESSING Step 3 Analysis How to communicate? How to exchange data? Need standards! Claims Management Applications Auditing Straight Through Processing (STP) The use of common, industry standard data elements, throughout all interactions of all parties, in all insurance transactions or processes. STP allows data to flow effortlessly through the industry without redefinition, mappings or translations. Step 4 Presentation of Results Insurance Agency Service Providers Premium transactions Payment transactions Agent/ Producer Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Data should have a steward responsible for defining the data, identifying and enforcing the business rules, reconciling the data to the benchmark source, assuring completeness, and managing data quality. Data should be input only once and edited, validated, and corrected at the point of entry. Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Data Management Best Practices & Guiding Principles Step 0 Data Requirements Data Management Best Practices & Guiding Principles Step 1 Data Collection Data should be captured and stored as informational values, not codes. Data must be readily available to all appropriate users and protected against inappropriate access and use. Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements PWC 2004 Study Step 1 Data Collection “The key is to understand the impact data is having on your business and do something about it.” “Data quality is at the core – if you improve your data you will directly impact your overall business results.” Global Data Management Survey 2004, PriceWaterhouseCoopers Step 2 Transformations Aggregations Step 3 Analysis KM Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Conclusion Step 1 Data Collection Data Quality is a core issue affecting the quality and usefulness of the actuarial work product Data Quality is not just about how data is coded: phrase ‘information quality’ is coined to emphasize the impact of processes on the quality of final product Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Conclusion Step 1 Data Collection Ways to improve actuarial IQ Applying Data Quality principles Defining and using Metadata Measuring data quality to track progress and awareness of quality audit Utilizing Exploratory Data Analysis to identify outliers and explore the structure of a dataset Testing the quality of actuarial models Clarifying actuarial presentations and reports Employing Actuarial Data Management best practices Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Presentation of Results Step 0 Data Requirements Conclusion Step 1 Data Collection Expansions of actuarial frame of reference Data is a corporate asset that needs to be managed and actuaries can play a role Data needs to be appropriate for all of its intended uses Expansion of data quality principles to support these broader perspectives Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Final Step Decisions

Acknowledgement The working party would like to thank the Insurance Data Management Association (www.idma.org) for their help in: Developing a shortlist of texts that would be relevant to actuaries Reviewing our papers, and Providing some of the slides of this presentation

Author, Author… This presentation is a publication of CAS Data Management and Information Educational Materials Working Party: Keith P. Allen Robert Neil Campbell, Chairperson Louise A. Francis David Dennis Hudson Gary W. Knoble Rudy A. Palenik Aleksey Popelyukhin Ph.D. Virginia R. Prevosto Lijuan Zhang