1. Data and Applications Security Developments and Directions Dr. Bhavani Thuraisingham The University of Texas at Dallas Lecture #17 Data Warehousing, Data Mining and Security March 23, 2009

2. Outline l Background on Data Warehousing l Security Issues for Data Warehousing l Data Mining and Security

3. What is a Data Warehouse? l A Data Warehouse is a: - Subject-oriented - Integrated - Nonvolatile - Time variant - Collection of data in support of management’s decisions - From: Building the Data Warehouse by W. H. Inmon, John Wiley and Sons l Integration of heterogeneous data sources into a repository l Summary reports, aggregate functions, etc.

4. Example Data Warehouse Oracle DBMS for Employees Sybase DBMS for Projects Informix DBMS for Medical Data Warehouse: Data correlating Employees With Medical Benefits and Projects Could be any DBMS; Usually based on the relational data model Users Query the Warehouse

5. Some Data Warehousing Technologies l Heterogeneous Database Integration l Statistical Databases l Data Modeling l Metadata l Access Methods and Indexing l Language Interface l Database Administration l Parallel Database Management

6. Data Warehouse Design l Appropriate Data Model is key to designing the Warehouse l Higher Level Model in stages - Stage 1: Corporate data model - Stage 2: Enterprise data model - Stage 3: Warehouse data model l Middle-level data model - A model for possibly for each subject area in the higher level model l Physical data model - Include features such as keys in the middle-level model l Need to determine appropriate levels of granularity of data in order to build a good data warehouse

7. Distributing the Data Warehouse l Issues similar to distributed database systems Distributed Warehouse Central Bank Branch ABranch B Central Warehouse Central Bank Branch A Branch B Central Warehouse Branch B Warehouse Branch A Warehouse Non-distributed Warehouse

8. Multidimensional Data Model

9. Indexing for Data Warehousing l Bit-Maps l Multi-level indexing l Storing parts or all of the index files in main memory l Dynamic indexing

10. Metadata Mappings

11. Data Warehousing and Security l Security for integrating the heterogeneous data sources into the repository - e.g., Heterogeneity Database System Security, Statistical Database Security l Security for maintaining the warehouse - Query, Updates, Auditing, Administration, Metadata l Multilevel Security - Multilevel Data Models, Trusted Components

12. Example Secure Data Warehouse

13. Secure Data Warehouse Technologies

14. Security for Integrating Heterogeneous Data Sources l Integrating multiple security policies into a single policy for the warehouse - Apply techniques for federated database security? - Need to transform the access control rules l Security impact on schema integration and metadata - Maintaining transformations and mappings l Statistical database security - Inference and aggregation - e.g., Average salary in the warehouse could be unclassified while the individual salaries in the databases could be classified l Administration and auditing

15. Security Policy for the Warehouse Federated policies become warehouse policies? Component Policy for Component A Component Policy for Component B Component Policy for Component C Generic Policy for Component A Generic Policy for Component B Generic policy for Component C Export Policy for Component A Export Policy for Component B Export Policy for Component C Federated Policy for Federation F1 Federated Policy for Federation F2 Export Policy for Component B Security Policy Integration and Transformation

16. Security Policy for the Warehouse - II

17. Secure Data Warehouse Model

18. Methodology for Developing a Secure Data Warehouse

19. Multi-Tier Architecture Tier 1:Secure Data Sources Tier 2: Builds on Tier 1 Tier N: Data Warehouse Builds on Tier N-1 * * Tier 1:Secure Data Sources Tier 2: Builds on Tier 1 Tier N: Secure Data Warehouse Builds on Tier N-1 * * Each layer builds on the Previous Layer Schemas/Metadata/Policies

20. Administration l Roles of Database Administrators, Warehouse Administrators, Database System Security officers, and Warehouse System Security Officers? l When databases are updated, can trigger mechanism be used to automatically update the warehouse? - i.e., Will the individual database administrators permit such mechanism?

21. Auditing l Should the Warehouse be audited? - Advantages l Keep up-to-date information on access to the warehouse - Disadvantages l May need to keep unnecessary data in the warehouse l May need a lower level granularity of data l May cause changes to the timing of data entry to the warehouse as well as backup and recovery restrictions l Need to determine the relationships between auditing the warehouse and auditing the databases

22. Multilevel Security l Multilevel data models - Extensions to the data warehouse model to support classification levels l Trusted Components - How much of the warehouse should be trusted? - Should the transformations be trusted? l Covert channels, inference problem

23. Inference Controller

24. Status and Directions l Commercial data warehouse vendors are incorporating role- based security (e.g., Oracle) l Many topics need further investigation - Building a secure data warehouse - Policy integration - Secure data model - Inference control

25. Data Mining for Counter-terrorism

26. Data Mining Needs for Counterterrorism: Non-real-time Data Mining l Gather data from multiple sources - Information on terrorist attacks: who, what, where, when, how - Personal and business data: place of birth, ethnic origin, religion, education, work history, finances, criminal record, relatives, friends and associates, travel history,... - Unstructured data: newspaper articles, video clips, speeches, emails, phone records,... l Integrate the data, build warehouses and federations l Develop profiles of terrorists, activities/threats l Mine the data to extract patterns of potential terrorists and predict future activities and targets l Find the “needle in the haystack” - suspicious needles? l Data integrity is important l Techniques have to SCALE

27. Data Mining for Non Real-time Threats Integrate data sources Clean/ modify data sources Build Profiles of Terrorists and Activities Examine results/ Prune results Report final results Data sources with information about terrorists and terrorist activities Mine the data

28. Data Mining Needs for Counterterrorism: Real-time Data Mining l Nature of data - Data arriving from sensors and other devices l Continuous data streams - Breaking news, video releases, satellite images - Some critical data may also reside in caches l Rapidly sift through the data and discard unwanted data for later use and analysis (non-real-time data mining) l Data mining techniques need to meet timing constraints l Quality of service (QoS) tradeoffs among timeliness, precision and accuracy l Presentation of results, visualization, real-time alerts and triggers

29. Data Mining for Real-time Threats Integrate data sources in real-time Build real-time models Examine Results in Real-time Report final results Data sources with information about terrorists and terrorist activities Mine the data Rapidly sift through data and discard irrelevant data

30. Data Mining Outcomes and Techniques for Counter-terrorism

31. Example Success Story - COPLINK l COPLINK developed at University of Arizona - Research transferred to an operational system currently in use by Law Enforcement Agencies l What does COPLINK do? - Provides integrated system for law enforcement; integrating law enforcement databases - If a crime occurs in one state, this information is linked to similar cases in other states - It has been stated that the sniper shooting case may have been solved earlier if COPLINK had been operational at that time

32. Where are we now? l We have some tools for - building data warehouses from structured data - integrating structured heterogeneous databases - mining structured data - forming some links and associations - information retrieval tools - image processing and analysis - pattern recognition - video information processing - visualizing data - managing metadata

33. What are our challenges? l Do the tools scale for large heterogeneous databases and petabyte sized databases? l Building models in real-time; need training data l Extracting metadata from unstructured data l Mining unstructured data l Extracting useful patterns from knowledge-directed data mining l Rapidly forming links and associations; get the big picture for real- time data mining l Detecting/preventing cyber attacks l Mining the web l Evaluating data mining algorithms l Conducting risks analysis / economic impact l Building testbeds

34. IN SUMMARY: l Data Mining is very useful to solve Security Problems - Data mining tools could be used to examine audit data and flag abnormal behavior - Much recent work in Intrusion detection (unit #18) l e.g., Neural networks to detect abnormal patterns - Tools are being examined to determine abnormal patterns for national security l Classification techniques, Link analysis - Fraud detection l Credit cards, calling cards, identity theft etc. BUT CONCERNS FOR PRIVACY