Data Management Information Management Knowledge Management for Network Centric Operations Dr. Bhavani Thuraisingham The University of Texas at Dallas October 2005

2 8/29/ :12 Data, Information and Knowledge Management: Definitions Information Management: Extracting information from the data Visualizing the data Data Management: Data administration Database management Knowledge Management: Acquiring knowledge Collaboration and sharing Managing the processes Disseminating the knowledge Taking action

3 8/29/ :12 What is data management? 0 One proposal: Data Management = Database System Management + Data Administration 0 Includes data analysis, data administration, database administration, auditing, data modeling, database system development, database application development

4 8/29/ :12 Data Administration 0 Identifying the data -Data may be in files, paper, databases, etc. 0 Analyzing the data -Is the data of good quality? -Is the data complete? 0 Data standardization -Should one standardize all the data elements and metadata? -Repositories for handling semantic heterogeneity? 0 Data Security -How should data be secured? 0 Data modeling -Structure the data, model the data and the processes

5 8/29/ :12 Data Administration (Continued) 0 Data quality provides some measure for determining the accuracy of the data -Is the data current? Can we trust the source? -Data quality parameters can be passed from source to source =E.g., Trust A 50% and Trust B 30% 0 Data may have different semantics -E.g, Bank A may send out statement on the 20 th day of each month and Bank B may send out statements on the 5 th day of each month -Fighter jet and Passenger plane may be considered to be one and the same

6 8/29/ :12 Data Administration (Concluded) 0 Data Standards -Standards for data semantics and administration -E.g., XML (eXtensible Markup Language) for document interchange 0 Data security includes data confidentiality and integrity -Confidentiality is about preventing unauthorized access to the data -Integrity is about preventing malicious corruption to the data

7 8/29/ :12 An Example Database System

8 8/29/ :12 Metadata 0 Metadata describes the data in the database -Example: Database D consists of a relation EMP with attributes SS#, Name, and Salary 0 Metadatabase stores the metadata -Could be physically stored with the database 0 Metadatabase may also store constraints and administrative information 0 Metadata is also referred to as the schema or data dictionary

9 8/29/ :12 Three-level Schema Architecture: Details External Schema A External Schema B Conceptual Schema Internal Schema User A1 User A2User A3User B1 User B2 External Model A External Model B Conceptual Model Stored Database Internal Model External/Conceptual Mapping B External/Conceptual Mapping A Conceptual/Internal Mapping

10 8/29/ :12 Functional Architecture User Interface Manager Query Manager Transaction Manager Schema (Data Dictionary) Manager (metadata) Security/ Integrity Manager File Manager Disk Manager Data Management Storage Management

11 8/29/ :12 Types of Database Systems 0 Relational Database Systems 0 Distributed and Federated Database Systems 0 Object Database Systems 0 Deductive Database Systems 0 Other -Real-time, Secure, Parallel, Scientific, Temporal, Wireless, Functional, Entity-Relationship, Sensor/Stream Database Systems, etc.

12 8/29/ :12 Relational Database: Example Relation S: S# SNAME STATUS CITY S1 Smith 20 London S2 Jones 10 Paris S3 Blake 30 Paris S4 Clark 20 London S5 Adams 30 Athens Relation P: P# PNAME COLOR WEIGHT CITY P1 Nut Red 12 London P2 Bolt Green 17 Paris P3 Screw Blue 17 Rome P4 Screw Red 14 London P5 Cam Blue 12 Paris P6 Cog Red 19 London Relation SP: S# P# QTY S1 P1 300 S1 P2 200 S1 P3 400 S1 P4 200 S1 P5 100 S1 P6 100 S2 P1 300 S2 P2 400 S3 P2 200 S4 P2 200 S4 P4 300 S4 P5 400

13 8/29/ :12 Example Object Composite Document Object Section 1 Object Section 2 Object Paragraph 1 Object Paragraph 2 Object

14 8/29/ :12 Distributed Database System Communication Network Distributed Processor 1 DBMS 1 Data- base 1 Data- base 3 Data- base 2 DBMS 2 DBMS 3 Distributed Processor 2 Distributed Processor 3 Site 1 Site 2 Site 3

15 8/29/ :12 DBMS 1 DQP DBMS 2 DQP DBMS 3 EMP1 (20) EMP2 (30) DEPT2 (20) EMP1 (20) EMP3 (50) DEPT3 (30) Network Query at site 1: Join EMP and DEPT on D# Move EMP2 to site 3; Merge EMP1, EMP2, EMP3 to form EMP Move DEPT2 to site 3; Merge DEPT2 and DEPT3 to form DEPT Join EMP and DEPT; Move result to site 1 Query Processing Example DQP (Distributed Query Processor)

16 8/29/ :12 Transaction Processing Example Site 1 Coordinator Transaction Tj Site 2 Participant Site 3 Participant Site 4 Participant Subtransaction Tj2 Subtransaction Tj3 Subtransaction Tj4 Issues: Concurrency control Recovery Data Replication Two-phase commit: Coordinator queries participants whether they are ready to commit If all participants agree, then coordinator sends request for the participants to commit DTM (Distributed Transaction Manager) responsible for executing the distributed transaction

17 8/29/ :12 Interoperability of Heterogeneous Database Systems Database System A Database System B Network Database System C (Legacy) Transparent access to heterogeneous databases - both users and application programs; Query, Transaction processing (Relational) (Object- Oriented)

18 8/29/ :12 Technical Issues on the Interoperability of Heterogeneous Database Systems 0 Heterogeneity with respect to data models, schema, query processing, query languages, transaction management, semantics, integrity, and security policies 0 Interoperability based on client-server architectures 0 Federated database management -Collection of cooperating, autonomous, and possibly heterogeneous component database systems, each belonging to one or more federations

19 8/29/ :12 Different Data Models Node A Node B Database Relational Model Network Model Node C Database Object- Oriented Model Network Node D Database Hierarchical Model Developments: Tools for interoperability; commercial products Challenges: Global data model

20 8/29/ :12 Schema Integration and Transformation: An approach Schema describing the network database Schema describing the hierarchical database Schema describing the object-oriented database Global Schema: Integrate the generic schemas External Schema I External Schema II External Schema III Schema describing the relational database Generic schema describing the relational database Generic schema describing the network database Generic schema describing the hierarchical database Generic schema describing the object-oriented database Challenges: Selecting appropriate generic representation; maintaining consistency during transformations;

21 8/29/ :12 Semantic Heterogeneity 0 Semantic heterogeneity occurs when there is a disagreement about the meaning or interpretation of the same data; or same data interpreted differently Object O Node A Node B Database Object O interpreted as a passenger ship Object O interpreted as a submarine Challenges: Standard definitions; Repositories

22 8/29/ :12 Federated Database Management Database System A Database System B Database System C Cooperating database systems yet maintaining some degree of autonomy Federation F1 Federation F2

23 8/29/ :12 Autonomy Component A Component B Component C local request request from component communication through federation component A does not communicate with component C component A honors the local request first Challenges: Adapt techniques to handle autonomy - e.g., transaction processing, schema integration; transition research to products

24 8/29/ :12 Federated Data and Policy Management Export Data/Policy Component Data/Policy for Agency A Data/Policy for Federation Export Data/Policy Component Data/Policy for Agency C Component Data/Policy for Agency B Export Data/Policy

25 8/29/ :12 What is Information Management? 0 Information management essentially analyzes the data and makes sense out of the data 0 Several technologies have to work together for effective information management -Data Warehousing: Extracting relevant data and putting this data into a repository for analysis -Data Mining: Extracting information from the data previously unknown -Multimedia: managing different media including text, images, video and audio -Web: managing the databases and libraries on the web

26 8/29/ :12 Data Warehouse Oracle DBMS for Employees Sybase DBMS for Projects Informix DBMS for Medical Data Warehouse: Data correlating Employees With Medical Benefits and Projects Could be any DBMS; Usually based on the relational data model Users Query the Warehouse

27 8/29/ :12 What is Data Mining? Data Mining Knowledge Mining Knowledge Discovery in Databases Data Archaeology Data Dredging Database Mining Knowledge Extraction Data Pattern Processing Information Harvesting Siftware The process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data, often previously unknown, using pattern recognition technologies and statistical and mathematical techniques (Thuraisingham 1998)

28 8/29/ :12 Steps to Data Mining Data Sources Integrate data sources Clean/ modify data sources Mine the data Examine Results/ Prune results Report final results/ Take actions

29 8/29/ :12 Data Mining Needs for Counterterrorism: Non-real-time Data Mining 0 Gather data from multiple sources -Information on terrorist attacks: who, what, where, when, how -Personal and business data: place of birth, ethnic origin, religion, education, work history, finances, criminal record, relatives, friends and associates, travel history,... -Unstructured data: newspaper articles, video clips, speeches, s, phone records,... 0 Integrate the data, build warehouses and federations 0 Develop profiles of terrorists, activities/threats 0 Mine the data to extract patterns of potential terrorists and predict future activities and targets 0 Find the “needle in the haystack” - suspicious needles? 0 Data integrity is important 0 Techniques have to SCALE

30 8/29/ :12 Data Mining Needs for Counterterrorism: Real-time Data Mining 0 Nature of data -Data arriving from sensors and other devices =Continuous data streams -Breaking news, video releases, satellite images -Some critical data may also reside in caches 0 Rapidly sift through the data and discard unwanted data for later use and analysis (non-real-time data mining) 0 Data mining techniques need to meet timing constraints 0 Quality of service (QoS) tradeoffs among timeliness, precision and accuracy 0 Presentation of results, visualization, real-time alerts and triggers

31 8/29/ :12 Data Mining as a Threat to Privacy 0 Data mining gives us “facts” that are not obvious to human analysts of the data 0 Can general trends across individuals be determined without revealing information about individuals? 0 Possible threats: -Combine collections of data and infer information that is private =Disease information from prescription data =Military Action from Pizza delivery to pentagon 0 Need to protect the associations and correlations between the data that are sensitive or private

32 8/29/ :12 Privacy Preserving Data Mining User Interface Manager Constraint Manager Privacy Constraints Query Processor: Constraints during query and release operations Data Miner: Makes correlations Ensures privacy Database Design Tool Structures the database Database DBMS

33 8/29/ :12 Current Status, Challenges and Directions 0 Status -Data Mining is now a technology -Several prototypes and tools exist; Many or almost all of them work on relational databases 0 Challenges -Mining large quantities of data; Dealing with noise and uncertainty, reasoning with incomplete data, Eliminating False positives and False negatives 0 Directions -Mining multimedia and text databases, Web mining (structure, usage and content), Mining metadata, Real- time data mining, Privacy

34 8/29/ :12 Semantic Web: Overview 0 According to Tim Berners Lee, The Semantic Web supports -Machine readable and understandable web pages -Enterprise application integration -Nodes and links that essentially form a very large database Premise: Semantic Web Applications: Web Database Management + Web Services + Information Integration Semantic Web Technologies: XML, RDF, Ontologies, Rules-ML

35 8/29/ :12 Layered Architecture for Dependable Semantic Web 0 Some Challenges: Interoperability between Layers; Security and Privacy cut across all layers; Integration of Services; Composability XML, XML Schemas Rules/Query Logic, Proof and Trust TRUSTTRUST Other Services RDF, Ontologies URI, UNICODE PRIVACYPRIVACY 0 Adapted from Tim Berners Lee’s description of the Semantic Web

36 8/29/ :12 What is XML all about? 0 XML is needed due to the limitations of HTML and complexities of SGML 0 It is an extensible markup language specified by the W3C (World Wide Web Consortium) 0 Designed to make the interchange of structured documents over the web easier 0 Key to XML are Document Type Definitions (DTDs) and XML Schemas 0 Allows users to bring multiple files together to form compound documents

37 8/29/ :12 What is Knowledge Management? 0 Knowledge management, or KM, is the process through which organizations generate value from their intellectual property and knowledge-based assets 0 Gartner group: KM is a discipline that promotes an integrated approach to identifying and sharing all of an enterprise's information assets, including databases, documents, policies and procedures as well as unarticulated expertise and experience resident in individual workers 0 Peter Senge: Knowledge is the capacity for effective action, this distinguishes knowledge from data and information; KM is just another term in the ongoing continuum of business management evolution

38 8/29/ :12 Knowledge Management Components Components: Strategies Processes Metrics Cycle: Knowledge, Creation Sharing, Measurement And Improvement Technologies: Expert systems Collaboration Training Web Components of Knowledge Management: Components, Cycle and Technologies

39 8/29/ :12 KM: Strategy, Process and Metrics 0 Strategy -Motivation for KM and how to structure a KM program 0 Process -Use of KM to make existing practice more effective 0 Metrics -Measure the impact of KM on an organization

40 8/29/ :12 Strategy: Building Learning Organizations 0 Adaptive learning and Generative learning -Need to adapt to the changing environment -Total quality movement (TQM) in Japan has migrated to a generative learning model =Look at the world in a new way 0 Changing roles of the leader -Migrating from decision makers to designers, teachers and stewards 0 Building a shared vision -Encouraging ideas, Requesting support, Moving beyond blame, Effective communication 0 Learning tools -Learning laboratory

41 8/29/ :12 Knowledge Management in Process Management 0 Types of Processes -Simple processes: Low level operation -Complex and nonadapative processes: Systems that use the same rules -Complex and adaptive: Agents carrying out the processes are intelligent and adaptive 0 Linking knowledge management with processes -Knowledge management is needed for all processes; critical for complex and adaptive processes -Learn from experience and use the experience in unknown situations

42 8/29/ :12 Metrics: The Balanced Scorecard 0 Employee Capabilities: Measuring the following -Employee satisfaction -Employee retention -Employee productivity 0 Information system capabilities: Measuring the following -Whether each employee segment has information to carry out its operations. 0 Motivation and Empowerment: Measuring the following -Suggestions made and implemented -Improvement -Team performance

43 8/29/ :12 Knowledge Management Architecture Knowledge Creation and Acquisition Manager Knowledge Representation Manager Knowledge Manipulation Manager Knowledge Dissemination and Sharing Manager

44 8/29/ :12 Secure Knowledge Management 0 Protecting the intellectual property of an organization 0 Access control including role-based access control 0 Security for process/activity management and workflow -Users must have certain credentials to carry out an activity 0 Composing multiple security policies across organizations 0 Security for knowledge management strategies and processes 0 Risk management and economic tradeoffs 0 Digital rights management and trust negotiation

45 8/29/ :12 Status and Directions 0 Knowledge management has exploded due to the web 0 Knowledge Management has different dimensions -Technology, Business -Goal is to take advantage of knowledge in a corporation for reuse 0 Tools are emerging 0 Need effective partnerships between business leaders, technologists and policy makers 0 Knowledge management may subsume information management and data management -Vague boundaries

46 8/29/ :12 Other Ideas and Directions? Prof. Bhavani Thuraisingham -Director Cyber Security Center -Department of Computer Science -Erik Jonsson School of Engineering and Computer Science -The University of Texas at Dallas -Richardson, Texas President Dr-Bhavani Security Consulting Dallas, TX