CSE494/598 Principles of Information Engineering Information Life Cycle

Lesson Objectives: 1. Describe the parts of the Information Life Cycle. 2. Explain the advantages and disadvantages of coding and compression. 3. Discuss considerations for information storage. 4. Describe common factors that must be addressed for proper information presentation. 5. Relate information analysis to information retrieval. 6. Classify characteristics of various generations of database management. Advance Reading Material: Read “The Top 10 Data Mining Questions”. It can be found at:

Information Life Cycle Information Science Acquisition Analysis/mining/ processing Coding/ Compression Storage Re-engineering Preservation Retrieval Packaging/ Visualization Presentation Transport Discard Information ScienceInformation Life Cycle Acquisition Analysis/mining/ processing Coding/ Compression Storage Re-engineering Preservation Retrieval Packaging/ Visualization Presentation Transport Discard

1. Information Acquisition Acquiring of business-related information in digital form Traditionally, record based data mostly in table form Now multimedia data –Conversion to digital form for on-line processing –Overall organization for seamless integration

2. Coding & Compression Coding of data in order to minimize its representation for reducing the storage requirement and reducing the bandwidth requirement in communication Need different techniques for each type of media and even each type of object –Facsimile vs. aerial pictures vs. portrait Technique must be fast, one-pass, adaptive and invertible, and must not impose unreasonable requirements on resources.

3. Analysis & mining Raw numbers, words, images and sounds are not immediately useful – their contents must be analyzed and represented in machine processable form: –Mining of databases for useful information –Extraction of contents from images and video –Conceptualization of text –Feature analysis of audio segments

4. Storage Business data can be very large and heterogeneous with respect to all parameters Appropriate storage techniques ensure: proper management, location and distribution, and the flow of objects. Among issues to be considered: –Data placement –What technology (medium) to use for storage –Distribution: local, remote, out-sourced –Speed of delivery

5. Re-engineering Legacy systems make up most of the business data systems Maintenance and modernization of these systems represents a large portion of IT efforts Important decisions: –Maintenance –replace & migrate –modernize for co-existence

Is legacy code like Chernobyl? Remember Chernobyl? The meltdown of the Nuclear reactor. Officials poured concrete over it and hoped that, someday, it would just go away! Legacy code and Chernobyl: Too messy to clean up but too dangerous to ignore!

Legacy code... Theory: rebuild the legacy system from ground up with –a relational (or OO) database –graphical user interfaces –client/server architecture Practice: expensive and risky, because of size, complexity and poor documentation.

Case study clients 120,000,000 credit cards (mid-90s figure) Over 14 tera bytes of data 2 billion transactions per month –19 billion disk/tape I/O per month Around 23 million transactions are processed from 8:00 pm to 2:00 am

Case study 2 22 million telephone customers “zero downtime” must be guaranteed COBOL code: Hundreds of millions of lines Many tera bytes of data “owned” by applications – no sharing -> redundant storage Regulatory change: “ rate of return” to “price cap” Reengineer 80% of the business process

Case study 2 Incremental migration into a client server computing architecture Began in late 80s ago, still on-going… Around 10,000 workstations, and growing Biggest challenge: Inability of mainframe to participate in distributed C/S computing –CICS unable to cooperate in a nested sub- transaction  Integrity?

About Legacy Systems Large, with millions of lines of code Over 10 years old Mission-critical - 24-hour operation Difficulty in supporting current/future business requirements 80-90% of IT budget Instinctive solution: Migrate!

Migration Strategies Complete rewrite of legacy code –Many problems –Risky –Prone to failure Incremental migration –Migrate the legacy system in place by small incremental steps –Control risk by choosing increment size.

One-step Migration Impediments Business conditions never stand still Specifications rarely exist Undocumented dependencies Management of large projects –too hard, tend to bloat Migration with live data Analysis paralysis sets in Fear of change

Incremental Migration Incrementally analyze the legacy IS Incrementally decompose Incrementally design the target interfaces Incrementally design the target applications Incrementally design the target database Incrementally install the target environment Create and install the necessary gateways Incrementally migrate the legacy database Incrementally migrate the legacy applications Incrementally migrate the legacy interfaces Incrementally cut over to the target IS

A Comparison

6. Preservation Similar to physical security measures for protecting buildings, cash and other tangible assets, information must be protected while recorded, processed, stored, shared, transmitted, or retrieved. Must protect against loss, alteration, and disclosure Must prevent unauthorized access and unauthorized use of –Computer systems –Networks –Information

7. Retrieval Query languages have come a long way from old style navigational queries to today’s content-based query languages Important: Any constraint (e.g., a processable feature) may be used as the criterion for search Require efficient retrieval techniques, similar to those for data retrieval, for all types of information

Web Search Engines A text retrieval system with a Web interface The document collection of a search engine can be either a pre-compiled special collection or a set of Web pages collected from many web servers by a program called Web robot. Each document is preprocessed and represented as a vector of terms with weights.

Web Search Engines (cont’d..) The steps are, Stopward removal: Remove non-content words such as “a” and “is” from each document. Stemming: Map variations of the same word into a term. Term weighting: Assign a weight to each term in a document to indicate the importance of the term in representing the importance of the term in representing the contents of the document.

Web Search Engines (cont’d..) A query is also transformed into a vector with weights. The similarity between a query and a document can be measured by the dot product of their respective vectors. When documents are HTML web pages, other factors can influence a term's weight in a document. –title or the header –enclosed in special tags or in special fonts –Google and AltaVista use tag and location information

Web Search Engines (cont’d..) Web pages are hyperlinked. There are pointers going from one page to another. Associated with each pointer are words (anchor terms), which show the users what trey are likely to find if the pointer is followed. Anchor terms are utilized to index referenced/pointed pages. Linkage information can also be combined with similarity information to improve the retrieval effectiveness.

Meta-Search Engines Has a number of modules. The user interface module accepts the user’s query which will be forwarded, with necessary reformatting, by the query dispatcher module to the various search engines. When the search engines return the sets of the retrieved documents to the metasearch engine, these sets are merged by the result merger module into a single ranked list of documents.

Meta-Search Engines (cont’d..) Certain number of the top documents from this list are displayed to the user. When the number of search engines underlying a metasearch engine is large, forwarding each user query to each search engine is very inefficient. To overcome this, a database selection module is included. Its function is to identify for each user query the search engines that are likely to return useful documents to the user.

8. Presentation Information must be presented to the user in a form that is usable –Cookies take care of part of the issue Issues are diverse and range from formatting, visualization, language, and even cultural barriers In the case of multimedia information, both temporal and spatial issues must be dealt with

9. Transport Moving of data/information from one location to another –Most common form: digital communication Technology selection for information transport: –What communication service? –What protocols? –What quality of service? –What physical resources?

10. Information discard Destruction of information once its useful life is over –Generally, preserve data – unless discard is needed Methods for discard Legal issues must be taken into account

3. Analysis & mining Additional notes…

Information Analysis and Mining In multimedia objects: –Extraction of features –Their representation –Indexing on the basis of contents For data: Mining in order to find useful patterns and correlations For text: –Conceptual representation –Ontological classification of concepts

Analysis of Images Extract features –Color –Shape –Texture –Spatial relationships Create a logical representation for the image –Semantic nets are effective Classify and index so that the search process will be efficient

Analysis of Video Determine video segments by detecting scene cuts (Scene cut detection process) Select a representative frame for each segment Extract Spatial features : –color, texture, shape, and relative object positions Extract Temporal features: –object trajectories, camera motion, viewing perspective –temporal relationships among objects Represent each segment with an object that can be efficiently indexed by its features

Video Indexing process

Analysis of Audio For Speech: –Textual information from speech (then sound retrieval becomes text retrieval) –Speaker Information (identification) For Generic Sound: –Loudness –Pitch –Tone –Cepstrum –Derivatives For Music: –Rhythm –Event –Instrument

Analysis of data The hardest task: Integration of data from multiple databases –Despite many years of work, we still have difficulty in this area Data mining tasks: descriptive, predictive –Descriptive: Characterize general properties of the data –Predictive: Perform inference on the data to make predictions Most common types: Specialized abstracts and integrated tables

Data Collection and Database Creation (1960s and earlier) -Primitive file processing Early days of databases

Database Management Systems (1970s-early 1980s) -Hierarchical and network database systems -Relational database systems -Data modeling tools: entity-relationship model, etc. -Indexing and data organization techniques: B+ -tree, hashing etc. -Query languages: SQL, etc. -User interfaces, forms and reports -Query processing and query optimization -Transaction management: recovery, concurrency control,etc. -On-line transaction processing(OLTP) Database management systems

Advanced Database Systems (mid-1980s-present) -Advanced data models: extended-relational, object-oriented, object-relational, deductive -Application-oriented: spatial, temporal, multimedia, active, scientific, knowledge bases Current databases

Data Warehousing and Data Mining (late 1980s-present) -Data warehouse and OLAP technology -Data mining and knowledge discovery Data Integration Web-based Database Systems (1990s-present) -XML based database systems -Web mining

Data Collection and Database Creation (1960s and earlier) -Primitive file processing Database Management Systems (1970s-early 1980s) -Hierarchical and network database systems -Relational database systems -Data modeling tools: entity-relationship model, etc. -Indexing and data organization techniques: B+ -tree, hashing etc. -Query languages: SQL, etc. -User interfaces, forms and reports -Query processing and query optimization -Transaction management: recovery, concurrency control,etc. -On-line transaction processing(OLTP) Data Warehousing and Data Mining (late 1980s-present) -Data warehouse and OLAP technology -Data mining and knowledge discovery New Generation of Integrated Information Systems (2000-…) Advanced Database Systems (mid-1980s-present) -Advanced data models: extended-relational, object-oriented, object-relational, deductive -Application-oriented: spatial, temporal, multimedia, active, scientific, knowledge bases Web-based Database Systems (1990s-present) -XML based database systems -Web mining Putting it all Together…

Information mining process Data cleaning –Reformatting and conversation may be necessary Data integration –Heterogeneity possible in any aspect Data selection Data transformation Data mining and evaluation of patterns Presentation of knowledge

Cleaning and Integration Data Warehouse Selection and Transformation Data Mining Patterns Knowledge Evaluation and Presentation Flat files Flat files Databases Databases

Data Warehousing and ETL An organized repository of data from multiple data sources –A unified schema for all of the participating databases Provides data analysis capabilities, collectively known as On-Line Analytical Processing (OLAP) A number of pieces are needed: tools, gateways, and conversion routines

Data source in Location 2 Data source in Location 1 Data source in Location 3 Data source in Location 4 Clean Transform Integrate Load Data Warehouse Query and analysis tools Client Typical architecture of a data warehouse