Presentation is loading. Please wait.

Presentation is loading. Please wait.

DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES BY PROF. CHIEDU F. MAFIANA DIRECTOR, QUALITY ASSURANCE NATIONAL UNIVERSITIES COMMISSION Being a paper presented.

Similar presentations


Presentation on theme: "DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES BY PROF. CHIEDU F. MAFIANA DIRECTOR, QUALITY ASSURANCE NATIONAL UNIVERSITIES COMMISSION Being a paper presented."— Presentation transcript:

1 DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES BY PROF. CHIEDU F. MAFIANA DIRECTOR, QUALITY ASSURANCE NATIONAL UNIVERSITIES COMMISSION Being a paper presented at the 2013 Training Workshop of the Committee of Directors of Academic Planning of Nigerian Universities (CODAPNU) at the National Mathematical Centre, Kwali, Abuja from 28 – 31 October 2013

2 FELICITATIONS! CODAPNU NUC PARTNERING FOR PROGRESS & QUALITATIVE DEVELOPMENT

3 21 ST CENTURY CHALLENGES THE WORLD IS NOW A GLOBAL VILLAGE! HOW ARE UNIVERSITIES COPING WITH 21 ST CENTURY CHALLENGES ? WE ARE THE WORLD

4 21 ST CENTURY CHALLENGES NATION BUILDING ACCESS RELEVANCE OF CURRICULA FUNDING CROSS BORDER EDUCATION NUC GLOBALIZATION QA ISSUES ETC

5 21 ST CENTURY CHALLENGES DATA, DATA EVERYWHERE! HELP! DAP

6 21 ST CENTURY CHALLENGES BUT ARE WE MAKING GOOD USE OF ALL THE DATA?

7 ANALYTICS MR ANALYTICS AT YOUR SERVICE QUALITY INPUT + EFFECTIVE PROCESSES + COMPLIANCE WITH NUC GUIDELINES = QUALITY OUTPUT

8 DATA WAREHOUSING DATA WAREHOUSING? NEVER HEARD OF IT BUT MUSTN’T DISPLAY MY IGNORANCE!

9 CONCEPTS & DEFINITIONS MEET BILL INMON FATHER OF WAREHOUSING 1970s “It is a subject oriented, nonvolatile, integrated, time variant collection of data in support of management's decisions” What is a Data Warehouse? Centralized Data warehouses large size for big businesses

10 CONCEPTS & DEFINITIONS MEET RALPH KIMBALLMEET RALPH KIMBALL 2 ND FATHER OF WAREHOUSING 1998 What is a Data Warehouse? “A database of snapshots and aggregations of data throughout an enterprise to be used for querying and decision-making” INTEGRATED SYSTEMS, SMALLER DATA MARTS FOR SMALLER BUSINESSES AND BUDGETS

11 CONCEPTS & DEFINITIONS HMMM! IT’S GOING TO BE A LONG, LONG DAY OLAP STAR SCHEMA SNOWFLAKE SCHEMA HYPERCUBE METADATA LEGACY CODE DATA MART

12 CONCEPTS AND DEFINITIONS Hmmm! At least now I know what Data Warehouse is. I hope the process will be clearer after the Case Study. Oga teacher U try. Thank U Sir.

13 TABLE I: COMPARISON OF DW DATA AND OP DATA OPERATIONAL DATADW DATA Application orientedSubject oriented DetailedSummarized, otherwise refined Accurate, as of the moment of access Represents values over time, snapshots Serves the clerical community Serves the managerial community

14 TABLE I: COMPARISON OF DW DATA AND OP DATA OPERATIONAL DATADW DATA Can be updatedIs not updated Run repetitively and non reflectively Run heuristically Requirements for processing understood before initial development Requirements for processing not completely understood before development Compatible with the Software Development Life Cycle Completely different life cycle

15 TABLE I: COMPARISON OF DW DATA AND OP DATA OPERATIONAL DATADW DATA Performance sensitive (immediate response required when entering a transaction) Performance relaxed (immediacy not required) Accessed a unit at a time (limited number of data elements for a single record) Accessed a set at a time (many records of many data elements) Transaction drivenAnalysis driven Control of update a major concern in terms of ownership Control of update no issue

16 TABLE I: COMPARISON OF DW DATA AND OP DATA OPERATIONAL DATADW DATA High availabilityRelaxed availability Managed in its entiretyManaged by subsets Non-redundancyRedundancy is a fact of life Static structure; variable contents Flexible structure Small amount of data used in process Large amount of data used in a process

17 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) Q1 Do I Need a Data Warehouse? No Idea. Do You?

18 CASE STUDY Q2 What Specific Problems Will It Solve? Leave me alone. I have my own problems

19 CASE STUDY Q3 What are my available resources (time, money, & personnel ? Am I your P.A?

20 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) Q4 What criteria will I use to measure my success ? Your Hummer Jeep. Remember me in your kingdom.

21 CASE STUDY Q5 Should I outsource all, some or none of the development and operation? Just make sure you don’t get lynched in the process

22 CASE STUDY Q6 Am I upgrading an existing system, converting from a legacy system, or developing from scratch ? If your DICT does not know, sack him

23 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 1 REQUIREMENTS ANALYSIS 2 INFORMATION/DATA MODELLING 3 DESIGN & PROTOTYPING

24 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES a. RAPID PROTOTYPING b. STRUCTURED DEVELOPMENT 4. DEVELOPMENT & DOCUMENTATION

25 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 5. TEST & REVIEW 6. DEPLOYMENT & TRAINING 7. OPERATION

26 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 8. ENHANCEMENT 9. HELP DESK

27 CHALLENGES PART I

28 1 ENSURING ACCEPTABLE DATA QUALITY a. DISPARATE DATA SOURCES b. UNSTABILIZED SOURCE SYSTEMS

29 CHALLENGES PART I 2 ENSURING ACCEPTABLE PERFORMANCE a.a. PRIORITIZING PERFORMANCE b.b. SETTING REALISTIC GOALS

30 CHALLENGES PART I c. PERFORMANCE BY DESIGN 3. TESTING THE DATA WAREHOUSE a. TEST PLANNING

31 CHALLENGES PART I b. NO AUTOMATED TESTING 4. RECONCILIATION OF DATA a. COMPLEX

32 CHALLENGES PART I 5. USER ACCEPTANCE a. RELUCTANT USERS b. TRAIN THEM

33 CHALLENGES PART 2

34 1. USER EXPECTATION 2. SYSTEMS OPTIMIZATION 3. DATA STRUCTURING

35 CHALLENGES PART 2 4. PREFABRICATED VS CUSTOM WAREHOUSE 5. DATA STRUCTURING

36 THE LARGEST CHALLENGE OF THEM ALL GARBAGE IN GARBAGE OUT

37 QUALITY OF DATA SOURCE: PRACTICAL LIFE CHALLENGES IN IMPLEMENTING DATA WHAREHOUSE – Subramanya Hoysala 17th June 2011

38 LOW QUALITY DATA Accuracy Completeness Consistency Correctness

39 LOW QUALITY DATA Integrity Timelines Uniqueness

40 LOW QUALITY DATA FACTORS Lack of data validation in ERP-Systems; Older technologies do not support eg. drop down lists; The precise business rules are not known to many; People are busy;

41 LOW QUALITY DATA FACTORS Psychology: Correct data are not important to the user, who registers them, eg. the car sales person; Data may be loaded / corrected in one-off batch runs; and The rules have changed, but users are not aware of it.

42 LOW QUALITY DATA WHAT DO WE DO ABOUT IT?

43 LOW QUALITY DATA CORRECTIVE STEPS PROFILING CLEANSING & ENHANCEMENT CONSOLIDATION

44 LOW QUALITY DATA CORRECTIVE STEPS AUDITING HETEROGENEOUS DATA DATA INTEGRATION

45 SAMPLE UDW Source: New York University, Reporting Home page

46 EMERGING TRENDS CLOUD INTEGRATION REAL TIME DATA WAREHOUSING IN-MEMORY DBMSs BIG DATA

47 OVERVIEW: ICT IN NUS 2015 2013 2014 WORLD SUMMIT INFORMATION SOCIETY GLOBAL TARGET ICT FOR ALL 2015

48 ICT INITIATIVES IN NUS NUC NUS MY DEAR SON I HAVE SO MANY GIFTS FOR YOU DADDY! THANK YOU SIR NUTALP NgREN PRESSID NUSMAP NLOP VLP ACE

49 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE 1 STUDENT INFORMATION STAFF INFORMATION 2 UNIVERSITIES PROGRAMME ACCREDITATION BUDGETS AND EXPENDITURE 3 ENROLMENT/STATISTICS PHYSICAL PLANNING & DEVELOPMENT

50 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE 4 ACADEMI C PROGRAMMES 5 UNIVERSITY GEOGRAPHIC INFORMATION SYSTEM 6 RESEARCH AND INNOVATIONS

51 ICT INITIATIVES IN NUS NUSMAP – FEATURES 1 ROBUST ADMINISTRATION 2 SECURED LOGIN SYSTEM 3 UNIQUE IDENTITY FOR EACH SYSTEM

52 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE 4 USER FRIENDLY 5 CAPTURES A VAST RANGE OF INFORMATION 6 AUTOMTIC POPULATION OF STUDENTS & STAFF RECORDS

53 ICT INITIATIVES IN NUS NUSMAP – FEATURES 7 GENERATION OF PRINTALE REPORTS IN NUC-SPECIFIED FORMAT 8 EXPORT FEATURE 9 PROVISION FOR NUC TO VIEW SUBMITTED REPORTS

54 ICT INITIATIVES IN NUS NUSMAP – FEATURES 10 GENERATION OF PRINTALE REPORTS IN NUC-SPECIFIED FORMAT 11 AUTOMATIC CALCULTION OF FULL TIME EQUIVALENT (FTE) 12 AUTOMATIC SUMMATION/CALCULATION CAPABILITY

55 ICT INITIATIVES IN NUS NUSMAP – FEATURES 13 AUTOMATIC NOTIFICATION ON EXPIRY OF A PROJECT’S MASTERPLAN 14 AUTOMATIC GENERATION OF ANNUAL PAYROLL REPORT 15 STAFF PROGRESSION/STAFF AREA OF SPECIALIZATION AT A GLANCE

56 ICT INITIATIVES IN NUS NUSMAP – FEATURES 16 FACILITIES ONLINE UNIVERSITIES PROGRAMME ACCREDITATION 17 USARM NOW ONLINE

57 ICT INITIATIVES IN NUS NUSMAP – CHALLENGES 1 INITIAL RESISTANCE FROM SOME UNIVERSITIES 2 FORGOTTEN PASSWORDS PROBLEM OF UPLODING DUE TO WRONG ARRANGEMENT OF SOME EXCEL FILES

58 LEVEL OF ICT DEPLOYMENT IN NUS UNIVERSITY PORTAL POSTING OF VITAL INFORMATION ON- LINE ICT PARKS LAPTOPS FOR STAFF INTERNET CONNECTIVITY, AREAS GIVEN HIGH PRIORITY BY UNIVERSITIES

59 LEVEL OF ICT DEPLOYMENT IN NUS NUC ICT TEACHING AIDS? DATA BASES? DIGITAL VIRTUAL LBRARY? SIRS, WHAT ABOUT THESE ? PUBLISHING PAPERS ON-LINE? SATELLITE? VIDEO CONFERENCING? LOWER PRIORITY AREAS DIGITAL AUDIOS? ICT DRIVEN OPERATIONS? E-LEARNING SUPPORT & DEVELOPMENT ? Web Presence, Visibility and Access?. IP(Internet Protocol) Phones?

60 AREAS GIVEN PRIORITY BY UNIVERSITIES ICT Infrastructure  Internet Connectivity  ICT Parks  Laptops for Staff ICT Integration  University Portal  Posting of vital information on-line

61 LOWER PRIORITY AREAS Low level of ICT infrastructure Across Board  Digital Virtual Library  Satellite  Automated overhead projectors  Digital audios  IP(Internet Protocol) Phones

62 LOWER PRIORITY AREAS Low Level ICT Integration Across Board  Teaching  Publishing papers on-line  Advertising research proposals on-line  Work stations  Data bases

63 LEVEL OF ICT DEPLOYMENT IN NUS Findings NUS: LESS THAN 5% ICT OPERATIONS IN PROCESSES (Vision 20:2020 ICT Technical Report (2009) ) ICT CAPACITY & INFRASTRUCTURE ICT INTEGRATION ICT DEPLOYMENT IN OPERATIONS & SERVICE DELIVERY

64 REASONS FOR LOW LEVEL ICT DEPLOYMENT HIGH COSTS OF INFRASTRUCTURE, EQUIPMENT & ELECTRICITY; INADEQUATE FUNDING; LOW ICT SKILLS; INSUFFICIENT TRAINING; LOW PRIORITY RATING; RESISTANCE TO SOME ICT INITIATIVES; UNSUSTAINED INITIATIVES; & RESISTANCE TO CHANGE.

65 RECOMMENDATIONS Policies for higher ICT integration Serious investment in ICT infrastructures; Infrastructural support} Massive training and } Deployment of ICT skilled manpower} (Government to Universities)

66 RECOMMENDATIONS Collaboration with: Private Sector; International donors; and Civil Society (affordable and sustainable access to ICT infrastructure) In-service training programmes on effective ICT integration

67 RECOMMENDATIONS ICT integration competencies for promotion of staff; Improved Power Supply; Functional Records Management Programme; University Archive; and Benchmarking of functional Records Management Programme and University Archives by NUC.

68 THE WAY FORWARD Data Warehousing Initiative Realistic? Reserve for Affluent Universities? K.I.V.?

69 THE WAY FORWARD Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View LETS DO IT NUC

70 THE WAY FORWARD PLEASE LISTEN TO THE FOLLOWING ANNOUNCEMENTS 1. THE DATA WAREHOUSE PROJECT HAS NUC’S BACKING DAPS NUS

71 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF ARE AS FOLLOWS: A. LOBBY YOUR RESPECTIVE VICE- CHANCELLORS DAPS NUS

72 THE WAY FORWARD VICE-CHANCELLOR’S OFFICE UNIVERSITY OF DOGGED COMPLIANCE & INNOVATION (UDCI) WAREHOUSE? WHAT ABOUT THE ONE WE BUILT FOR THE CONSULT LAST YEAR? VC, SIR, I’VE COME TO DISCUSS THE ISSUE OF THE DATA WAREHOUSE PROJECT SORRY SIR, I MEAN THE ICT DATA WAREHOUSE PROJECT. IT’S VERY USEFUL FOR SRATEGIC PLANNING PURPOSES SIR. COME UP WITH A PROPOSAL PLEASE

73 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF : B. SOLICIT FOR SUPPORT FROM YOUR DICT & DQA GENTLEMEN PLS I NEED YOUR KIND ASSISTANCE AND SUPPORT IN PACKAGING A VIABLE PROPOSAL FOR THE DATA WAREHOUSE PROJECT DICT DQA

74 SOME BENEFITS OF DATA WAREHOUSING 1 Has a subject area orientation 2 Integrates data from multiple, diverse sources 3 Allows for analysis of data over time

75 SOME BENEFITS OF DATA WAREHOUSING 4 Adds ad hoc reporting and enquiry 5 Provides analysis capabilities to decision makers 6 Relieves the development burden on IT

76 SOME BENEFITS OF DATA WAREHOUSING 7 Provides improved performance for complex analytical queries 8 Relieves processing burden on transaction oriented databases 9 Allows for a continuous planning process

77 COSTS OF DATA WAREHOUSING 1 Time spent in careful analysis of measurable 2 Design and implementation effort 3 Hardware costs

78 COSTS OF DATA WAREHOUSING 4 Software costs 5 On-going support and maintenance 6 Resulting re-engineering effort

79 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF : C. CONSTRUCTION OF AN INTEGRATED INFORMATION ARCHITECTURE BRAINSTORMING COMMITTEE LADIES AND GENTS WE HAVE 3 TASKS a.Building an integrated information architecture b.Taxonomy of Administrative Activities c.Data Dictionary.

80 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF : D. CONSULTATION WITH THE DICT DICTDAP MY DICT SIR, SO WHEN ARE WE GOING TO MEET TO DISCUSS THE DATA WAREHOUSE WORKSHOP MATRIALS MY ABLE DAP SIR. LET’S MEET ON FRIDAY 2 PM IN MY OFFICE

81 MR. ANALYTICS FINDS A HOME CONGRATS! I HOPE YOU LIKE THE HOUSE MAGT. BOUGHT FOR YOU. I’M SO GRATEFUL MADAM DAP. NOW I CAN REALLY CONCENTRATE ON MY JOB. I WON’T LET YOU DOWN. DATA WAREHOUSE HOME SWEET HOME MR. ANALYTICSDAP, XYZ UNIVERSITY

82 FIVE YEARS LATER UNI. OF DOGGED COMPLIANCE & INNOVATION (UDCI) MOST ICT COMPLIANT UNI. IN NUS 2018 BEST ALL ROUND UNIV. IN NUS 2018 MY DYNAMIC VC SIR. UDIC IS NOW RATED AS A WORLD CLASS UNIV. HOW DID YOU PERFORM THIS FEAT IN JUST 3 YEARS. MY AMIABLE VC SIR. I GIVE GOD ALL THE GLORY. MY SECRET IS EFFECTIVE STRATEGIC PLANNING THRU DATA WAREHOUSE BASED ANALYSIS

83 GOODWILL MESSAGE WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!

84 YES WE CAN NUS WORLD CLASS STATUS 2020

85 YES WE CAN THE SKY IS THE LIMIT

86 CLOSING REMARKS CODAPNU NUC PARTNERING TO ATTAIN WORLD CLASS STATUS

87

88

89 THE END

90

91 LEVEL OF ICT DEPLOYMENT IN NUS UNIVERSITY PORTAL POSTING OF VITAL INFORMATION ON- LINE ICT PARKS LAPTOPS FOR STAFF INTERNET CONNECTIVITY, AREAS GIVEN HIGH PRIORITY BY UNIVERSITIES

92 LEVEL OF ICT DEPLOYMENT IN NUS NUC ICT TEACHING AIDS? DATA BASES? DIGITAL VIRTUAL LBRARY? SIRS, WHAT ABOUT THESE ? PUBLISHING PAPERS ON-LINE? SATELLITE? VIDEO CONFERENCING? LOWER PRIORITY AREAS DIGITAL AUDIOS? ICT DRIVEN OPERATIONS? E-LEARNING SUPPORT & DEVELOPMENT ? Web Presence, Visibility and Access?. IP(Internet Protocol) Phones?

93 AREAS GIVEN PRIORITY BY UNIVERSITIES ICT Infrastructure  Internet Connectivity  ICT Parks  Laptops for Staff ICT Integration  University Portal  Posting of vital information on-line

94 LOWER PRIORITY AREAS Low level of ICT infrastructure Across Board  Digital Virtual Library  Satellite  Automated overhead projectors  Digital audios  IP(Internet Protocol) Phones

95 LOWER PRIORITY AREAS Low Level ICT Integration Across Board  Teaching  Publishing papers on-line  Advertising research proposals on-line  Work stations  Data bases

96 LEVEL OF ICT DEPLOYMENT IN NUS Findings NUS: LESS THAN 5% ICT OPERATIONS IN PROCESSES (Vision 20:2020 ICT Technical Report (2009) ) ICT CAPACITY & INFRASTRUCTURE ICT INTEGRATION ICT DEPLOYMENT IN OPERATIONS & SERVICE DELIVERY

97 REASONS FOR LOW LEVEL ICT DEPLOYMENT HIGH COSTS OF INFRASTRUCTURE, EQUIPMENT & ELECTRICITY; INADEQUATE FUNDING; LOW ICT SKILLS; INSUFFICIENT TRAINING; LOW PRIORITY RATING; RESISTANCE TO SOME ICT INITIATIVES; UNSUSTAINED INITIATIVES; & RESISTANCE TO CHANGE.

98 RECOMMENDATIONS Policies for higher ICT integration Serious investment in ICT infrastructures; Infrastructural support} Massive training and } Deployment of ICT skilled manpower} (Government to Universities)

99 RECOMMENDATIONS Collaboration with: Private Sector; International donors; and Civil Society (affordable and sustainable access to ICT infrastructure) In-service training programmes on effective ICT integration

100 RECOMMENDATIONS ICT integration competencies for promotion of staff; Improved Power Supply; Functional Records Management Programme; University Archive; and Benchmarking of functional Records Management Programme and University Archives by NUC.

101 THE WAY FORWARD Data Warehousing Initiative Realistic? Reserve for Affluent Universities? K.I.V.?

102 THE WAY FORWARD Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View LETS DO IT NUC

103

104

105

106

107 THE WAY FORWARD PLEASE LISTEN TO THE FOLLOWING ANNOUNCEMENTS 1. THE DATA WAREHOUSE PROJECT HAS NUC’S BACKING DAPS NUS

108 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF ARE AS FOLLOWS: A. LOBBY YOUR RESPECTIVE VICE- CHANCELLORS

109 THE WAY FORWARD

110

111 MR. ANALYTICS FINDS A HOME CONGRATS! I HOPE YOU LIKE THE HOUSE MAGT. BOUGHT FOR YOU. I’M SO GRATEFUL MADAM DAP. NOW I CAN REALLY CONCENTRATE ON MY JOB. I WON’T LET YOU DOWN. DATA WAREHOUSE HOME SWEET HOME MR. ANALYTICSDAP, XYZ UNIVERSITY

112 FIVE YEARS LATER UNI. OF DOGGED COMPLIANCE & INNOVATION (UDCI) MOST ICT COMPLIANT UNI. IN NUS 2018 BEST ALL ROUND UNIV. IN NUS 2018 MY DYNAMIC VC SIR. UDIC IS NOW RATED AS A WORLD CLASS UNIV. HOW DID YOU PERFORM THIS FEAT IN JUST 3 YEARS. MY AMIABLE VC SIR. I GIVE GOD ALL THE GLORY. MY SECRET IS EFFECTIVE STRATEGIC PLANNING THRU DATA WAREHOUSE BASED ANALYSIS

113 GOODWILL MESSAGE WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!

114 YES WE CAN NUS WORLD CLASS STATUS 2020

115 YES WE CAN THE SKY IS THE LIMIT

116 CLOSING REMARKS CODAPNU NUC PARTNERING TO ATTAIN WORLD CLASS STATUS

117

118

119 THE END

120

121

122

123

124

125

126 LEVEL OF ICT COMPLIANCE

127

128

129

130


Download ppt "DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES BY PROF. CHIEDU F. MAFIANA DIRECTOR, QUALITY ASSURANCE NATIONAL UNIVERSITIES COMMISSION Being a paper presented."

Similar presentations


Ads by Google