Presentation is loading. Please wait.

Presentation is loading. Please wait.

DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES

Similar presentations


Presentation on theme: "DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES"— Presentation transcript:

1 DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES
BY PROF. CHIEDU F. MAFIANA DIRECTOR, QUALITY ASSURANCE NATIONAL UNIVERSITIES COMMISSION Being a paper presented at the 2013 Training Workshop of the Committee of Directors of Academic Planning of Nigerian Universities (CODAPNU) at the National Mathematical Centre, Kwali, Abuja from 28 – 31 October 2013

2 PARTNERING FOR PROGRESS & QUALITATIVE DEVELOPMENT
FELICITATIONS! NUC CODAPNU PARTNERING FOR PROGRESS & QUALITATIVE DEVELOPMENT

3 21ST CENTURY CHALLENGES WE ARE THE WORLD
THE WORLD IS NOW A GLOBAL VILLAGE! HOW ARE UNIVERSITIES COPING WITH 21ST CENTURY CHALLENGES ? WE ARE THE WORLD

4 21ST CENTURY CHALLENGES NUC CROSS BORDER EDUCATION NATION BUILDING
GLOBALIZATION ACCESS RELEVANCE OF CURRICULA ETC ETC QA ISSUES FUNDING NUC

5 21ST CENTURY CHALLENGES HELP! DAP DATA, DATA EVERYWHERE!

6 21ST CENTURY CHALLENGES BUT ARE WE MAKING GOOD USE OF ALL THE DATA?

7 MR ANALYTICS AT YOUR SERVICE
QUALITY INPUT + EFFECTIVE PROCESSES + COMPLIANCE WITH NUC GUIDELINES = QUALITY OUTPUT

8 DATA WAREHOUSING? NEVER HEARD OF IT BUT MUSTN’T DISPLAY MY IGNORANCE!

9 CONCEPTS & DEFINITIONS
What is a Data Warehouse? “It is a subject oriented, nonvolatile, integrated, time variant collection of data in support of management's decisions” Centralized Data warehouses large size for big businesses MEET BILL INMON FATHER OF WAREHOUSING 1970s

10 CONCEPTS & DEFINITIONS
What is a Data Warehouse? “A database of snapshots and aggregations of data throughout an enterprise to be used for querying and decision-making” INTEGRATED SYSTEMS, SMALLER DATA MARTS FOR SMALLER BUSINESSES AND BUDGETS MEET RALPH KIMBALL 2ND FATHER OF WAREHOUSING 1998

11 CONCEPTS & DEFINITIONS
? METADATA ? ? SNOWFLAKE SCHEMA OLAP LEGACY CODE HMMM! IT’S GOING TO BE A LONG, LONG DAY ? STAR SCHEMA DATA MART ? ? HYPERCUBE

12 CONCEPTS AND DEFINITIONS
Hmmm! At least now I know what Data Warehouse is. I hope the process will be clearer after the Case Study . Oga teacher U try. Thank U Sir.

13 TABLE I: COMPARISON OF DW DATA AND OP DATA
OPERATIONAL DATA DW DATA Application oriented Subject oriented Detailed Summarized, otherwise refined Accurate, as of the moment of access Represents values over time, snapshots Serves the clerical community Serves the managerial community

14 TABLE I: COMPARISON OF DW DATA AND OP DATA
OPERATIONAL DATA DW DATA Can be updated Is not updated Run repetitively and non reflectively Run heuristically Requirements for processing understood before initial development Requirements for processing not completely understood before development Compatible with the Software Development Life Cycle Completely different life cycle

15 TABLE I: COMPARISON OF DW DATA AND OP DATA
OPERATIONAL DATA DW DATA Performance sensitive (immediate response required when entering a transaction) Performance relaxed (immediacy not required) Accessed a unit at a time (limited number of data elements for a single record) Accessed a set at a time (many records of many data elements) Transaction driven Analysis driven Control of update a major concern in terms of ownership Control of update no issue

16 TABLE I: COMPARISON OF DW DATA AND OP DATA
OPERATIONAL DATA DW DATA High availability Relaxed availability Managed in its entirety Managed by subsets Non-redundancy Redundancy is a fact of life Static structure; variable contents Flexible structure Small amount of data used in process Large amount of data used in a process

17 Do I Need a Data Warehouse?
CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) No Idea. Do You? Q1 Do I Need a Data Warehouse?

18 CASE STUDY Leave me alone. I have my own problems Q2
What Specific Problems Will It Solve?

19 What are my available resources (time, money, & personnel ?
CASE STUDY Am I your P.A? Q3 What are my available resources (time, money, & personnel ?

20 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com)
Your Hummer Jeep. Remember me in your kingdom. Q4 What criteria will I use to measure my success ?

21 CASE STUDY Just make sure you don’t get lynched in the process Q5
Should I outsource all, some or none of the development and operation?

22 If your DICT does not know, sack him
CASE STUDY If your DICT does not know, sack him Q6 Am I upgrading an existing system, converting from a legacy system, or developing from scratch ?

23 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 1 REQUIREMENTS ANALYSIS 2 INFORMATION/DATA MODELLING 3 DESIGN & PROTOTYPING

24 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES a. RAPID PROTOTYPING b. STRUCTURED DEVELOPMENT 4. DEVELOPMENT & DOCUMENTATION

25 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 5. TEST & REVIEW 6. DEPLOYMENT & TRAINING 7. OPERATION

26 CASE STUDY A Logical Approach to Data Warehouse Design (Horsburgh.com) PROCESSES 8. ENHANCEMENT 9. HELP DESK

27 CHALLENGES PART I

28 CHALLENGES PART I 1 ENSURING ACCEPTABLE DATA QUALITY a.
DISPARATE DATA SOURCES b. UNSTABILIZED SOURCE SYSTEMS

29 CHALLENGES PART I 2 ENSURING ACCEPTABLE PERFORMANCE a.
PRIORITIZING PERFORMANCE b. SETTING REALISTIC GOALS

30 CHALLENGES PART I c. PERFORMANCE BY DESIGN 3.
TESTING THE DATA WAREHOUSE a. TEST PLANNING

31 CHALLENGES PART I b. NO AUTOMATED TESTING 4. RECONCILIATION OF DATA a.
COMPLEX

32 CHALLENGES PART I 5. USER ACCEPTANCE a. RELUCTANT USERS b. TRAIN THEM

33 CHALLENGES PART 2

34 CHALLENGES PART 2 1. USER EXPECTATION 2. SYSTEMS OPTIMIZATION 3.
DATA STRUCTURING

35 CHALLENGES PART 2 4. PREFABRICATED VS CUSTOM WAREHOUSE 5.
DATA STRUCTURING

36 THE LARGEST CHALLENGE OF THEM ALL
GARBAGE IN GARBAGE OUT

37 SOURCE: PRACTICAL LIFE CHALLENGES IN IMPLEMENTING DATA WHAREHOUSE – Subramanya Hoysala 17th June 2011 QUALITY OF DATA

38 LOW QUALITY DATA Accuracy Completeness Consistency Correctness

39 LOW QUALITY DATA Integrity Timelines Uniqueness

40 LOW QUALITY DATA FACTORS Lack of data validation in ERP-Systems;
Older technologies do not support eg. drop down lists; The precise business rules are not known to many; People are busy;

41 LOW QUALITY DATA FACTORS
Psychology: Correct data are not important to the user, who registers them, eg. the car sales person; Data may be loaded / corrected in one-off batch runs; and The rules have changed, but users are not aware of it.

42 LOW QUALITY DATA WHAT DO WE DO ABOUT IT?

43 LOW QUALITY DATA CORRECTIVE STEPS CLEANSING & ENHANCEMENT PROFILING
CONSOLIDATION

44 LOW QUALITY DATA CORRECTIVE STEPS AUDITING HETEROGENEOUS DATA
DATA INTEGRATION

45 SAMPLE UDW Source: New York University, Reporting Home page

46 EMERGING TRENDS CLOUD INTEGRATION REAL TIME DATA WAREHOUSING
IN-MEMORY DBMSs BIG DATA

47 OVERVIEW: ICT IN NUS 2015 2014 2013 WORLD SUMMIT INFORMATION SOCIETY
GLOBAL TARGET ICT FOR ALL 2015 2015 2014 2013

48 MY DEAR SON I HAVE SO MANY GIFTS FOR YOU
ICT INITIATIVES IN NUS NUTALP VLP MY DEAR SON I HAVE SO MANY GIFTS FOR YOU ACE NUSMAP NLOP DADDY! THANK YOU SIR NUC PRESSID NUS NgREN

49 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE
1 STUDENT INFORMATION STAFF INFORMATION 2 UNIVERSITIES PROGRAMME ACCREDITATION BUDGETS AND EXPENDITURE 3 ENROLMENT/STATISTICS PHYSICAL PLANNING & DEVELOPMENT

50 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE 4
ACADEMI C PROGRAMMES 5 UNIVERSITY GEOGRAPHIC INFORMATION SYSTEM 6 RESEARCH AND INNOVATIONS

51 ICT INITIATIVES IN NUS NUSMAP – FEATURES 1 ROBUST ADMINISTRATION 2
SECURED LOGIN SYSTEM 3 UNIQUE IDENTITY FOR EACH SYSTEM

52 ICT INITIATIVES IN NUS NUSMAP – NUC DATA WAREHOUSING INITIATIVE 4
USER FRIENDLY 5 CAPTURES A VAST RANGE OF INFORMATION 6 AUTOMTIC POPULATION OF STUDENTS & STAFF RECORDS

53 ICT INITIATIVES IN NUS NUSMAP – FEATURES 7
GENERATION OF PRINTALE REPORTS IN NUC-SPECIFIED FORMAT 8 EXPORT FEATURE 9 PROVISION FOR NUC TO VIEW SUBMITTED REPORTS

54 ICT INITIATIVES IN NUS NUSMAP – FEATURES 10
GENERATION OF PRINTALE REPORTS IN NUC-SPECIFIED FORMAT 11 AUTOMATIC CALCULTION OF FULL TIME EQUIVALENT (FTE) 12 AUTOMATIC SUMMATION/CALCULATION CAPABILITY

55 ICT INITIATIVES IN NUS NUSMAP – FEATURES 13
AUTOMATIC NOTIFICATION ON EXPIRY OF A PROJECT’S MASTERPLAN 14 AUTOMATIC GENERATION OF ANNUAL PAYROLL REPORT 15 STAFF PROGRESSION/STAFF AREA OF SPECIALIZATION AT A GLANCE

56 ICT INITIATIVES IN NUS NUSMAP – FEATURES 16
FACILITIES ONLINE UNIVERSITIES PROGRAMME ACCREDITATION 17 USARM NOW ONLINE

57 ICT INITIATIVES IN NUS NUSMAP – CHALLENGES 1
INITIAL RESISTANCE FROM SOME UNIVERSITIES 2 FORGOTTEN PASSWORDS PROBLEM OF UPLODING DUE TO WRONG ARRANGEMENT OF SOME EXCEL FILES

58 LEVEL OF ICT DEPLOYMENT IN NUS
AREAS GIVEN HIGH PRIORITY BY UNIVERSITIES ICT PARKS INTERNET CONNECTIVITY, LAPTOPS FOR STAFF UNIVERSITY PORTAL POSTING OF VITAL INFORMATION ON-LINE

59 LEVEL OF ICT DEPLOYMENT IN NUS
NUC LOWER PRIORITY AREAS SIRS, WHAT ABOUT THESE ? Web Presence, Visibility and Access?. DIGITAL VIRTUAL LBRARY? DATA BASES? ICT TEACHING AIDS? E-LEARNING SUPPORT & DEVELOPMENT ? PUBLISHING PAPERS ON-LINE? ICT DRIVEN OPERATIONS? SATELLITE? VIDEO CONFERENCING? DIGITAL AUDIOS? IP(Internet Protocol) Phones?

60 AREAS GIVEN PRIORITY BY UNIVERSITIES
ICT Infrastructure Internet Connectivity ICT Parks Laptops for Staff ICT Integration University Portal Posting of vital information on-line

61 LOWER PRIORITY AREAS Low level of ICT infrastructure Across Board
Digital Virtual Library Satellite Automated overhead projectors Digital audios IP(Internet Protocol) Phones

62 LOWER PRIORITY AREAS Low Level ICT Integration Across Board Teaching
Publishing papers on-line Advertising research proposals on-line Work stations Data bases

63 LEVEL OF ICT DEPLOYMENT IN NUS
Findings ICT CAPACITY & INFRASTRUCTURE ICT INTEGRATION ICT DEPLOYMENT IN OPERATIONS & SERVICE DELIVERY NUS: LESS THAN 5% ICT OPERATIONS IN PROCESSES (Vision 20:2020 ICT Technical Report (2009) )

64 REASONS FOR LOW LEVEL ICT DEPLOYMENT
HIGH COSTS OF INFRASTRUCTURE, EQUIPMENT & ELECTRICITY; INADEQUATE FUNDING; LOW ICT SKILLS; INSUFFICIENT TRAINING; LOW PRIORITY RATING; RESISTANCE TO SOME ICT INITIATIVES; UNSUSTAINED INITIATIVES; & RESISTANCE TO CHANGE.

65 RECOMMENDATIONS Policies for higher ICT integration
Serious investment in ICT infrastructures; Infrastructural support } Massive training and } Deployment of ICT skilled manpower } (Government to Universities)

66 RECOMMENDATIONS Collaboration with: Private Sector;
International donors; and Civil Society (affordable and sustainable access to ICT infrastructure) In-service training programmes on effective ICT integration

67 RECOMMENDATIONS ICT integration competencies for promotion of staff;
Improved Power Supply; Functional Records Management Programme; University Archive; and Benchmarking of functional Records Management Programme and University Archives by NUC.

68 THE WAY FORWARD Data Warehousing Initiative Realistic?
Reserve for Affluent Universities? K.I.V.?

69 THE WAY FORWARD LETS DO IT NUC Questions Data Warehousing Initiative
Realistic? Purview of Affluent Universities? Kept-in-View Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View THE WAY FORWARD LETS DO IT NUC

70 THE WAY FORWARD PLEASE LISTEN TO THE FOLLOWING ANNOUNCEMENTS
1. THE DATA WAREHOUSE PROJECT HAS NUC’S BACKING DAPS NUS

71 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF ARE AS FOLLOWS:
A. LOBBY YOUR RESPECTIVE VICE-CHANCELLORS DAPS NUS

72 THE WAY FORWARD WAREHOUSE? WHAT ABOUT THE ONE WE BUILT FOR THE CONSULT LAST YEAR? SORRY SIR, I MEAN THE ICT DATA WAREHOUSE PROJECT. IT’S VERY USEFUL FOR SRATEGIC PLANNING PURPOSES SIR. VC, SIR, I’VE COME TO DISCUSS THE ISSUE OF THE DATA WAREHOUSE PROJECT COME UP WITH A PROPOSAL PLEASE VICE-CHANCELLOR’S OFFICE UNIVERSITY OF DOGGED COMPLIANCE & INNOVATION (UDCI)

73 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF :
B. SOLICIT FOR SUPPORT FROM YOUR DICT & DQA GENTLEMEN PLS I NEED YOUR KIND ASSISTANCE AND SUPPORT IN PACKAGING A VIABLE PROPOSAL FOR THE DATA WAREHOUSE PROJECT DQA DICT

74 SOME BENEFITS OF DATA WAREHOUSING
1 Has a subject area orientation 2 Integrates data from multiple, diverse sources 3 Allows for analysis of data over time

75 SOME BENEFITS OF DATA WAREHOUSING
4 Adds ad hoc reporting and enquiry 5 Provides analysis capabilities to decision makers 6 Relieves the development burden on IT

76 SOME BENEFITS OF DATA WAREHOUSING
7 Provides improved performance for complex analytical queries 8 Relieves processing burden on transaction oriented databases 9 Allows for a continuous planning process

77 COSTS OF DATA WAREHOUSING
1 Time spent in careful analysis of measurable 2 Design and implementation effort 3 Hardware costs

78 COSTS OF DATA WAREHOUSING
4 Software costs 5 On-going support and maintenance 6 Resulting re-engineering effort

79 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF :
C. CONSTRUCTION OF AN INTEGRATED INFORMATION ARCHITECTURE LADIES AND GENTS WE HAVE 3 TASKS Building an integrated information architecture Taxonomy of Administrative Activities Data Dictionary . BRAINSTORMING COMMITTEE

80 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF :
D. CONSULTATION WITH THE DICT MY DICT SIR, SO WHEN ARE WE GOING TO MEET TO DISCUSS THE DATA WAREHOUSE WORKSHOP MATRIALS MY ABLE DAP SIR. LET’S MEET ON FRIDAY 2 PM IN MY OFFICE DICT DAP

81 MR. ANALYTICS FINDS A HOME
DATA WAREHOUSE HOME SWEET HOME I’M SO GRATEFUL MADAM DAP. NOW I CAN REALLY CONCENTRATE ON MY JOB. I WON’T LET YOU DOWN. CONGRATS! I HOPE YOU LIKE THE HOUSE MAGT. BOUGHT FOR YOU. MR. ANALYTICS DAP, XYZ UNIVERSITY

82 FIVE YEARS LATER MY DYNAMIC VC SIR. UDIC IS NOW RATED AS A WORLD CLASS UNIV. HOW DID YOU PERFORM THIS FEAT IN JUST 3 YEARS. UNI. OF DOGGED COMPLIANCE & INNOVATION (UDCI) MOST ICT COMPLIANT UNI. IN NUS 2018 BEST ALL ROUND UNIV. IN NUS 2018 MY AMIABLE VC SIR. I GIVE GOD ALL THE GLORY. MY SECRET IS EFFECTIVE STRATEGIC PLANNING THRU DATA WAREHOUSE BASED ANALYSIS

83 WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!
GOODWILL MESSAGE WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!

84 YES WE CAN WORLD CLASS STATUS 2020 NUS

85 YES WE CAN THE SKY IS THE LIMIT

86 CLOSING REMARKS CODAPNU NUC PARTNERING TO ATTAIN WORLD CLASS STATUS

87

88

89 THE END

90

91 LEVEL OF ICT DEPLOYMENT IN NUS
AREAS GIVEN HIGH PRIORITY BY UNIVERSITIES ICT PARKS INTERNET CONNECTIVITY, LAPTOPS FOR STAFF UNIVERSITY PORTAL POSTING OF VITAL INFORMATION ON-LINE

92 LEVEL OF ICT DEPLOYMENT IN NUS
NUC LOWER PRIORITY AREAS SIRS, WHAT ABOUT THESE ? Web Presence, Visibility and Access?. DIGITAL VIRTUAL LBRARY? DATA BASES? ICT TEACHING AIDS? E-LEARNING SUPPORT & DEVELOPMENT ? PUBLISHING PAPERS ON-LINE? ICT DRIVEN OPERATIONS? SATELLITE? VIDEO CONFERENCING? DIGITAL AUDIOS? IP(Internet Protocol) Phones?

93 AREAS GIVEN PRIORITY BY UNIVERSITIES
ICT Infrastructure Internet Connectivity ICT Parks Laptops for Staff ICT Integration University Portal Posting of vital information on-line

94 LOWER PRIORITY AREAS Low level of ICT infrastructure Across Board
Digital Virtual Library Satellite Automated overhead projectors Digital audios IP(Internet Protocol) Phones

95 LOWER PRIORITY AREAS Low Level ICT Integration Across Board Teaching
Publishing papers on-line Advertising research proposals on-line Work stations Data bases

96 LEVEL OF ICT DEPLOYMENT IN NUS
Findings ICT CAPACITY & INFRASTRUCTURE ICT INTEGRATION ICT DEPLOYMENT IN OPERATIONS & SERVICE DELIVERY NUS: LESS THAN 5% ICT OPERATIONS IN PROCESSES (Vision 20:2020 ICT Technical Report (2009) )

97 REASONS FOR LOW LEVEL ICT DEPLOYMENT
HIGH COSTS OF INFRASTRUCTURE, EQUIPMENT & ELECTRICITY; INADEQUATE FUNDING; LOW ICT SKILLS; INSUFFICIENT TRAINING; LOW PRIORITY RATING; RESISTANCE TO SOME ICT INITIATIVES; UNSUSTAINED INITIATIVES; & RESISTANCE TO CHANGE.

98 RECOMMENDATIONS Policies for higher ICT integration
Serious investment in ICT infrastructures; Infrastructural support } Massive training and } Deployment of ICT skilled manpower } (Government to Universities)

99 RECOMMENDATIONS Collaboration with: Private Sector;
International donors; and Civil Society (affordable and sustainable access to ICT infrastructure) In-service training programmes on effective ICT integration

100 RECOMMENDATIONS ICT integration competencies for promotion of staff;
Improved Power Supply; Functional Records Management Programme; University Archive; and Benchmarking of functional Records Management Programme and University Archives by NUC.

101 THE WAY FORWARD Data Warehousing Initiative Realistic?
Reserve for Affluent Universities? K.I.V.?

102 THE WAY FORWARD LETS DO IT NUC Questions Data Warehousing Initiative
Realistic? Purview of Affluent Universities? Kept-in-View Questions Data Warehousing Initiative Realistic? Purview of Affluent Universities? Kept-in-View THE WAY FORWARD LETS DO IT NUC

103

104

105

106

107 THE WAY FORWARD PLEASE LISTEN TO THE FOLLOWING ANNOUNCEMENTS
1. THE DATA WAREHOUSE PROJECT HAS NUC’S BACKING DAPS NUS

108 THE WAY FORWARD 2. IMMEDIATE STRATEGIES FOR TAKEOFF ARE AS FOLLOWS:
A. LOBBY YOUR RESPECTIVE VICE-CHANCELLORS

109 THE WAY FORWARD

110

111 MR. ANALYTICS FINDS A HOME
DATA WAREHOUSE HOME SWEET HOME I’M SO GRATEFUL MADAM DAP. NOW I CAN REALLY CONCENTRATE ON MY JOB. I WON’T LET YOU DOWN. CONGRATS! I HOPE YOU LIKE THE HOUSE MAGT. BOUGHT FOR YOU. MR. ANALYTICS DAP, XYZ UNIVERSITY

112 FIVE YEARS LATER MY DYNAMIC VC SIR. UDIC IS NOW RATED AS A WORLD CLASS UNIV. HOW DID YOU PERFORM THIS FEAT IN JUST 3 YEARS. UNI. OF DOGGED COMPLIANCE & INNOVATION (UDCI) MOST ICT COMPLIANT UNI. IN NUS 2018 BEST ALL ROUND UNIV. IN NUS 2018 MY AMIABLE VC SIR. I GIVE GOD ALL THE GLORY. MY SECRET IS EFFECTIVE STRATEGIC PLANNING THRU DATA WAREHOUSE BASED ANALYSIS

113 WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!
GOODWILL MESSAGE WISHING ALL UNIVERSITIES AS MUCH SUCCESS AS UDCI!

114 YES WE CAN WORLD CLASS STATUS 2020 NUS

115 YES WE CAN THE SKY IS THE LIMIT

116 CLOSING REMARKS CODAPNU NUC PARTNERING TO ATTAIN WORLD CLASS STATUS

117

118

119 THE END

120

121

122

123

124

125

126 LEVEL OF ICT COMPLIANCE

127

128

129

130


Download ppt "DATA WAREHOUSING, METHODOLOGIES AND CHALLENGES"

Similar presentations


Ads by Google