Presentation is loading. Please wait.

Presentation is loading. Please wait.

TH Technology Karen Cannell How Do I Load Data … Let Me Count The Ways APEX Data Loading Options.

Similar presentations


Presentation on theme: "TH Technology Karen Cannell How Do I Load Data … Let Me Count The Ways APEX Data Loading Options."— Presentation transcript:

1 TH Technology Karen Cannell How Do I Load Data … Let Me Count The Ways APEX Data Loading Options

2 How Do I Load Data … TH Technology How Do I Load Data? : Agenda Oracle Data Loading Options DBAs - Developers - End Users APEX Data Loading Options File Upload Data Load Wizard 2 XLS Upload – APEX Listener WebSheets – Copy/Paste, Add Row

3 TH Technology How Do I Load Data … About Me … Karen Cannell ~ Consultant, TH Technology SW Engineer 25+ years, Oracle since Building APEX apps for government, medical, engineering industries. Leveraging the Oracle 10g,11g (now 12c) suite of tools Beginning Application Express, APress, 2011 Agile Application Express, APress, 2011 Editor, ODTUG Technical Journal Volunteer to author ODTUG Journal Articles! Using APEX since the HTMLDB beginning

4 How Do I Load Data … TH Technology About You … ( Audience Background) New to APEX? APEX Experience? Previous Tools? APEX Training? Version 2.0? 3.1 ? 3.2 ? 4.1?

5 How Do I Load Data … TH Technology How DO I Load Data into Oracle? Load Data Into Oracle 7,340,000 hits SQL Loader Oracle Database Utilities Load Excel Data into Oracle

6 How Do I Load Data … TH Technology How DO I Load Data into Oracle? Load Data Into APEX 317,000 hits Import Excel Data … APEX Data Loader Load Text File into APEX

7 How Do I Load Data … TH Technology How DO I Load Data into Oracle? Load XLS Data Into APEX 1,830,000 hits Import Excel File … APEX Data Loader Load Excel Data … Yikes!

8 TH Technology How Do I Load Data … Load Data into Oracle - Options What Are the Options? Commercial Solutions - Excel-DB, Quickload Warehouse Builder, ETL Tools Oracle Data Integrator integrator/overview/index.html Oracle Database Utilities Oracle Data Pump SQL Loader External Tables

9 TH Technology How Do I Load Data … Load Data into Oracle, contd ODBC Connections IDEs w/built-In Data Loaders Custom Code: Java PL/SQL Using UTL_FILE Scripting from XLS Columns Java and PL/SQL Perl Other

10 TH Technology How Do I Load Data … Lets Focus …

11 TH Technology How Do I Load Data … Load Data into APEX Import Excel Data into Oracle … APEX Data Loader Load Excel Data 95% Say XLS but Really are CSV The Apex Data Loader is a java based application that can bulk process the insert, update and delete on all object data into and build queries to extract data out of salesforce.com using the Apex Web Services (SOAP) API. salesforce.com

12 TH Technology How Do I Load Data … Our Problem Load Data into Oracle 40% Fixed-Format Text Files 60% Excel Spreadsheets Several XLS Formats 200+ Files Monthly Locations All Over the World Some Networked, Some Not Varying Volume of Data

13 TH Technology How Do I Load Data … Which to Choose? It Depends Application Source Target ( Table?) Amount of Data How Often ? 1 or Many Audience End Users Resources $$ Time $$

14 TH Technology How Do I Load Data … Which Make Sense for APEX ? Oracle Database Utilities PL/SQL IDE No Additional Licenses PL/SQL Code No Complex Setup Developer-Friendly End User Friendly

15 TH Technology How Do I Load Data … Oracle Database Utilities Oracle Data Pump Oracle DB to Oracle DB Faster than EXP and IMP EXPDP 2x faster than EXP IMPDP 15-45x faster than IMP Jobs can be Restarted Supports Network Import and Export Load one instance form another Remote export

16 TH Technology How Do I Load Data … Oracle Database Utilities, contd SQL Loader External files into Tables Variety of formats Filtering Multiple Table Load in One Load Session Conventional Direct Path External Table Load

17 TH Technology How Do I Load Data … Oracle Database Utilities, contd External Tables External files into Tables Variety of Formats Preprocessing Capability External Tables to Pull RPT Files CSV Files - That We Can Get

18 TH Technology How Do I Load Data … Oracle Database – Database Link Dont Forget the Database Link! SELECT … FROM Simple, Direct Requires Communication Ask!

19 TH Technology How Do I Load Data … Use your IDE SQL Developer PL/SQL Developer Toad Others … Best for One-Time Load

20 TH Technology How Do I Load Data … SQL Developer – Data Load Option Import Data … Wizard

21 TH Technology How Do I Load Data … SQL Developer – Import Data … Import File Formats.XLS / XLSX.CSV.TSV (tab).DSV ( SQL Server)

22 TH Technology How Do I Load Data … SQL Developer – Import Data … Import Wiz Detects File type Delimiter Skip Row

23 TH Technology How Do I Load Data … SQL Developer – Import Data … INSERT or Generate INSERT Script)

24 TH Technology How Do I Load Data … SQL Developer – Import Data … Select Columns to Import

25 TH Technology How Do I Load Data … SQL Developer – Import Data … Map Source to Table Columns

26 TH Technology How Do I Load Data … SQL Developer – Import Data … Verify

27 TH Technology How Do I Load Data … SQL Developer – Import Data … Finished! XLS and XLSX* File Load in Very Few Clicks! *XLSX in SQL Developer 3.1

28 TH Technology How Do I Load Data … SQL Developer – Import Data … What About XLSX? SQL Dev 3.0 – No SQL Developer 3.1 – Yes XLS/XLSX File Load in Very Few Clicks!

29 TH Technology How Do I Load Data … SQL Developer – Import Data … Pros One Shot Data Load Delimiter (, TAB, others) Flexible Columns Flexible Mapping XLS File Can be Faster than APEX Data Load Wizard Cons Table Must Be Defined Cannot Integrate into App Tedious to Repeat IDE Specific One Shot Data Load Not for End Users

30 TH Technology How Do I Load Data … APEX Data Load/Unload Utility APEX From the Beginning Easy Load of Spreadsheet or XML Data New or Existing Table Upload File Or Copy/Paste Delimiter Option Column by Column Selection

31 TH Technology How Do I Load Data … APEX Data Load/Unload Utility

32 TH Technology How Do I Load Data … APEX Data Load/Unload Utility 1 st Row Column Heading

33 TH Technology How Do I Load Data … APEX Data Load/Unload Utility

34 TH Technology How Do I Load Data … APEX Data Load/Unload Utility Specify PK and Trigger Finish

35 TH Technology How Do I Load Data … APEX Data Load Utility Pros Simple: Point-Click- Done Flexible Format Spreadsheet Convenience Creates Table Copy/Paste Option Cons Many Steps Single Table No XLS Upload (Must Copy/Paste or Save as CSV) One-Time Not for End Users

36 TH Technology How Do I Load Data … What About the End User? APEX Standard File Browse + Custom Code Data Load Wizard Wizard APEX Manual Data Load Wizard End User-ized APEX Web Sheet Data Grid Copy/Paste, Data Entry APEX Listener XLS Upload File Browse + XLS2COLLECTION

37 TH Technology How Do I Load Data … APEX File Browse + Custom Code Most Common (Still) File Browse Item Process (to Staging) Process To Destination Table(s) MANY MANY OPTIONS … All Require Custom Code

38 TH Technology How Do I Load Data … APEX File Browse + Custom Code File Browse Item WWV_FLOW_FILES – Original Way BLOB column – New Option w APEX 4+

39 TH Technology How Do I Load Data … APEX File Browse + Custom Code WWV_FLOW_FILES – Traditional Way File uploads to APEX Table WWV_FLOW_FILES Move Records to a Custom Table (i.e. Staging Table) Clean up! Files accumulate in APEX_FILES Tablespace

40 TH Technology How Do I Load Data … APEX File Browse + Custom Code BLOB Column – New w/ APEX 4 File Stored in BLOB Column of Table Specified in Automatic Row Processing (DML) Process Column Specified in Item Source Table Must Have BLOB Filename MIME Type CHARSET Saves an Archiving Step

41 TH Technology How Do I Load Data … APEX File Browse + Custom Code BLOB Column – New w/ APEX 4

42 TH Technology How Do I Load Data … APEX File Browse + Custom Code WWV_FLOW_FILES

43 TH Technology How Do I Load Data … APEX File Browse + Custom Code LOCAL_UPLOAD_FILES

44 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse / Load Process Read Blob into local BLOB variable Read Records from Blob Load Records into Destination Objects

45 TH Technology How Do I Load Data … APEX File Browse + Custom Code Reusable Parse/ Load Process File Browse Item Load File to WWV_FLOW_FILE Generic Parse File Package Generic File Upload Utility Generic Store to APEX Collection Process Code per Use/Table/Format Custom Code, But Less of It

46 TH Technology How Do I Load Data … Generic Parse File Utility Package Parse File to Collection Load Collection to Specified Table Columns Must Match, or Adjust Code Reusable Code, Pages for Any Upload File/Table Saves One Part of Custom Code Custom Processing from Stage Table Required Custom Code, But Less of It

47 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse File Process Read Blob into local BLOB variable BEGIN SELECT blob_content INTO l_blob FROM wwv_flow_files WHERE name=p_file_name; EXCEPTION WHEN NO_DATA_FOUND THEN raise_application_error(-20000, 'File not found, id='||p_file_name); END; Or BLOB column from your BLOB table

48 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse File Process Read Records from Blob Variable TYPE varchar2_t IS TABLE OF VARCHAR2(32767) INDEX BY binary_integer; l_records VARCHAR2_T; … get_records(l_blob,l_records); … -- Initialize the APEX collection apex_collection.create_or_truncate_collection( p_collection_name);

49 TH Technology How Do I Load Data … PROCEDURE get_records(p_blob IN blob,p_records OUT varchar2_t) IS l_sep VARCHAR2(2) := chr(13)||chr(10); l_last INTEGER; l_current INTEGER; BEGIN IF (NVL(DBMS_LOB.INSTR(p_blob, utl_raw.cast_to_raw(l_sep),1,1),0)=0) THEN l_sep := chr(10); END IF; l_last := 1; LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP; END get_records;

50 TH Technology How Do I Load Data … LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP; Loop to read records from the BLOB, line by line (separator to separator)

51 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse File Process: Read from Blob, Load Collection FOR i IN 2..l_records.count LOOP csv_to_array(l_records(i),l_record); l_seq_id := apex_collection.add_member(p_collection_name,'dummy'); FOR i IN 1..l_record.count LOOP apex_collection.update_member_attribute( p_collection_name=> p_collection_name, p_seq => l_seq_id, p_attr_number => i, p_attr_value => l_record(i)); END LOOP;

52 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse File Process: Read Blob, Load Collection Insert from Collection to Table … l_ddl := INSERT INTO '||p_table_name||' '|| SELECT '||v(p_columns_item)||' '|| FROM apex_collections '|| WHERE seq_id >= 1 and collection_name='''|| p_collection_name||''''; EXECUTE IMMEDIATE l_ddl; …

53 TH Technology How Do I Load Data … APEX File Browse + Custom Code Parse File Process: Read Blob, Load Collection, Insert, Remove File -- Cleanup file! DELETE FROM wwv_flow_files WHERE name=p_file_name;

54 TH Technology How Do I Load Data … APEX File Browse + Custom Code Process to Destination Table(s) INSERT /* +append */ … SELECT … FROM staging MERGE statement Whatever is Required for Your Business Custom Code

55 TH Technology How Do I Load Data … File Browse & Custom Code Pros Custom Code Custom Solution Cons Custom Code Resources Maintenance One-of Solution

56 TH Technology How Do I Load Data … Data Load Wizard Wizard The SQL Workshop Data Load Utility End-User-ized Page Type Creates a Shared Component

57 TH Technology How Do I Load Data … Data Load Wizard Wizard Wizard to Create A Data Load Wizard Create Page Data Loading

58 TH Technology How Do I Load Data … Data Load Wizard Wizard

59 TH Technology How Do I Load Data … Data Load Wizard Wizard Define Table Lookups

60 TH Technology How Do I Load Data … Data Load Wizard Wizard Add Transforms To Upper/Lower, R/L/Trim, Single Whitespace

61 TH Technology How Do I Load Data … Data Load Wizard Wizard Page/Region/Breadcrumb Names

62 TH Technology How Do I Load Data … Data Load Wizard Wizard Define Tabs (if any)

63 TH Technology How Do I Load Data … Data Load Wizard Wizard Define Branches

64 TH Technology How Do I Load Data … Data Load Wizard Wizard Confirmation

65 TH Technology How Do I Load Data … Data Load Wizard - Result Working Upload for our USERS table

66 TH Technology How Do I Load Data … Data Load Wizard End User Sees: Data Load Progression Chance to Select Row to Include/Ignore Failed Rows at the End Chance to Back Out Load? Chance to Reload Clean?

67 TH Technology How Do I Load Data … Data Load Wizard Upload OR Copy/Paste Data Limit == Specify Delimiter Specify Enclosed By Specify Currency, Group Separator, Decimal Characters

68 TH Technology How Do I Load Data … Data Load Wizard - Options

69 TH Technology How Do I Load Data … Data Load Wizard – Pros/Cons Pros Flexible Form Delimiter Option 1 st Row Heading Opt Currency – Group Separator – Decimal Mapping Built In Repeatable for End User Reusable for Developer Cons Too Much for End User?? No XLS Copy/Paste Size Limit Upload Size Limit ?? Single Transform per Column Home

70 TH Technology How Do I Load Data … Data Load Wizard Wizard Is It Foolproof? No: Ex: RPT file …. What does this means? Too Many Columns? Too Few?

71 TH Technology How Do I Load Data … Data Load Wizard Wizard Can I trick it into taking an XLS? No:

72 TH Technology How Do I Load Data … Data Load Wizard Can I Make it Better? Try: Default/Eliminate All Options Eliminate Load/Unload Column Mapping Catch/Warn of File Type, # Columns Errors Load the Collection Behind the Schemes

73 TH Technology How Do I Load Data … Data Load Wizard – Behind the Scenes Usual Expected Items FILE_BROWSE

74 TH Technology How Do I Load Data … Data Load Wizard – P 2 Parse Uploaded Data Single Process of Type Parse Uploaded Data List of Parameters Fires upon Next

75 TH Technology How Do I Load Data …

76 TH Technology How Do I Load Data … Data Load Wizard – P 3 Process Uploaded Data Single Process of Type Process Uploaded Data List of Parameters Fires upon Next

77 TH Technology How Do I Load Data … Data Load Wizard – P 3 Session Info Collections Character Columns Numeric Columns Date Columns

78 TH Technology How Do I Load Data …

79 TH Technology How Do I Load Data … Data Load Wizard Wizard Pros In-App Straightforward Reusable No Custom Code for Data Load into DB Cons Too Many End User Steps Still Need Custom Processing Code for Staging to Tables IF get P1 Finish by Loading Collection, have a Two-Step File Upload for End Users!

80 TH Technology How Do I Load Data … APEX Listener XLS Upload File Browse Item Load File to WWV_FLOW_FILE Parsing File Custom Code per Use/Table/Format Generic File Upload Utility Custom Process to Destination Tables

81 TH Technology How Do I Load Data … Parse File to Collection Loads Collection to Specified Table Table Exist or Not Options Reusable for Any Upload File/Table Saves One Part of Custom Code Custom Processing from Stage Table Required APEX Listener XLS Upload

82 TH Technology How Do I Load Data … APEX Listener XLS Upload Demo

83 TH Technology How Do I Load Data … XLS Upload via APEX Listener Must Deploy via APEX Listener Current Release 1.1.3… Allows Upload of XLS File into a Collection Best Reference: (from RMOUG 2010) apex-listener-ea-more-knobs-to.html

84 TH Technology How Do I Load Data … XLS Upload via APEX Listener Install APEX Listener Deploy Edit apex_config.xml Add Configuration Choices: true EXCEL_COLLECTION true

85 TH Technology How Do I Load Data … XLS Upload via APEX Listener Configuration Options: Turn Feature On: true One Collection per Worksheet: true|false

86 TH Technology How Do I Load Data … XLS Upload via APEX Listener Configuration Options: Specify Name for All Collections: true|false false == sequential sheet name true == uses sheet name Specify Name of Collection for All Collections: MY_COLLECTION

87 TH Technology How Do I Load Data … XLS Upload via APEX Listener File Browse Item WWV_FLOW_FILES option Page Item Button Name or REQUEST = XLS2COLLECTION Report Region: SELECT * FROM apex_collections WHERE collection_name = MY_COLLECTION

88 TH Technology How Do I Load Data … XLS Upload via APEX Listener

89 TH Technology How Do I Load Data … XLS Upload via APEX Listener

90 TH Technology How Do I Load Data … XLS Upload via APEX Listener Files ending with.xls or.xlsx will be processed. BUT.xls process Macro-Enabled.xls process.XLSX do NOT Process as of APEX Listener XLSX will Process as of v2 Coming Soon!

91 TH Technology How Do I Load Data … XLSX Upload via APEX Listener

92 TH Technology How Do I Load Data … XLSX Upload via APEX Listener SEVERE: service exception: org.apache.poi.poifs.filesystem.OfficeXmlFileException: The supplied data appears to be in the Office XML. You are calling the part of POI that deals with OLE2 Office Documents. You need to call a different part of POI to process this data (eg XSSF instead of HSSF)

93 TH Technology How Do I Load Data … XLS Upload via APEX Listener Formulas Translate Literally Ex: IF(E2="","",VLOOKUP(E2,$AE$4:$AF$218,2,FALSE))

94 TH Technology How Do I Load Data … XLS Upload via APEX Listener STILL Need Custom Code to Process Data from Collection to Database Table(s)

95 TH Technology How Do I Load Data … APEX Listener XLS Upload Pros XLS File – No Need to Save As CSV Straightforward Reusable Less Custom Code Cons APEX Listener Deployment Embedded Formulas No XLSX - May Need a Save As Step Will Have XLSX in V2

96 TH Technology How Do I Load Data … APEX Websheets Data Grid Allows Copy/Paste Spreadsheet-Like Edits Data Grid w/in Page Custom Code Required to Transfer Data to Main Schema Tables

97 TH Technology How Do I Load Data … Editable Data in Websheet

98 TH Technology How Do I Load Data … APEX Websheets Demo

99 TH Technology How Do I Load Data … Pull Data From Websheet into App Look at APEX$_WS Tables: APEX$_WS_ROW – the data APEX$_WS_HISTORY – iff you care about who changed what when Need to process from SQL Dev table to real tables

100 TH Technology How Do I Load Data … APEX WebSheets Pros Copy/Paste Spreadsheet-Like Entry, Edits End User Can Build Cons Custom Code to Create Websheet, Data Grid? Custom Code Required to Pull Data into Schema Objects Custom Code to Maintain Worksheet / Data Grid More Code Than Other Solutions ?

101 TH Technology How Do I Load Data … Our Solution Problem RPT Files XLS files Different Formats 200+, Monthly Users Cannot Save as CSV Must Support Edits Varying Volume Limited Network Solution Tabular Forms - 4 Macro to Save as RPT External Tables Generated RPT LIS Output RPT Files Upload File Process (to be retired) Custom Processing RPT – LI Systems RPT – Spreadsheet Templates

102 TH Technology How Do I Load Data … APEX Data Load Options

103 TH Technology How Do I Load Data … Nice to Have … Future? 1-Step Data Load Wizard Copy/Paste OR Upload XLSX Easiest for Non-Networked, Low(er) Volume End Users

104 TH Technology How Do I Load Data … Some Last Thoughts … Forms Create/Edit One Row Tabular Forms Enter Many Rows Java Solution.JSP to load XLS directly Presume this means XLSX too!

105 TH Technology How Do I Load Data … How Do I Load Data ? APEX-based Solution Weigh Requirements Source Data End User Capabilities Resources Time No One-Size-Fits-All Apply Common Sense

106 TH Technology How Do I Load Data …

107 TH Technology How Do I Load Data … Common Sense Test … Two men play 5 games of checkers. Each wins the same # of games. There are no ties. How can this be? They are not playing each other.

108 How Do I Load Data … TH Technology References Upload File into APEX oraexplorer.com – Ittichai text-file-and-write-into/ More at avdeo.com In Oracle Milieu sheet-using-oracle-application-express-apex/

109 How Do I Load Data … TH Technology References Oracle Data Integrator integrator/overview/index.html Oracle Database Utilities SQL Loader – External Tables edition/index htm l SQL Developer tools/sql-developer/overview/index.html

110 How Do I Load Data … TH Technology References APEX Listener XLS Upload – Kris Rice Blog apex-listener-ea-more-knobs-to.html A Java-and-PL/SQL based XLS to CSV Option

111 TH Technology How Do I Load Data? Let Me Count The Ways Questions ? Please fill out the evaluations!

112 TH Technology Thank You Please fill out the evaluations! How Do I Load Data? Let Me Count The Ways


Download ppt "TH Technology Karen Cannell How Do I Load Data … Let Me Count The Ways APEX Data Loading Options."

Similar presentations


Ads by Google