Download presentation
Presentation is loading. Please wait.
Published byDarrell Patterson Modified over 6 years ago
1
Data Warehousing Group Teradata Tools & Utilities
Last Updated : 27th April 2004 Center of Excellence Data Warehousing Group Teradata Tools & Utilities
2
Teradata Tools & Utilities
Utilities exploits the capabilities provided by Teradata parallel architecture. All Teradata utilities communicate with Teradata database via Call Level Interface (CLI). CLI can establish multiple sessions with DBS to send multiple task to Teradata at the same time. Utilities are supported on Mainframe as well as Workstation platforms.
3
Teradata Load/Unload Utilities
BTEQ Fastload Multiload FastExport Tpump
4
Client – Server Connectivity
Parser Engine Message Passing Layer AMP Vdisk Gateway S/W Ethernet Adapter TDP CLI Client Req MTDP Client Req. MOSI Channel Adapter Bus Adapter Channel attached system Network
5
Bteq For submitting SQL requests to Teradata database.
Also used widely for reporting purposes. Can read input records from flat file and apply DMLs (INSERT, UPDATE, DELETE) to the Teradata database. Can also export data to client system from Teradata database.
6
Data Import Using Bteq Bteq .IMPORT can be used to load data from a client system to Teradata Database. {DATA} .IMPORT {INDICDATA} FILE = data.dat; {REPORT} {VARTEXT} USING col1(integer), col2(char(10) INSERT INTO table1 VALUES(:col1, :col2); Use REPEAT * if you want all input records to be inserted.
7
Data Export Using Bteq Bteq .EXPORT command can be used to export data records from Teradata database to a client system. {DATA} .EXPORT {INDICDATA} FILE = data.dat; {REPORT} SELECT * from table1 where col1 = 10; .EXPORT RESET;
8
Multiple Sessions Multiple sessions can be established by a client utility to perform multiple tasks in parallel. Multiple sessions are useful for only those types access which use row hash locking (UPI,NUPI,USI). Avoid establishing multiple sessions where these is a full table lock is involved.
9
FastLoad Utility High speed data load tool for large amount of data.
Can load an empty table only. It has restart capability. Data can be loaded in multiple stages. Can not load a table with Secondary Indexes, RIs and Join Indexes defined. INMOD exit routines can be used to load from external data sources.
10
FastLoad Utility Phase 1 Establishes the sessions for FastLoad task.
Host sends data blocks through the sessions. PEs receive the data blocks and directly send them to AMPs. AMPs redistributes the data rows to their target AMPs. AMPs writes the blocks (unsorted) to the disk.
11
FastLoad Utility Phase 2
Phase 2 start when END LOADING command received. AMPs read their blocks and sort them based on row hash sequence. Write the sorted data blocks back to the disk. Fallback copy of the data is created.
12
FastLoad Utility AMPs Data R1 PE R2 R3 Vdisk R4 R5 R6 R7 M R8 P R9 L
13
Example LOGON tdpid/userid,password; BEGIN LOADING Customers
ERRORFILES Custerr1, Custerr2; DEFINE Cust_Id (INTEGER) ,Cust_Name (VARCHAR(25)) FILE = custdata.dat; INSERT INTO Customers VALUES (:Cust_Id, :Cust_Name); END LOADING; LOGOFF; Input data definition Phase 1 Starts Phase 2 Starts
14
FastLoad Error Tables FastLoad uses two error tables.
Error table 1 captures the information of the rows which can not be loaded due to data conversion errors. Error Table 2 captures the information of the rows which can not be applied due to duplicate UPIs. FastLoad discards empty error tables after successful completion of load task.
15
Restartability If CHECKPOINT option is used, a FastLoad job can be restarted from the last checkpoint. No need to re-run the entire FastLoad job. As taking checkpoint is a time consuming task it recommended to have bigger checkpoint intervals for larger tables. By default Fastload take checkpoint at every records.
16
Loading Multiple Data Files
LOGON tdpid/userid,password; BEGIN LOADING Customers ERRORFILES Custerr1, Custerr2; DEFINE Cust_Id (INTEGER) ,Cust_Name (VARCHAR(25)) FILE = custdata1.dat; INSERT INTO Customers VALUES (:Cust_Id, :Cust_Name); END LOADING; LOGOFF; LOGON tdpid/userid,password; BEGIN LOADING Customers ERRORFILES Custerr1, Custerr2; DEFINE Cust_Id (INTEGER) ,Cust_Name (VARCHAR(25)) FILE = custdata2.dat; INSERT INTO Customers VALUES (:Cust_Id, :Cust_Name); END LOADING; LOGOFF;
17
FastLoad Limitations Table should be empty. Only INSERT allowed.
Can not load table with secondary Indexes defined. Loads only one table in one job task.
18
Multiload Used for high volume data maintenance
Supports INSERTs, UPDATEs and DELETEs. Loads up to five tables in a single job. Tables may not necessarily be empty. Block level data maintenance. Input data rows can be applied conditionally to DML requests. Supports UPSERTs. Can not load tables with USI and RI defined. Tables with NUSI can be loaded using Multiload. INMOD exit routines can be used to load from external data sources.
19
Advantage of Multiload over normal Inserts/Updates
Input data buffers are sent directly to AMPs which minimizes PE utilization. All AMPs works in parallel to apply the change. No transient journal and permanent journal overhead. Update up to 5 tables in one Multiload Task.
20
Example .LOGTABLE Logtable001_ml; .LOGON tdp3/user2,tyler;
.BEGIN MLOAD TABLES Employee, Employee_History; .LAYOUT Employee_Trans; .FILLER in_Transcode 1 CHAR(3); .FIELD in_EmpNo * SMALLINT; .FIELD in_DeptNo * SMALLINT; .FIELD in_Salary * DECIMAL (8,2); .DML LABEL Payroll DO INSERT FOR MISSING UPDATE ROWS ; UPDATE Employee SET Salary = :in_Salary WHERE EmpNo = :in_EmpNo; INSERT INTO Employee (EmpNo, Salary) VALUES (:in_EmpNo, :in_Salary); .DML LABEL Terminate ; DELETE FROM Employee WHERE EmpNo = :in_EmpNo; INSERT INTO Employee_History (EmpNo, DeptNo) VALUES (:in_EmpNo, :in_DeptNo); .IMPORT INFILE infile1 LAYOUT Employee_Trans APPLY Payroll WHERE in_Transcode = 'PAY' APPLY Terminate WHERE in_Transcode = 'DEL'; .END MLOAD; .LOGOFF;
21
Two Distinct Multiload Tasks
Import Task Multiple combinations of INSERTs/UPDATEs/DELETEs/UPSERTs are supported. Where condition must be an equality condition on primary Index. Delete Task Can delete large number of rows in parallel. No equality condition on Primary Index value.
22
Phases of Multiload Import Task
Preliminary Phase Start Multiload sessions Create temporary tables. Apply utility locks (disallow DDLs) on the tables. DML Transaction Phase Send DML requests, USING clause and Match-Tags to server. Acquisition Phase Client sends data buffers to server AMPs receives the data blocks are redistributes (by row hash) each data records to its target AMPs. Data rows are accumulated in work tables of corresponding target table. At the end of acquisition phase it puts write locks on the target tables.
23
Phases of Multiload Import Task
Application Phase Apply work table changes to target table. Clean-up Phase Release locks Drop empty error tables, work table and log table. Report load statistics Logoff sessions.
24
Restartability Multiload reinitiated automatically after a database recovery. Multiload job terminated due to a client failure can be resubmitted to continue from the point of failure. Multiload uses Checkpoint and log table mechanism for restart recovery. No user interaction is required.
25
FastExport To export large volume of data from Teradata database to client. Takes advantage of multiple sessions. Fastexport sorts data rows before sending them through sessions. Can be used with INMOD/OUTMOD feature for external data sources and destinations. Restartable feature supported.
26
Example .LOGTABLE sample_log; .LOGON tdpid/username,password;
.BEGIN EXPORT SESSIONS 4; .EXPORT OUTFILE out.dat; SELECT * FROM employees WHERE loc_name = ‘MGRD’ ORDER BY 1; .END EXPORT; .LOGOFF; .LOGTABLE sample_log; .LOGON tdpid/username,password; .BEGIN EXPORT SESSIONS 4; .EXPORT OUTMOD out.dat; SELECT * FROM employees WHERE loc_name = ‘MGRD’ ORDER BY 1; .END EXPORT; .LOGOFF;
27
Tpump Allows near real-time updates from transactional systems into the warehouse. Best fit for low volume data maintenance. Unlike Multiload, Tpump uses row hash lock allows concurrent updates on the same table. INSERT,UPDATE, DELETE supported. No restrictions applied for tables with SI, RI, triggers etc. No limit on number of concurrent sessions. Speed can be tuned dynamically.
28
Example .LOGTABLE Logtable001_ml; .LOGON tdp3/user2,tyler;
.BEGIN LOAD SESSIONS 4 ERRORTABLE ET_Employee; .LAYOUT Employee_Trans; .FILLER in_Transcode 1 CHAR(3); .FIELD in_EmpNo * SMALLINT; .FIELD in_DeptNo * SMALLINT; .FIELD in_Salary * DECIMAL (8,2); .DML LABEL Payroll DO INSERT FOR MISSING UPDATE ROWS ; UPDATE Employee SET Salary = :in_Salary WHERE EmpNo = :in_EmpNo; INSERT INTO Employee (EmpNo, Salary) VALUES (:in_EmpNo, :in_Salary); .DML LABEL Terminate ; DELETE FROM Employee WHERE EmpNo = :in_EmpNo; INSERT INTO Employee_History (EmpNo, DeptNo) VALUES (:in_EmpNo, :in_DeptNo); .IMPORT INFILE infile1 LAYOUT Employee_Trans APPLY Payroll WHERE in_Transcode = 'PAY' APPLY Terminate WHERE in_Transcode = 'DEL'; .END LOAD; .LOGOFF;
29
Teradata Warehouse Builder
Parallel Load/Unload environment that reduces the cost of building and maintaining your warehouse. Load data from multiple and dissimilar sources simultaneously Read, transform and load a table in a single job Extensible environment allow users to integrate custom transformations through open APIs Jobs restart at the last checkpoint if a DBS reset is encountered Executes multiple instances of operators Eliminates the need of intermediate storage.
30
TWB Infrastructure TWB Teradata Warehouse Builder Processor Script
Teradata Warehouse Builder Operator Interface INMOD Routine OUTMOD Access Modules Inmod Adapter Operator Filter Data Source/ Destination Files, Devices, Databases TWB Script Teradata Database Outmod Adapter Operator Consumer Operator Data Connector Operator Producer Operator
31
Data Modes and Data Formats
Indicators Mode Data Formats FastLoad Format Binary Format Unformatted Text Vartext Data Indic bits Data 2 Bytes Data OA/0D Data OA/0D Data Datadatadatadatadata (Fixed length) Data|data|data|data|data ( Variable length)
32
Questions ?
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.