1. Incremental Load
using qvd files

2. Incremental Load
Incremental Load
Differential Load
Delta Load

3. Incremental Load
Load only the new or changed records from the database. The rest should already be available, one way or another.

4. Comments on Buffer Load
Buffer (Incremental) Load … is a solution only for Log files, but not for DBs.
Buffer (Stale after 7 days) Select … is not a solution. It needs a full Load after 7 days.

5. Incremental Load
Load new data from Database table (slow, but few records)
Load old data from QVD file (many records, but fast)
Create new QVD file
Procedure must be repeated for each table

6. Different DB-changes If source allows … Append only. (Logfiles)
Insert only. (No Update or Delete)
Insert and Update. (No Delete)
Insert, Update and Delete.

7. 1) Append only Must be Log file
Loads records added in the end of the file

8. 1) Append only Buffer (Incremental) Load * From LogFile.txt
(ansi, txt, delimiter is '\t', embedded labels);

9. 2) Insert only Can be any DB Loads INSERTed records
Needs the field ModificationDate

10. 2) Insert only QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

11. 2) Insert only QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;

12. 2) Insert only QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;
STORE QV_Table INTO File.QVD;

13. 2) Insert only QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#
AND ModificationTime < #$(BeginningThisExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;
STORE QV_Table INTO File.QVD;

14. 3) Insert and Update Can be any DB Loads INSERTed and UPDATEd records
Needs the fields ModificationDate and PrimaryKey

15. 3) Insert and Update QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT Exists(PrimaryKey);
STORE QV_Table INTO File.QVD;

16. 4) Insert, Update and Delete
Can be any DB
Loads INSERTed and UPDATEd records
Removes DELETEd records
Needs the fields ModificationDate and PrimaryKey
Tricky to implement

17. 4) Insert, Update and Delete
QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT EXISTS(PrimaryKey);
Inner Join
SQL SELECT PrimaryKey FROM DB_TABLE;
STORE QV_Table INTO File.QVD;

18. 4) Insert, Update and Delete
ListOfDeletedEntries:
SQL SELECT PrimaryKey AS Deleted FROM DB_TABLE
WHERE DeletionFlag = 1;
QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;
Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT Exists(PrimaryKey)
AND NOT Exists(Deleted,PrimaryKey);
Drop Table ListOfDeletedEntries;
STORE QV_Table INTO File.QVD;

19. LastExecutionTime & Error handling
Let ThisExecTime = ReloadTime();
{ Load sequence }
If ScriptErrorCount = 0 then
Let LastExecTime = ThisExecTime;
End If

20. Final Script Let ThisExecTime = ReloadTime(); QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#
AND ModificationTime < #$(ThisExecTime)#;
Concatenate LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT EXISTS(PrimaryKey);
Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;
If ScriptErrorCount = 0 then
STORE QV_Table INTO File.QVD;
Let LastExecTime = ThisExecTime;
End If

21. Summary 1 Incremental Load possible for… Append only. (Logfiles)
Insert only. (No Update or Delete)
Insert and Update. (No Delete)
Insert, Update and Delete.

22. Summary 2
„Incremental Load“ normally not equivalent to Buffer (Incremental) Load …