Benchmarks Title: A Measure of Transaction Processing Power Authors: Anon Et. Al. Datamation, 1985

A Measure of Transaction Processing Power Problem –Problem Statement –Why is this problem important? –Why is this problem hard? Approaches –Approach description, key concepts –Contributions (novelty, improved) –Assumptions

Problem Statement – Benchmark Given –Transaction processing systems –Application Workloads Find –Measures of performance of transaction processing systems Objectives –Compare TP systems Constraints –Workload is not uniform –Economics is relevant

Why is this problem important? –Applications: Transaction Processing – online, batch Airline reservation system Banking ATM E-commerce Why is this problem hard? –Transaction performance depends on many components CPUs, I/O architecture, DBMS software –Limitations of tradition performance measures Focus on single components, e.g. MIPS, MFlops is for CPU Non-standard, e.g. TPS – no common transaction definition

Contributions Benchmark Workloads –Database – For a bank Branches – (1000, 0.1 MB) Tellers – (10K, 1 MB) Accounts – (10 Million, 1 GB) History – (90 days, 10GB) –Queries DebitCredit – test of transaction system –Code on p. 8 Sort, Copy – test of file system and I/O –Data generation methods

Contributions Metrics –Economic 5 year capital cost –Performance Elapsed time, a.k.a response time Throughput = work per unit time TPS (Transactions Per Second): –Peak DebitCredit transactions per second –with Pr. [ response time < 1 second ] = 95% –Combined Cost per TPS

Proposed Approaches, Key Concepts Reference: A Guttman ‘R-tree a dynamic index structure for spatial searching’, 1984 SortA disc sort of one million records. The source and target files are sequential. Relevant measures: elapsed time & cost ScanA mini-batch Cobol transaction sequentially scans and updates one thousands records. Relevant measures: elapsed time & cost DebitCreditA banking transaction interacts with a block-mode terminal connected via x.25. The system does presentation services to map the input for a Cobol program which in turn uses a database system to debit a bank account, do the standard double-entry bookkeeping and then reply to the terminal. 95% of the transactions must provide one-second response time. Relevant measures are throughput and cost.

Proposed Approaches, Key Concepts Reference: A Guttman ‘R-tree a dynamic index structure for spatial searching’, 1984 Metric Benchmark Elapsed TimeCost SortThe time from the start to the end of the sort program Time weighted cost of the sort S/W, H/W packages it uses. ScanThe average time between successive BeginTransaction steps. Time-weighted system cost of Scan

Validation Data from many real systems (table on p. 9) –Ranking by TPS is similar to ranking by IO and K-instructions. Expertise –A large group of experienced researchers Adoption by community –Self-reporting by vendors on tpc.org –DebitCredit  TPC-A and TPC-C –Sort 1M records  PennySort, MinuteSort, Terabyte Sort

Discussion, Self-Critique Authors identify many limitations (page 10) –Does not capture the performance bugs. History file is a hotspot – all transactions write it! log grows rapidly –Ignores the performance of some operations system startup and transaction startup. –Cost model ignored many aspects communication, terminal, development, maintenance, and outages costs.

Assumptions DebitCredit benchmark reflects workload across applications Vendor benchmark results predict Customer experience –Perfect tuning by vendors –Constant change and growth at customer site Customer centric rather than designer centric

Rewrite today Update with current bencharks –DebitCredit  TPC-A and TPC-C –Sort 1M records  PennySort, MinuteSort, Terabyte Sort Workloads beyond transaction processing –Decision Support Systems, Web-based systems Consider Alternatives –Trace driven workload characterization and simulation –Analytical model Allow quicker comparison of design choices Queueing model (Fig. 1., p. 7, Thomasian,1998) A. Thomasian, Concurrency control: methods, performance, and analysis, ACM Computing Surveys, March 1998.A. Thomasian, Concurrency control: methods, performance, and analysis, ACM Computing Surveys, March 1998.