INTRODUCTION TO NONSTATISTICAL SAMPLING FOR AUDITORS Jeanne H. Yamamura CPA, MIM, PhD APIPA 2010

SITUATION You are auditing the Dept. of Admissions & Records for Micronesia College. One of your objectives is to verify that student records are being updated correctly and timely. You decide to select a sample of grades posted from the most recent semester completed. APIPA 2010

SITUATION What would you normally document about this sample? Sample size Selection method Population Procedures to be performed Purpose of test What kind of test is this? APIPA 2010

OBJECTIVES Review of sampling concepts Types of sampling - overview Nonstatistical attribute sampling Steps in applying Additional coverage of: Sampling methods Compliance auditing APIPA 2010

Applicable Professional Standards SAS 39 Audit Sampling SAS 111 Amendment to SAS 39 Audit Sampling ISA 530 Audit Sampling APIPA 2010

AUDIT SAMPLING Application of an audit procedure to less than 100% of the items in a population Account balance Class of transactions Examination “on a test basis” Key: Sample is intended to be representative of the population. Objective: To reach a conclusion about the population based on the sample items tested. APIPA 2010

SAMPLING RISK Possibility that the sample is NOT representative of the population As a result, auditor will reach WRONG conclusion Decision errors Type I – Risk of incorrect rejection Type II – Risk of incorrect acceptance APIPA 2010

TYPE I – RISK OF INCORRECT REJECTION Internal control: Risk that sample supports conclusion that control is NOT operating effectively when it really is AKA – Risk of underreliance, risk of assessing control risk too high Substantive testing: Risk that sample supports conclusion that balance is NOT properly stated when it really is APIPA 2010

TYPE II – RISK OF INCORRECT ACCEPTANCE Internal control: Risk that sample supports conclusion that control is operating effectively when it really isn’t AKA – Risk of overreliance, risk of assessing control risk too low Substantive testing: Risk that sample supports conclusion that balance is properly stated when it really isn’t APIPA 2010

WHICH RISK POSES THE GREATER DANGER TO AN AUDITOR? Type I - Risk of incorrect rejection Efficiency Type II - Risk of incorrect acceptance Effectiveness Auditor focus on Type II Also provides coverage for Type I APIPA 2010

NONSAMPLING RISK Risk of auditor error Controlled through Sample wrong population Fail to detect a misstatement when applying audit procedure Misinterpret audit result Controlled through Adequate training Proper planning Effective supervision APIPA 2010

SAMPLE SIZE FACTORS Desired level of assurance (confidence level) Acceptable defect rate (tolerable error) Historical defect rate (expected error) APIPA 2010

CONFIDENCE LEVEL Complement of sampling risk 5% sampling risk, 95% confidence level How much reliance will be placed on test results The greater the reliance and the more severe the consequences of Type II error, the higher the confidence level needed Sample size increases with confidence level (decreases with sampling risk) APIPA 2010

TOLERABLE ERROR AND EXPECTED ERROR “Precision” – the gap between tolerable error and expected error Expected population error rate = 1% Auditor’s tolerable error rate = 3% AKA Allowance for sampling risk Sample size increases as precision decreases APIPA 2010

WHEN DO YOU SAMPLE? Inspection of tangible assets, e.g., inventory observation Inspection of records or documents, e.g., internal control testing Reperformance, e.g., internal control testing Confirmation, e.g., verification of AR balances APIPA 2010

WHEN IS SAMPLING INAPPROPRIATE? Selection of all items with a particular characteristic, e.g., all disbursements > $100,000 Testing only one or a few items, e.g., automated IT controls, walk throughs Analytical procedures Scanning Inquiry Observation APIPA 2010

WALKTHROUGHS Designed to provide evidence regarding the design and implementation of controls Can provide some assurance of operating effectiveness BUT Depends on nature of control (automated or manual) Depends on nature of auditor’s procedures to test control (also includes inquiry and observation combined with strong control environment and adequate monitoring) Walkthough = sample of 1 APIPA 2010

STATISTICAL VS NONSTATISTICAL SAMPLING Statistical computation of sample size Statistical evaluation of results Nonstatistical sampling Sample sizes should be approximately the same (AU 350.22) Sample sizes must be sufficient to support reliance on controls and assertions being tested APIPA 2010

WHEN IS SAMPLING NONSTATISTICAL? If sample size determined judgmentally If sample selected haphazardly If sample results evaluated judgmentally APIPA 2010

TYPES OF SAMPLING Attribute sampling Monetary unit sampling Classical variables sampling APIPA 2010

ATTRIBUTE SAMPLING Used to estimate proportion of a population that possesses a specific characteristic Most commonly used for T of C Can also be used for dual purpose testing (T of C and Substantive T of T) APIPA 2010

MONETARY-UNIT SAMPLING AKA probability proportional to size (PPS) sampling, cumulative monetary unit sampling Used to estimate dollar amount of misstatement APIPA 2010

CLASSICAL VARIABLES SAMPLING Uses normal distribution theory to identify amount of misstatement Useful when large number of differences expected Smaller sample size than MUS Effective for both overstatements and understatements Can easily incorporate zero balances APIPA 2010

STEPS IN NONSTATISTICAL ATTRIBUTE SAMPLING APPLICATION Planning Determine the test objectives Define the population characteristics Determine the sample size Performance Select sample items Perform the auditing procedures Evaluation Calculate the results Draw conclusions APIPA 2010

STEP 1: DETERMINE THE TEST OBJECTIVES Objective for T of C: To determine the operating effectiveness of the internal control Support control risk assessment below maximum (FS audit) Identify controls to be tested and understand why they are to be tested APIPA 2010

TESTS OF CONTROLS Concerned primarily with Were the necessary controls performed? How were they performed? By whom were they performed? Appropriate when documentary evidence of performance exists APIPA 2010

SUBSTANTIVE TEST OF TRANSACTIONS Objective for S T of T: To determine whether the transactions contain monetary misstatements Alternatively, to determine whether the system is operating as designed Identify transactions to be tested and understand why they are to be tested APIPA 2010

STEP 2: DEFINE THE POPULATION CHARACTERISTICS Define the sampling population Can be defined however desired BUT must include entire population as defined Test population for completeness Define the sampling unit Determined by available records Based on definition of population and audit objective Define the control deviation conditions APIPA 2010

STEP 3: DETERMINE THE SAMPLE SIZE Consider desired confidence level, tolerable deviation rate, and expected population deviation rate Judgmentally determine sample size NOTE: Check against statistical sample size tables to verify adequacy APIPA 2010

TOLERABLE RATE GUIDELINES Significance of the transactions and related account balances that the IC are intended to affect Highly significant balances Tolerable Rate of 4% Significant balances Tolerable Rate of 5% Less significant balances Tolerable Rate of 6% Preliminary Assessment of CR Tolerable Rate Low < = 5% Moderate < = 10% High Do not test controls APIPA 2010

TOLERABLE RATE GUIDELINES Assessed importance of the control Tolerable Rate Highly important 3 - 5% Moderately important 6 – 10% APIPA 2010

ESTIMATE OF POPULATION ERROR RATE Prior year results Preliminary sample Should be low – 0, 1% Higher rates increase sample size APIPA 2010

STEP 3: DETERMINE THE SAMPLE SIZE Guidelines for nonstatistical sample sizes for tests of controls If any errors found, increase sample size or increase control risk (Probably not applicable to Public Auditor) Desired level of controls reliance (how important is the control/process) Sample size Low 15-20 Moderate 25-35 High 40-60 APIPA 2010

SMALL POPULATIONS AND INFREQUENTLY OPERATING CONTROLS Small Population Sample Size Table Control Frequency and Population Size Sample Size Quarterly (4) 2 Monthly (12) 2-4 Semimonthly (24) 3-8 Weekly (52) 5-9 APIPA 2010

STEP 4: SELECT SAMPLE ITEMS Random sample Systematic sample (with random start) Haphazard selection APIPA 2010

RANDOM SELECTION Every possible combination of population items has an equal chance of being included in the sample Random number tables Computer generation of random numbers APIPA 2010

SYSTEMATIC SELECTION Interval calculated and items selected based on size of interval Interval = Population / Desired Sample Size Starting point is random number within interval Need to consider if bias present due to patterns in data APIPA 2010

HAPHAZARD SELECTION Selection by auditor without any conscious bias If you select large, risky, or unusual items, it is NOT haphazard selection and it is NOT audit sampling. Instead – targeted or directed selection Still desire representative sample Avoid unusual, large, first or last Useful for certain situations Example: Tracing credits from AR to CR/other sources looking for fictitious credits Less costly and simpler APIPA 2010

STEP 5: PERFORM THE AUDITING PROCEDURES Conduct planned audit procedures What if? Voided documents - if properly voided, not a deviation; replace with new sample item Unused or inapplicable documents – replace with new sample item Inability to examine sample item – deviation Stopping test before completion – large number of deviations detected APIPA 2010

STEP 5: PERFORM THE AUDITING PROCEDURES Deviations observed Investigate nature, cause, and consequence of every exception Unintentional error? Or fraud? Monetary misstatement resulted? Cause – misunderstanding of instructions? Carelessness? Effect on other areas? APIPA 2010

STEP 6: CALCULATE THE RESULTS No computed upper deviation rate (per table in statistical sampling) Compute Calculated Sampling Error = Tolerable Error Rate – Sample Error Rate. APIPA 2010

STEP 7: DRAW CONCLUSIONS Control not effective (system not working as designed) if Calculated Sampling Error too small Depends on sample size used Sample Error Rate > Tolerable Error Rate Sample Error Rate > Expected Population Error Rate APIPA 2010

COMPLIANCE AUDITING Performance of auditing procedures to determine whether an entity is complying with specific requirements of laws, regulations, or agreements Governmental entities and other recipients of governmental financial assistance Compliance with laws and regulations that materially affect each major federal assistance program APIPA 2010

COMPLIANCE AUDITING OF FEDERAL ASSISTANCE PROGRAMS Definition of population for testing of an internal control procedure that applies to more than one program Define items from each major program as a separate population, OR Define all items to which control is applicable as a single population Second choice usually more efficient APIPA 2010

COMPLIANCE AUDITING - EXAMPLE Federal financial assistance for Island City Three major federal financial assistance programs Four nonmajor programs Control: Transaction review to ensure that only legally allowable costs are charged to each program APIPA 2010

COMPLIANCE AUDITING - EXAMPLE More efficient to select one sample from population of all transactions (major and nonmajor programs) Confidence level = 95% Tolerable deviation rate = 9% Expected population deviation rate = 1% Sample size: 51 1 allowable deviation APIPA 2010

T of C versus S T of T Test of Control Verifies that a control is operating effectively Substantive Test of Transactions Verified that a transaction does not contain a misstatement APIPA 2010

ASSERTIONS FOR CLASSES OF TRANSACTIONS Occurrence: Transaction actually occurred and pertains to the entity (existence/validity) Completeness: All transactions have been recorded Accuracy: Amounts and other data have been recorded correctly APIPA 2010

ASSERTIONS FOR CLASSES OF TRANSACTIONS Cutoff: Transactions have been recorded in the correct accounting period Classification: Transactions have been recorded in the proper accounts APIPA 2010

CALCULATED SAMPLING ERROR Tolerable error rate – Sample error rate = Calculated sampling error Sample error rate = Population error rate due to sampling error Auditor must evaluate calculated sampling error to see if it is big enough (sufficiently large to allow for sampling error in population) APIPA 2010

CALCULATED SAMPLING ERROR If Sample error rate > Tolerable error rate = REJECT – CONTROL NOT WORKING or PROCEDURE NOT BEING FOLLOWED If Sample error rate > Expected population error rate, REJECT – CONTROL NOT WORKING OR PROCEDURE NOT BEING FOLLOWED APIPA 2010

STEPS IN NONSTATISTICAL SUBSTANTIVE SAMPLING APPLICATION Planning Determine the test objectives Define the population characteristics Determine the sample size Performance Select sample items Perform the auditing procedures Evaluation Calculate the results Draw conclusions APIPA 2010

STEP 2: DEFINE THE POPULATION CHARACTERISTICS Identify individually significant items Some items too risky, must be audited, OR Easier to pull out and test large items Stratify population Divide population into homogeneous units For example, all items > $10,000 Items tested 100% are not part of the sample APIPA 2010

STEP 2: DEFINE THE POPULATION CHARACTERISTICS Define the sampling population Consists of an account balance or class of transactions Will project sample results to population Must be sure to adequately identify population For example: Accounts Receivable could be defined as All accounts Accounts with zero balances Accounts with debit balances Accounts with credit balances APIPA 2010

STEP 2: DEFINE THE POPULATION CHARACTERISTICS Define the sampling unit Any item in the defined population Could be an account or a transaction APIPA 2010

STEP 3: DETERMINE THE SAMPLE SIZE Subjective determination OK Factors to consider: Amounts of individual items Accounting populations usually include a few very large items, a number of moderately large amounts, and a large number of small amounts If not stratified, will need larger sample Variability and size of population The greater the variability, the larger the sample size needed Population size – little effect on sample size so usually ignored APIPA 2010

STEP 3: DETERMINE THE SAMPLE SIZE Factors to consider: Risk of incorrect acceptance (RIA) As RIA increased, sample size decreases If controls good, can accept larger RIA for substantive testing Tolerable misstatement and expected misstatement Larger tolerable misstatement, smaller sample size Larger expected misstatement, larger sample size APIPA 2010

STEP 4: SELECT SAMPLE ITEMS Any method that will result in representative sample Random sample Systematic sample (with random start) Haphazard selection APIPA 2010

STEP 5: PERFORM THE AUDITING PROCEDURES Deviations observed Investigate nature, cause, and consequence of every exception Unintentional error? Or fraud? Monetary misstatement resulted? Cause – misunderstanding? Carelessness? Effect on other areas? APIPA 2010

STEP 6: CALCULATE THE RESULTS Compute sample error amount or sample error rate Project to population Projected misstatement Error * number of items in population Error rate * dollar population value APIPA 2010

STEP 7: DRAW CONCLUSIONS Compare projected misstatement to tolerable misstatement If projected misstatement < tolerable misstatement, population OK If projected misstatement > tolerable misstatement, population misstated APIPA 2010

STEP 7: DRAW CONCLUSIONS Consider sampling risk If projected misstatement < expected misstatement, probably safe to conclude that population is OK (i.e., there is an acceptably LOW risk that the true misstatement exceeds the tolerable misstatement) If projected misstatement > expected misstatement, greater risk present (i.e., there is an UNACCEPTABLY HIGH risk that the true misstatement exceeds the tolerable misstatement). APIPA 2010

STEP 7: DRAW CONCLUSIONS If recorded amount believed to be misstated, need more work! Investigate misstatements Adjust recorded amounts APIPA 2010

RESOURCES Audit Sampling: An Introduction, 3rd Edition, Guy, Carmichael & Whittington Audit Guide: Audit Sampling, New Edition as of May 1, 2008, AICPA Auditing & Assurance Services, 6th Edition, Messier, Glover, & Prawitt Auditing & Assurance Services, 12th Edition, Arens, Elder & Beasley APIPA 2010

QUESTIONS? APIPA 2010