Professional Judgment in Auditing: Professional Judgment Frameworks as Teaching Tools

John D. Keyser National Director of Assurance Services McGladrey LLP Douglas F. Prawitt Glen Ardis Professor of Accountancy Brigham Young University Jason L. Smith (Moderator) Associate Professor of Accounting University of Nevada, Las Vegas

Agenda Topic Time Introduction to two Frameworks: KPMG, McGladrey 20 minutes What is happening in the Profession 15 minutes Integrating Professional Judgment into the Classroom Q&A 10 minutes

Why focus on Professional Judgment? Increased emphasis on judgment in accounting & auditing Principles-based accounting and auditing standards Movement toward fair value accounting Etc. Increased importance of ability to exercise effective professional judgment, professional skepticism Doug: There are many factors that are leading professional accounting firms to focus on professional judgment. The FASB has moved away from prescriptive accounting rules like lease accounting where there was a clear threshold dividing operating leases from capital leases (90% of cash flows, 75% of economic life). Instead, new accounting standards are much more principles based. Further, accounting has gone away from historical cost and toward fair value as the measurement basis for many line items on the balance sheet. Both the application of principles and the inherent uncertainties involved in estimates require the exercise of professional judgment. John: Just as the accounting standards have become more principles based, the auditing standards similarly require the exercise of professional judgment. Further, auditors face challenges to their judgments not only from litigation, but also from regulatory scrutiny in the form of PCAOB inspections and peer review. Doug: Top professionals are and will be those individuals that consistently and confidently exercise good judgment.

Professional Judgment: What it looks like, and common threats

The KPMG Professional Judgment Framework

The KPMG Professional Judgment Framework Consultation Doug: Briefly introduce KPMG framework, note that frameworks may be simple but they accomplish at least three important things: 1) establish a simple but effective process for making good judgments/decisions; 2) provide a consistent, shared conceptual framework to enhance understanding and communication in firms; and 3) provide a context in which to understand where judgments tend to go wrong so we can be aware of and on guard for traps and biases that can derail an effective judgment process! 6

Steps in the judgment process Doug: Focus in on the steps at center of KPMG framework…noting that “mindset” at center of full diagram includes professional skepticism

Polling Question 1 (CPE Credit) Within an auditing context, what is professional judgment? Professional judgment is the process of using relevant training, knowledge, and experience to reach a decision or draw a conclusion in evaluating evidence, estimating probabilities, or selecting between options. Professional judgment is professional skepticism, which is an attitude that includes a questioning mind and a critical assessment of audit evidence. Professional judgment is the application of one’s experience to make a judgment in the absence of supporting evidence, based on the facts and circumstances of the audit engagement. Professional judgment is the construction of a logical justification to support an outcome or conclusion that is otherwise not supported by the available evidence.

The McGladrey Professional Judgment Framework

McGladrey Professional Judgment Framework John: This slide depicts McGladrey’s judgment framework in a useful diagram. You may notice that we have numbered certain elements of the framework. While I will begin by discussing these number steps, it is important to point out that the judgment process is iterative, and so the sequence of the steps is non-linear.

Steps in McGladrey Judgment Process John: The first step in the process is to define and frame the issue. While it may seem simple to do, the failure to adequately define the issue will short-circuit the judgment process with potentially disastrous results.

Step 1 - Frame the Issue Consider other perspectives and the implications of those views: Regulator, manager, user Error vs. fraud Owner, lender, investor, legal, insurer, customer, audit committee How might a judgment look in the future with the benefit of hindsight, assuming various outcomes? John: When we define an issue, it is very important to consider multiple perspectives. I believe that this can be one of the most challenging aspects of the judgment process. Doug: We often use the illustration of a courtyard to illustrate the concept of framing. Frames are mental structures that we use, usually subconsciously, to simplify, organize, and guide our understanding of a situation. They shape our perspectives and determine the information that we will see as relevant or irrelevant, important or unimportant. They are a necessary aspect of judgment, but our judgment frame provides only one particular perspective, just as looking out one window frame in your home provides one view that might be quite different from the view through another window. We often are not even aware of the view or frame we have adopted and are using or the fact that there are other valid perspectives. Proper use of judgment frames improves the performance is an important aspect of the judgment process and while it is listed in the first step, the overarching depiction of the frames at the top if the framework illustrates that the proper consideration of frames improves our judgment processes throughout all the steps in the good judgment process. Also key in developing sense of professional skepticism in students and professionals! 12

Steps in McGladrey Judgment Process John: The next step in the McGladrey framework is to determine objectives and identify alternatives.

Step 2 - Determine Objectives What is wanted or needed? Ask how you would justify a judgment or decision Are you comfortable at a gut level moving ahead with a judgment process? John: Objectives help determine needed information and the importance of the judgment or decision, and thus the effort that is necessary. Now this sounds simple, but just like with defining and framing the issue that requires the application of professional judgment, this is a major source of inefficient and ineffective decision making. We tend to spend too little time and effort specifying objectives because we are in a hurry to “solve”—we assume we already know what our objectives are. Doug: Determining objectives that are truly fundamental is not as easy as it sounds. A few key points about determining fundamental objectives are: -Invest time to determine fundamental objectives. To identify, you might ask questions like: -How would you justify a judgment or decision? -Are you comfortable at a “gut level” moving ahead with the judgment process? If not, you may not have properly defined the problem or the objectives. -what assumptions (if changed) would have the greatest impact on the judgment? - ask “why” questions John: Test your objectives by asking what is important and why it is important, to get down to the root objectives. For example, we might initially answer a “what” question regarding retirement goals with, “I want to have a certain amount of money in a retirement fund.” That certainly is an objective; however as with many initial objectives it is a means to an end. Following up by asking “why” we want a certain amount of money, can help us uncover the more “root” or fundamental objective, which might be something like “to maintain a high quality of life in retirement.” You’ll note that by framing the objective this way, a number of alternative approaches to accomplishing high quality of life come to mind. Carefully determining underlying objectives is a key step in making important decisions. 14

Step 2 - Identify Alternatives A decision or judgment can only be as good as the best alternative considered Common Pitfalls: Consider only typical or first alternative that comes to mind Do the same thing as prior year Decision “triggers”—an alternative masquerading as a problem or “issue” to be solved John: Step 2 of our framework also involves identification of alternatives. It is important to invest sufficiently in this step in order to identify multiple alternatives. In fact, a common pitfall in the judgment process is when we stop at the first alternative that comes to mind. In the auditing realm, this is often seen in a “same-as-last-year” or SALY approach.

Steps in McGladrey Judgment Process John: As we move to Step 3, I do want to point out that the diagram reflects the expansive nature of steps 2 and 3. In other words, at these points in the judgment process, we want to identify a sufficient number of alternatives and we also want to be sure to gather sufficient evidence regarding those alternatives. As we move to step 4, the diagram reflects the narrowing down of the alternatives as we ultimately select the single best alternative that resolves the issue in light of our objectives.

Steps in McGladrey Judgment Process John: The fourth step in the judgment process is to reach a conclusion.

Steps in McGladrey Judgment Process John: The final step in the judgment process is to document the judgment process and the basis for the conclusion.

Fundamental Elements of Professional Judgment Framework Apply Relevant Knowledge and Learn from Experience Professional knowledge is brought to bear and reflection on experiences allows for enhanced professional judgment going forward Professional Skepticism As depicted the judgment process is performed with a mindset characterized by professional skepticism Potential Frames Judgments are informed through explicit consideration of different perspectives or “frames” John: There’s more to good judgment than just following the five steps. At the base of the diagram, we have placed two key elements of professional judgment – applying relevant knowledge and learn from experience. When making a judgment, it is critical to bring the right knowledge to bear. Sometimes the decision-maker has all of the relevant knowledge, but often it is necessary to gather the relevant knowledge through research and consultation with others. Our framework also recognizes that we should learn from our past decisions, both good and bad. If we reflect upon our judgment process subsequent to a decision, we can identify where we did well and where we did not do so well – so that we can improve for the next judgment. Our framework also emphasizes the application of professional skepticism. We talked earlier about the use of frames or multiple perspectives when making a judgment.

Polling Question 2 (CPE Credit) Which of the following statements about judgment frames is correct? A situation cannot have more than one appropriate frame. There is often no single best frame for a given situation. Frames are not used by risk averse individuals. Professionals should eliminate the use of frames from their judgment processes.

Judgment Biases

Factors Affecting Judgment External factors Time pressure Limited resources Client, regulatory, industry Internal factors Limitations due to judgment “frames” Judgment short-cuts Bias caused by self interest Judgment traps Rush to “solve” John: While the judgment frameworks depict basic models of good judgment, we all recognize that judgments are not made in a vacuum. This slide lists several external elements that can affect the application of our judgment processes. There are elements of our environment that are within our control and some that are not. On the right side of the screen are some of the internal factors that affect our judgments. Some of which are within our control and others are not. Doug: One of the most common judgment traps that we inadvertently fall into is the tendency to want to immediately solve or make a decision. As a result we under-invest in the important early steps in the judgment process and often go with the first workable alternative that is presented or that comes to mind. Sometimes we end up solving the wrong problem, or settle for a suboptimal outcome because we didn’t consider a full set of alternatives. Most of the time we never even realize when this happens because the trap is essentially one of not seeing the issues clearly, so we have a limited view of the issues and the alternatives that are available.

Reality - Descriptive Bounded Rationality Lack important information Time and cost constraints Limited memory capacity Limitations on intelligence and perceptions Doug: Now, while the McGladrey professional judgment framework illustrates what we should do in making a judgment, the reality is that our rationality is bounded, or limited. In reality we might lack important information and we usually face time and cost constraints. We also have limited memory capacity and there are limitations on our intelligence and perceptions.

What we do: Systematic shortcuts System 1 (quick) and Satisfice Heuristics We use simplifying judgment strategies. Rules of thumb developed over time because of our bounded rationality. Example crossing the street in New York… John: Due to these limitations on our judgment, we often tend to use what we refer to as “system 1,” heuristic-based thinking, and we rely on quick, unconscious short-cuts, or rules of thumb, developed over time because of our bounded rationality. These short-cuts can be appropriate sometimes given the time and resource constraints we face. However, when we are not aware of these shortcuts, they can often mess up a good judgment process. Doug: Now, this is not to say that these short cuts necessarily represent bad thinking. Rather, the shortcuts can be seen as simplifying judgment strategies, or rules of thumb, that we have developed over time to help us cope with the complex environment we operate in. Here’s a quick example of a simplifying shortcut. Given our busy schedules, when we are crossing a city street, say in NYC, we first look to the left for on-coming traffic and if the coast is clear, we will take a step out into the street and then look to the right for traffic coming the other way. This is a very efficient and often effective shortcut strategy and it becomes an automatic part of how we cross the street—we don’t even think about it, we just do it. However, if we use this shortcut strategy in London, where they drive on the other side of the street, it could result in us being hit by a car.

Paradox Our minds routinely solve problems too difficult for the mightiest computers, but we also make errors in the simplest of judgments about everyday events Doug: There’s a real paradox here. Our minds are incredibly capable. Despite all the talk, computers are still far from being able to mimic the human mind, and yet we are also capable of some very predictable systematic judgment failures. The good news is that by becoming aware of where we are likely to be susceptible, we can develop logical, intuitive approaches to mitigate these biases.

Judgment Tendencies Overconfidence: Tendency to be overconfident in our judgment abilities Availability: Tendency to judge likelihood of events by how readily available specific examples are in our memory Anchoring: Tendency to insufficiently adjust away from an initial anchor Confirmation: Tendency to seek and overweight confirming evidence Doug:  There are several other judgment heuristics that have the potential to impair our judgment process. Examples include: Availability: Tendency to judge likelihood of events by how readily available specific examples are in our memory Anchoring: Tendency to insufficiently adjust away from an initial anchor Confirmation: Tendency to seek and overweight confirming evidence Overconfidence: Tendency to be overconfident in our judgment abilities There are other biases and pitfalls that we haven’t had time to discuss today, such as the hindsight bias and self-serving explanations, both of which relate to learning from experience. T 26

A Pervasive Judgment Tendency… Overconfidence Bias Most of us are overconfident in our judgment abilities and do not acknowledge the actual level of uncertainty that exists. John: Let’s first talk about a bias that commonly affects professionals’ judgment, that is Overconfidence. This refers to the fact that most of us are overconfident in our estimation abilities and as a result do not acknowledge the actual uncertainty that exists.

Overconfident Experts “Heavier-than-air flying machines are impossible” “They couldn’t hit an elephant at this dist…” Doug: Heavier than air” quote--Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, c. 1895 “Couldn’t hit an elephant” quote--General John B. Sedgwick--Union Army Civil War officer’s last words, uttered during the Battle of Spotsylvania, 1864. Research indicates that most people, including very experienced professionals, are consistently overconfident when attempting to estimate outcomes or likelihoods.

What’s So Bad About Overconfidence? Overconfidence Can Lead To: Taking on too many projects Over promising on deadlines Considering only one alternative Truncating information search or even skipping evidence gathering Snap judgments Avoidance or poor execution of judgment model John: So what’s so bad about overconfidence? It is a good question. Some suggest that a touch of overconfidence may be helpful in not getting bogged down or falling prey to paralysis by analysis. On the other hand, in some cases it can detrimentally affect important judgments.   [Debrief the points on this slide] 29

Confirmation Tendency We tend to have preferences We tend to seek confirming evidence We give confirming evidence greater weight than disconfirming evidence We often cannot know something to be true without checking to see how it might be false Doug: Another example of a bias that can have a tremendous impact on auditors and auditing is the confirmation tendency. The confirmation tendency is powerful and pervasive in human judgment—we tend to pay more attention to and place more weight on information that confirms our beliefs or preferences. Note third bullet point in slide—serious issue for auditors, given our job is to exercise professional skepticism in providing an opinion that is independent of management, and unbiased by our own preferences (e.g., not to find issues because they mean more work).

What is Happening in the Profession with Respect to Enhancement of Professional Judgment?

Goals of McGladrey Professional Judgment Initiative Shared frame of reference, common vocabulary Enhance the ability of audit professionals to consistently exercise quality judgment Enhance on-the-job training/coaching during engagements Build confidence to exercise professional judgment effectively and efficiently John: Many firms have developed their own professional judgment framework. With the help of Doug, Steve Glover, and Jeff Wilks, we developed the McGladrey framework a couple of years ago to help our auditors consistently and confidently exercise good judgment. Our goals in the development of a judgment framework include those listed on this slide. We believe that a common point of reference and a shared vocabulary can help us lift the judgment maturity of our engagement teams. Research shows that specialized training can help even the most experienced people improve the quality and consistency of their judgment.

McGladrey Training Timing Course Name Audience Summer 2012 Introduction to the Framework Partners, Managers, In-charges Fall 2012 Professional Judgment in the Trenches All Spring 2013 Professional Judgment Self-study New hires and those needing refresher Summer 2013 Fraud and the McGladrey PJF Analytical Procedures In-charges Summer 2014 Auditing Accounting Estimates under the McGladrey PJF Partners, managers, in-charges Ongoing Infuse into all courses Various John: We introduced the framework at our large conferences for partners and managers and our experienced in-charge conference. Doug and Steve presented that training. We followed up with a case-based, day long course that was developed by Jeff Wilks and presented at the local office level. Doug and Jeff helped us present the train-the-trainer session. At McGladrey, our professional judgment framework is not a passing fad like the “flavor of the month”. Instead, it is a long-term effort to improve the judgment skills of all of our professionals. As such, we continue to focus on the framework in our training. Last year, we delivered a four-hour course at our large conferences related to the exericse of professional judgment with regard to fraud. For example, we applied the framework to the selection of journal entries for testing. Mark Zimbelman assisted us in the development of that course and we introduced the concept of strategic reasoning. We also worked with Kim Moreno and Sudip Bhattacharjee to develop a course in the performance of analytical procedures which was delivered at the in-charge level.

McGladrey – Other Efforts Consultation forms Practice aids Interoffice inspection John: Some other ways that we have tried to encourage the use of the professional judgment framework are listed here. We incorporated the elements of the framework into our consultations. For example, we require engagement teams to explicitly document the potentially disconfirming evidence that they have identified and how they overcame it. Also, when we identify deficiencies through our internal inspection process, we inquire regarding the application of the professional judgement framework.

CAQ Working Group Professional Judgment Thought Leadership Member firms participate Professional Judgment Thought Leadership Description of a judgment process Discussion of common traps and biases John: The CAQ has been working on their own professional judgment document that should be issued in the near term. For those of you who are not aware, the CAQ is an autonomous, nonpartisan, nonprofit group based in Washington, D.C. It is governed by a Board that comprises leaders from the public company auditing firms, the American Institute of CPAs and three members from outside the public company auditing profession. Representatives of each of the major firms participated in the working group to develop the CAQ’s professional judgment resource. I was McGladrey’s representative on the task force. The judgment resource describes a judgment process and the common traps and biases. I think that the CAQ’s decision to invest in a judgment resource evidences the importance of this topic to the profession.

Polling Question 3 (CPE Credit) The confirmation bias is a subconscious tendency to do which of the following? Seek evidence that confirms a biased judgment Seek evidence that confirms a previously held view Underutilize confirmations in the testing of accounts receivable Seek evidence that disconfirms a previously held view

Integrating Professional Judgment into the Classroom: Can Accounting Students Learn Professional Judgment in the Classroom?

Can You Really Teach Good Judgment? Is it a natural ability or a skill that can be developed? True or False: Either you have it or you do not… Good News! It is a skill that can be learned and developed It helps to know what “good judgment” looks like and the common threats to good judgment The sooner you start learning how to make good professional judgments, the better!

Factors in Developing the Ability to Exercise Good Professional Judgment Do you agree the following factors contribute to the development of good judgment? Natural ability Personal experience Observing others including mentors Self-studies/books AND… Formal training

How does a visual illusion relate to judgment? Source: http://www.michaelbach.de/ot/sze_shepardTables/index.html

GRAPHIC OF GOLF SWING

The KPMG Professional Judgment Framework Consultation 42

Practice makes permanent Phil Mickelson Golf Swing

Threats to good judgment Not following a good process Judgment “traps” and “biases” Mental shortcuts Help simplify a complex situation Can facilitate more efficient judgments USUALLY effective NEW

The purpose and benefits of the topic Mindset, skills, and techniques behind good judgment begin to form at a young age Training, deliberate practice, and experience can improve judgment It is important for students to have a solid foundation in good judgment

How does awareness help with this illusion? Source: http://www.michaelbach.de/ot/mot_mib/index.html

KPMG Professional Judgment Framework Monograph John: With the help of Doug, Steve, and Jeff, we developed a monograph that explains our judgment framework. We distributed this framework to all of our professionals in May 2012. We continue to provide copies of the monograph to new professionals when the begin their career at McGladrey.

McGladrey Monograph John: With the help of Doug, Steve, and Jeff, we developed a monograph that explains our judgment framework. We distributed this framework to all of our professionals in May 2012. We continue to provide copies of the monograph to new professionals when the begin their career at McGladrey.

Polling Question 4 (CPE Credit) You just cannot teach professional judgment; either you have it or you do not. True False

Q&A