0. Professional Judgment in Auditing: Professional Judgment Frameworks as Teaching Tools

1. John D. Keyser National Director of Assurance Services McGladrey LLP
Douglas F. Prawitt
Glen Ardis Professor of Accountancy
Brigham Young University
Jason L. Smith (Moderator)
Associate Professor of Accounting
University of Nevada, Las Vegas

2. Agenda Topic Time Introduction to two Frameworks: KPMG, McGladrey
20 minutes
What is happening in the Profession
15 minutes
Integrating Professional Judgment into the Classroom
Q&A
10 minutes

3. Why focus on Professional Judgment?
Increased emphasis on judgment in accounting & auditing
Principles-based accounting and auditing standards
Movement toward fair value accounting
Etc.
Increased importance of ability to exercise effective professional judgment, professional skepticism
Doug: There are many factors that are leading professional accounting firms to focus on professional judgment. The FASB has moved away from prescriptive accounting rules like lease accounting where there was a clear threshold dividing operating leases from capital leases (90% of cash flows, 75% of economic life). Instead, new accounting standards are much more principles based. Further, accounting has gone away from historical cost and toward fair value as the measurement basis for many line items on the balance sheet. Both the application of principles and the inherent uncertainties involved in estimates require the exercise of professional judgment.
John: Just as the accounting standards have become more principles based, the auditing standards similarly require the exercise of professional judgment. Further, auditors face challenges to their judgments not only from litigation, but also from regulatory scrutiny in the form of PCAOB inspections and peer review.
Doug: Top professionals are and will be those individuals that consistently and confidently exercise good judgment.

4. Professional Judgment: What it looks like, and common threats

5. The KPMG Professional Judgment Framework

6. The KPMG Professional Judgment Framework
Consultation
Doug: Briefly introduce KPMG framework, note that frameworks may be simple but they accomplish at least three important things: 1) establish a simple but effective process for making good judgments/decisions; 2) provide a consistent, shared conceptual framework to enhance understanding and communication in firms; and 3) provide a context in which to understand where judgments tend to go wrong so we can be aware of and on guard for traps and biases that can derail an effective judgment process! 6

7. Steps in the judgment process
Doug: Focus in on the steps at center of KPMG framework…noting that “mindset” at center of full diagram includes professional skepticism

8. Polling Question 1 (CPE Credit)
Within an auditing context, what is professional judgment?
Professional judgment is the process of using relevant training, knowledge, and experience to reach a decision or draw a conclusion in evaluating evidence, estimating probabilities, or selecting between options.
Professional judgment is professional skepticism, which is an attitude that includes a questioning mind and a critical assessment of audit evidence.
Professional judgment is the application of one’s experience to make a judgment in the absence of supporting evidence, based on the facts and circumstances of the audit engagement.
Professional judgment is the construction of a logical justification to support an outcome or conclusion that is otherwise not supported by the available evidence.

9. The McGladrey Professional Judgment Framework

10. McGladrey Professional Judgment Framework
John: This slide depicts McGladrey’s judgment framework in a useful diagram. You may notice that we have numbered certain elements of the framework. While I will begin by discussing these number steps, it is important to point out that the judgment process is iterative, and so the sequence of the steps is non-linear.

11. Steps in McGladrey Judgment Process
John: The first step in the process is to define and frame the issue. While it may seem simple to do, the failure to adequately define the issue will short-circuit the judgment process with potentially disastrous results.

12. Step 1 - Frame the Issue
Consider other perspectives and the implications of those views:
Regulator, manager, user
Error vs. fraud
Owner, lender, investor, legal, insurer, customer, audit committee
How might a judgment look in the future with the benefit of hindsight, assuming various outcomes?
John: When we define an issue, it is very important to consider multiple perspectives. I believe that this can be one of the most challenging aspects of the judgment process.
Doug: We often use the illustration of a courtyard to illustrate the concept of framing. Frames are mental structures that we use, usually subconsciously, to simplify, organize, and guide our understanding of a situation. They shape our perspectives and determine the information that we will see as relevant or irrelevant, important or unimportant. They are a necessary aspect of judgment, but our judgment frame provides only one particular perspective, just as looking out one window frame in your home provides one view that might be quite different from the view through another window. We often are not even aware of the view or frame we have adopted and are using or the fact that there are other valid perspectives. Proper use of judgment frames improves the performance is an important aspect of the judgment process and while it is listed in the first step, the overarching depiction of the frames at the top if the framework illustrates that the proper consideration of frames improves our judgment processes throughout all the steps in the good judgment process. Also key in developing sense of professional skepticism in students and professionals! 12

13. Steps in McGladrey Judgment Process
John: The next step in the McGladrey framework is to determine objectives and identify alternatives.

14. Step 2 - Determine Objectives
What is wanted or needed?
Ask how you would justify a judgment or decision
Are you comfortable at a gut level moving ahead with a judgment process?
John: Objectives help determine needed information and the importance of the judgment or decision, and thus the effort that is necessary. Now this sounds simple, but just like with defining and framing the issue that requires the application of professional judgment, this is a major source of inefficient and ineffective decision making. We tend to spend too little time and effort specifying objectives because we are in a hurry to “solve”—we assume we already know what our objectives are.
Doug: Determining objectives that are truly fundamental is not as easy as it sounds. A few key points about determining fundamental objectives are:
-Invest time to determine fundamental objectives. To identify, you might ask questions like:
-How would you justify a judgment or decision?
-Are you comfortable at a “gut level” moving ahead with the judgment process? If not, you may not have properly defined the problem or the objectives.
-what assumptions (if changed) would have the greatest impact on the judgment?
- ask “why” questions
John: Test your objectives by asking what is important and why it is important, to get down to the root objectives. For example, we might initially answer a “what” question regarding retirement goals with, “I want to have a certain amount of money in a retirement fund.” That certainly is an objective; however as with many initial objectives it is a means to an end. Following up by asking “why” we want a certain amount of money, can help us uncover the more “root” or fundamental objective, which might be something like “to maintain a high quality of life in retirement.” You’ll note that by framing the objective this way, a number of alternative approaches to accomplishing high quality of life come to mind. Carefully determining underlying objectives is a key step in making important decisions. 14

15. Step 2 - Identify Alternatives
A decision or judgment can only be as good as the best alternative considered
Common Pitfalls:
Consider only typical or first alternative that comes to mind
Do the same thing as prior year
Decision “triggers”—an alternative masquerading as a problem or “issue” to be solved
John: Step 2 of our framework also involves identification of alternatives. It is important to invest sufficiently in this step in order to identify multiple alternatives. In fact, a common pitfall in the judgment process is when we stop at the first alternative that comes to mind. In the auditing realm, this is often seen in a “same-as-last-year” or SALY approach.

16. Steps in McGladrey Judgment Process
John: As we move to Step 3, I do want to point out that the diagram reflects the expansive nature of steps 2 and 3. In other words, at these points in the judgment process, we want to identify a sufficient number of alternatives and we also want to be sure to gather sufficient evidence regarding those alternatives. As we move to step 4, the diagram reflects the narrowing down of the alternatives as we ultimately select the single best alternative that resolves the issue in light of our objectives.

17. Steps in McGladrey Judgment Process
John: The fourth step in the judgment process is to reach a conclusion.

18. Steps in McGladrey Judgment Process
John: The final step in the judgment process is to document the judgment process and the basis for the conclusion.

19. Fundamental Elements of Professional Judgment Framework
Apply Relevant Knowledge and Learn from Experience
Professional knowledge is brought to bear and reflection on experiences allows for enhanced professional judgment going forward
Professional Skepticism
As depicted the judgment process is performed with a mindset characterized by professional skepticism
Potential Frames
Judgments are informed through explicit consideration of different perspectives or “frames”
John: There’s more to good judgment than just following the five steps. At the base of the diagram, we have placed two key elements of professional judgment – applying relevant knowledge and learn from experience. When making a judgment, it is critical to bring the right knowledge to bear. Sometimes the decision-maker has all of the relevant knowledge, but often it is necessary to gather the relevant knowledge through research and consultation with others.
Our framework also recognizes that we should learn from our past decisions, both good and bad. If we reflect upon our judgment process subsequent to a decision, we can identify where we did well and where we did not do so well – so that we can improve for the next judgment.
Our framework also emphasizes the application of professional skepticism. We talked earlier about the use of frames or multiple perspectives when making a judgment.

20. Polling Question 2 (CPE Credit)
Which of the following statements about judgment frames is correct?
A situation cannot have more than one appropriate frame.
There is often no single best frame for a given situation.
Frames are not used by risk averse individuals.
Professionals should eliminate the use of frames from their judgment processes.

21. Judgment Biases

22. Factors Affecting Judgment
External factors
Time pressure
Limited resources
Client, regulatory, industry
Internal factors
Limitations due to judgment “frames”
Judgment short-cuts
Bias caused by self interest
Judgment traps
Rush to “solve”
John: While the judgment frameworks depict basic models of good judgment, we all recognize that judgments are not made in a vacuum. This slide lists several external elements that can affect the application of our judgment processes. There are elements of our environment that are within our control and some that are not.
On the right side of the screen are some of the internal factors that affect our judgments. Some of which are within our control and others are not.
Doug: One of the most common judgment traps that we inadvertently fall into is the tendency to want to immediately solve or make a decision. As a result we under-invest in the important early steps in the judgment process and often go with the first workable alternative that is presented or that comes to mind. Sometimes we end up solving the wrong problem, or settle for a suboptimal outcome because we didn’t consider a full set of alternatives. Most of the time we never even realize when this happens because the trap is essentially one of not seeing the issues clearly, so we have a limited view of the issues and the alternatives that are available.

23. Reality - Descriptive Bounded Rationality Lack important information
Time and cost constraints
Limited memory capacity
Limitations on intelligence and perceptions
Doug: Now, while the McGladrey professional judgment framework illustrates what we should do in making a judgment, the reality is that our rationality is bounded, or limited. In reality we might lack important information and we usually face time and cost constraints. We also have limited memory capacity and there are limitations on our intelligence and perceptions.

24. What we do: Systematic shortcuts
System 1 (quick) and Satisfice
Heuristics
We use simplifying judgment strategies. Rules of thumb developed over time because of our bounded rationality.
Example
crossing the street in New York…
John: Due to these limitations on our judgment, we often tend to use what we refer to as “system 1,” heuristic-based thinking, and we rely on quick, unconscious short-cuts, or rules of thumb, developed over time because of our bounded rationality. These short-cuts can be appropriate sometimes given the time and resource constraints we face. However, when we are not aware of these shortcuts, they can often mess up a good judgment process.
Doug: Now, this is not to say that these short cuts necessarily represent bad thinking. Rather, the shortcuts can be seen as simplifying judgment strategies, or rules of thumb, that we have developed over time to help us cope with the complex environment we operate in. Here’s a quick example of a simplifying shortcut. Given our busy schedules, when we are crossing a city street, say in NYC, we first look to the left for on-coming traffic and if the coast is clear, we will take a step out into the street and then look to the right for traffic coming the other way. This is a very efficient and often effective shortcut strategy and it becomes an automatic part of how we cross the street—we don’t even think about it, we just do it. However, if we use this shortcut strategy in London, where they drive on the other side of the street, it could result in us being hit by a car.

25. Paradox
Our minds routinely solve problems too difficult for the mightiest computers, but we also make errors in the simplest of judgments about everyday events
Doug: There’s a real paradox here. Our minds are incredibly capable. Despite all the talk, computers are still far from being able to mimic the human mind, and yet we are also capable of some very predictable systematic judgment failures. The good news is that by becoming aware of where we are likely to be susceptible, we can develop logical, intuitive approaches to mitigate these biases.

26. Judgment Tendencies
Overconfidence: Tendency to be overconfident in our judgment abilities
Availability: Tendency to judge likelihood of events by how readily available specific examples are in our memory
Anchoring: Tendency to insufficiently adjust away from an initial anchor
Confirmation: Tendency to seek and overweight confirming evidence
Doug:  There are several other judgment heuristics that have the potential to impair our judgment process. Examples include:
Availability: Tendency to judge likelihood of events by how readily available specific examples are in our memory
Anchoring: Tendency to insufficiently adjust away from an initial anchor
Confirmation: Tendency to seek and overweight confirming evidence
Overconfidence: Tendency to be overconfident in our judgment abilities
There are other biases and pitfalls that we haven’t had time to discuss today, such as the hindsight bias and self-serving explanations, both of which relate to learning from experience. T 26

27. A Pervasive Judgment Tendency…
Overconfidence Bias
Most of us are overconfident in our judgment abilities and do not acknowledge the actual level of uncertainty that exists.
John: Let’s first talk about a bias that commonly affects professionals’ judgment, that is Overconfidence. This refers to the fact that most of us are overconfident in our estimation abilities and as a result do not acknowledge the actual uncertainty that exists.

28. Overconfident Experts
“Heavier-than-air flying machines are impossible”
“They couldn’t hit an elephant at this dist…”
Doug: Heavier than air” quote--Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, c “Couldn’t hit an elephant” quote--General John B. Sedgwick--Union Army Civil War officer’s last words, uttered during the Battle of Spotsylvania, 1864.
Research indicates that most people, including very experienced professionals, are consistently overconfident when attempting to estimate outcomes or likelihoods.

29. What’s So Bad About Overconfidence?
Overconfidence Can Lead To:
Taking on too many projects
Over promising on deadlines
Considering only one alternative
Truncating information search or even skipping evidence gathering
Snap judgments
Avoidance or poor execution of judgment model
John: So what’s so bad about overconfidence? It is a good question. Some suggest that a touch of overconfidence may be helpful in not getting bogged down or falling prey to paralysis by analysis. On the other hand, in some cases it can detrimentally affect important judgments.
[Debrief the points on this slide] 29

30. Confirmation Tendency
We tend to have preferences
We tend to seek confirming evidence
We give confirming evidence greater weight than disconfirming evidence
We often cannot know something to be true without checking to see how it might be false
Doug: Another example of a bias that can have a tremendous impact on auditors and auditing is the confirmation tendency. The confirmation tendency is powerful and pervasive in human judgment—we tend to pay more attention to and place more weight on information that confirms our beliefs or preferences. Note third bullet point in slide—serious issue for auditors, given our job is to exercise professional skepticism in providing an opinion that is independent of management, and unbiased by our own preferences (e.g., not to find issues because they mean more work).

31. What is Happening in the Profession with Respect to Enhancement of Professional Judgment?

32. Goals of McGladrey Professional Judgment Initiative
Shared frame of reference, common vocabulary
Enhance the ability of audit professionals to consistently exercise quality judgment
Enhance on-the-job training/coaching during engagements
Build confidence to exercise professional judgment effectively and efficiently
John: Many firms have developed their own professional judgment framework. With the help of Doug, Steve Glover, and Jeff Wilks, we developed the McGladrey framework a couple of years ago to help our auditors consistently and confidently exercise good judgment. Our goals in the development of a judgment framework include those listed on this slide. We believe that a common point of reference and a shared vocabulary can help us lift the judgment maturity of our engagement teams. Research shows that specialized training can help even the most experienced people improve the quality and consistency of their judgment.

33. McGladrey Training Timing Course Name Audience Summer 2012
Introduction to the Framework
Partners, Managers, In-charges
Fall 2012
Professional Judgment in the Trenches
All
Spring 2013
Professional Judgment Self-study
New hires and those needing refresher
Summer 2013
Fraud and the McGladrey PJF
Analytical Procedures
In-charges
Summer 2014
Auditing Accounting Estimates under the McGladrey PJF
Partners, managers, in-charges
Ongoing
Infuse into all courses
Various
John: We introduced the framework at our large conferences for partners and managers and our experienced in-charge conference. Doug and Steve presented that training. We followed up with a case-based, day long course that was developed by Jeff Wilks and presented at the local office level. Doug and Jeff helped us present the train-the-trainer session.
At McGladrey, our professional judgment framework is not a passing fad like the “flavor of the month”. Instead, it is a long-term effort to improve the judgment skills of all of our professionals. As such, we continue to focus on the framework in our training. Last year, we delivered a four-hour course at our large conferences related to the exericse of professional judgment with regard to fraud. For example, we applied the framework to the selection of journal entries for testing. Mark Zimbelman assisted us in the development of that course and we introduced the concept of strategic reasoning.
We also worked with Kim Moreno and Sudip Bhattacharjee to develop a course in the performance of analytical procedures which was delivered at the in-charge level.

34. McGladrey – Other Efforts
Consultation forms
Practice aids
Interoffice inspection
John: Some other ways that we have tried to encourage the use of the professional judgment framework are listed here. We incorporated the elements of the framework into our consultations. For example, we require engagement teams to explicitly document the potentially disconfirming evidence that they have identified and how they overcame it.
Also, when we identify deficiencies through our internal inspection process, we inquire regarding the application of the professional judgement framework.

35. CAQ Working Group Professional Judgment Thought Leadership
Member firms participate
Professional Judgment Thought Leadership
Description of a judgment process
Discussion of common traps and biases
John: The CAQ has been working on their own professional judgment document that should be issued in the near term. For those of you who are not aware, the CAQ is an autonomous, nonpartisan, nonprofit group based in Washington, D.C. It is governed by a Board that comprises leaders from the public company auditing firms, the American Institute of CPAs and three members from outside the public company auditing profession.
Representatives of each of the major firms participated in the working group to develop the CAQ’s professional judgment resource. I was McGladrey’s representative on the task force. The judgment resource describes a judgment process and the common traps and biases.
I think that the CAQ’s decision to invest in a judgment resource evidences the importance of this topic to the profession.

36. Polling Question 3 (CPE Credit)
The confirmation bias is a subconscious tendency to do which of the following?
Seek evidence that confirms a biased judgment
Seek evidence that confirms a previously held view
Underutilize confirmations in the testing of accounts receivable
Seek evidence that disconfirms a previously held view

37. Integrating Professional Judgment into the Classroom:
Can Accounting Students Learn Professional Judgment in the Classroom?

38. Can You Really Teach Good Judgment?
Is it a natural ability or a skill that can be developed?
True or False: Either you have it or you do not…
Good News!
It is a skill that can be learned and developed
It helps to know what “good judgment” looks like and the common threats to good judgment
The sooner you start learning how to make good professional judgments, the better!

39. Factors in Developing the Ability to Exercise Good Professional Judgment
Do you agree the following factors contribute to the development of good judgment?
Natural ability
Personal experience
Observing others including mentors
Self-studies/books
AND…
Formal training

40. How does a visual illusion relate to judgment?
Source:

41. GRAPHIC OF GOLF SWING

42. The KPMG Professional Judgment Framework
Consultation 42

43. Practice makes permanent
Phil Mickelson Golf Swing

44. Threats to good judgment
Not following a good process
Judgment “traps” and “biases”
Mental shortcuts
Help simplify a complex situation
Can facilitate more efficient judgments
USUALLY effective
NEW

45. The purpose and benefits of the topic
Mindset, skills, and techniques behind good judgment begin to form at a young age
Training, deliberate practice, and experience can improve judgment
It is important for students to have a solid foundation in good judgment

46. How does awareness help with this illusion?
Source:

47. KPMG Professional Judgment Framework Monograph
John: With the help of Doug, Steve, and Jeff, we developed a monograph that explains our judgment framework. We distributed this framework to all of our professionals in May We continue to provide copies of the monograph to new professionals when the begin their career at McGladrey.

48. McGladrey Monograph
John: With the help of Doug, Steve, and Jeff, we developed a monograph that explains our judgment framework. We distributed this framework to all of our professionals in May We continue to provide copies of the monograph to new professionals when the begin their career at McGladrey.

49. Polling Question 4 (CPE Credit)
You just cannot teach professional judgment; either you have it or you do not.
True
False

50. Q&A