Testing for Expert Judgement

Expert judgement is the ability to make predictions and avoid problems in a given domain.  How can you test the judgement of somebody on your team?  You can observe them over time, or you can accelerate the process by asking them about a topic they are passionate about.  In The First 90 Days: Critical Success Strategies for New Leaders at All Levels, Michael Watkins writes about how to test a person's capacity for expert judgement.

Make sure You are Assessing Judgement
Watkins writes:

"Make sure you are assessing judgement and not technical competence or raw intelligence.  Some very bright people have lousy judgement, and some people of average competence have extraordinary judgment.  It is essential to be clear about the mix of knowledge and judgement you need from key people."

Making Predictions and Avoiding Problems
Watkins writes:

"One way to assess judgement is to work with a person for an extended time and observe whether he or she is able to (1) make sound predictions and (2) develop good strategies for avoiding problems.  Both abilities draw on an individual's mental models, or ways of identifying the essential features and dynamics of emerging situations and translating those insights into effective action.  This is what expert judgement is all about.  The problem of course, is that you don't have much time, and it can take a while to find out whether someone did or did not make good predictions.  Fortunately, there are ways you can accelerate this process."

Accelerating Testing for Expert Judgement
Watkins writes:

"One way is to test people's judgement in a domain in which feedback on their prediction abilities will emerge relatively quickly.  Experiment with the following approach.  Ask individuals whose judgement you want to evaluate about a topic that they are passionate about outside work.  It could be politics or cooking or baseball; it doesn't matter.  Challenge them to make predictions: 'Who do you think is going to do better in the debate?' 'What does it take to bake a perfect souffle?' 'Which team will win the game tonight?'  Press them to commit themselves -- unwillingness to go out on a limb is a warning sign in itself.  Then probe why they think their predictions are correct.  Does the rationale make sense?  If possible, follow up to see what happens.

What you are testing is a person's capacity to exercise expert judgement in a particular domain.  Someone who has become an expert in a private domain is likely to have done so in his or her chose field of business too, given enough passion about it.  However you do it, the key is to find ways, beyond just waiting to see how people will perform on the job, to probe for the hallmarks of expertise."

Key Take Aways
Here's my key take aways:

• Expert judgement is the ability to make predictions and avoid problems.
• Having technical expertise in a domain is not the same as demonstrating sound judgement.
• Find ways to test for expert judgement beyond waiting to see how people perform on the job.
• You can speed up the process by asking somebody about an area outside of work and seeing how well they make predictions.