Test Validation Explained
Dr. Wendall Williams
Validation means making sure that pre-hire test scores accurately predict performance. How can your argue with that? Yes, Virginia, interviews are pre-hiring tests. People get screened out by interviews. This means interviews are like every other form of pre-screen test: they should be validated. Otherwise, how can an interviewer know with any degree of certainty whether folks who pass interviews can do the job?
Coming to grips with interviews as tests gives most people a headache because it forces them to come face to face with hard data rather than soft opinions. It makes no difference if an interviewer is 100% convinced of the accuracy of his or her interview technique if there is no feedback or hard evidence to support it. Why? Because human decision-making is flawed.
Some people, for example, are convinced that flying is significantly more dangerous than driving, when exactly the opposite is true. Others believe they can win the lottery when there is a much greater chance of being hit by lightning. Just as surprisingly, although there is an embarrassing number of low producers on most payrolls, hiring managers still generally argue that "they know 'em when they see 'em."
Basically, recruiters must always be aware of the flaws that affect human decision-making. People tend to readily recall information that is vivid (i.e., a major accident) or recent (i.e., happened within the last few days, weeks, or months), that confirms existing opinions (i.e., we "stereotype" people), or that is readily available (i.e., we see it every night on CNN).
Flawed decision-making in recruiting leads to flaky job standards, hiring the wrong people, and rejecting the right ones. It is a major reason why Congress passed the Civil Rights Act and the Department of Labor wrote the 1978 Uniform Guidelines on Employee Selection Procedures.
Anecdotes Are Examples of Flawed Decision-Making
I know a brother of an aunt who knew an actress who had a good experience with a shopkeeper who hired people using the "Seems Like It Might" (SLIM) performance test. In fact, the "Seems Like it Might" company proudly markets the fact that recruiters have a SLIM chance of making a good hiring decision based on their test results.
Many people who avidly support certain tests do so because they think the test accurately predicts individual performance. When they are probed, we discover there is absolutely nothing except folklore and superstition to support their opinion. These opinions are not fact; they are homilies and anecdotes. They serve us well when there is nothing important at stake, but cost our organizations millions in lost productivity when they affect hiring decisions.
We really need to rethink our profession and hammer home the point that this is not a "learn as you earn" job. Questions and unfounded recommendations about "best" interview questions and generic interview-workshop competency lists indicate an embarrassing lack of professional knowledge. Imagine a group of physicians asking questions about where to find sharp scalpels, engineers asking for recommendations about building materials, or policemen asking about the best bullets for shooting suspects. Based on feedback and questions in the public forum, one would think that setting clear job standards before starting a job search was akin to discovering cold fusion.
How does a professional become more professional? Read the right books. Go back to school for a semester. Read the research. Do anything backed by good sense. But stamp out the idea once and for all that this business is as easy as giving an applicant a silly test. Want a good doctor? Find one with medical training. Want a good architect? Find one who has experience in construction. Want a good recruiter? Find one who knows how to set job standards and who can fairly and accurately measure applicant skills.
Correlation Is Not Causation
Suppose the National Enquirer published a nice ten-question hiring test. Furthermore, suppose we gave that test to our high producers and averaged their scores. Is that validation? (Do alligators make good house pets?)
Correlation means there is an association between two variables, i.e., high producers tend to be mentally sane. But correlation is not enough. Recruiters need to find causation. Does sanity cause high production (causation)? Or do high producers tend to be sane (correlation)? Toss all your training tests in the closet unless you know, for certain, that the content they test for causes productivity.
Styles as measured by the MBTI, DISC, Enneagram, Social Styles, and so forth, might occur more often among certain job holders, but life is too complex to assume style causes productivity. This is critical to remember because, while hiring managers might embrace an intuitively attractive test today, if is does not predict performance, it will fail over time.
Recruiters need to know that "what is measured" equals "on-the-job performance." "Getting to know the applicant" or playing amateur psychologist is pure bush league. Remember that blue eyes and blond hair might be correlated, but one does not cause the other.
Cutting Through the Dreck
Suppose we accept that 1) causation is the only way to accurately predict performance and, 2) our test content is a pre-cursor to job performance. Our next question is how to prove (i.e., validate) the test and arrive at good cut-off points. This takes a thorough knowledge of statistics and experimental design.
For example, we have to define what to predict. Is it supervisor ratings? Performance appraisals? Three-sixty survey results? No, these are probably filled with error and subjectivity that would yield untrustworthy results. We have to find "hard" data that is hard to fake, something we know we can trust.
Okay, let's suppose we have the right kind of hard data. What's next? We need to compare test scores with on-the-job performance. We can do this several ways:
By giving the test to everyone who applies, hiring them all, waiting until we get performance data, and comparing test scores with job performance.
By giving the test to people already in the job and comparing test scores with job performance.
Of course we'll need to correct for "restriction of range," that is, the people who are in the job will be more similar than people who apply for the job. So what? Well, for one thing, we might not see the same kind of big differences between high and low producers that we would see between applicants. Bummer! For example, we might find there is very little difference in skill between the top 10 players on the pro golf tour and the bottom 10. On the other hand, we would probably find a very large difference between the top 10 spectators and the bottom 10.
Adverse Impact
Discriminating against qualified people based on non-job-related factors such as race, gender, or disability is a bad practice (it's also against the law in the U.S.). But that does not mean we have to hire people just because of race, gender, or disability either. We need clear job standards that are based on job requirements, business necessity, plenty of documentation, validated tests, and monitored pass rates for each decision point. Can we discriminate? Yes — but only against people who do not have job skills.
This is a "quicksand" occupation. It looks calm on the surface, but it is dangerously deep for naive folks who wander too close. Recruiters should never rely on one tool, should always do job analyses, should never ignore the law, should validate every test, should acknowledge flawed decision-making, and should do everything they can to become more professional.
Articles
Finding Candidates
- 10 Reasons to Hire Vets
- Hire Older Workers
- Improving Candidate Quality
- Sourcing 101
- Sourcing Candidates Well
- Tips for Building Employment-Related Websites
- U.S. Employ of People with Disabilities: Free Workshops
- Virtual College Recruiting
Interviewing Basics
- 10 Commandments of Recruiting
- 5 Keys to Successful Hiring
- 7 Tips for Successful Phone Interviews
- Behavioral Interviewing Basics
- Contrary Evidence Questions
- Interview Questions: Do's and Don'ts
- Interviewing Opening and Closing Remarks
- Interviews: Common Weaknesses
- Mistakes Amateur Interviewers Make
- Phone Screen Interview Mistakes
- Probing Techniques Explained
- Screening Interviewing: Top 10 Red Flags
- Strengthen the Validity of Your Interviews
- Telephone Interviews: Basics
- Ten Bad Listening Habits of Interviewers
- Types of Interviews
- Typical Probes and Follow up Questions
- What Do Interviewers Need to Know to be Effective?
Interviewing Best Practices
- 7 Keys to Effective Selection Interviews
- A Closer Look at Behavior-Based Interviewing
- Advantages / Disadvantages of Interviewing
- Applying Core Competencies to Selection Interviews
- Are You Really a Behavior-Based Interviewer?
- Assessing Speaking and Listening Skills
- Best Practices in Interviewing Candidates
- Deadly Interview Mistakes
- Death by Interview
- Ensure Hiring Success in Every Situation
- Executive Assessment Should Be Mandatory
- Generational Interviewing
- Hiring Interview + Strategic Applicant Management
- Hold Evening and Off-Time Interviews
- How to Interview a Top Performer
- Improve your Interviewing Techniques
- Interview Questions to Assess Soft Skills
- Interviewing for Ethics
- Interviewing Millennials
- Interviewing: Business or Psychology
- Metrics Interview
- Peeling Back the Onion
- The Positives of Panel Interviews
- Time for Candidate Advocacy?
- Tips for Conducting Successful Interviews
- Two Critical Interviewing Questions
Laws & Documentation
- Applicant Reference Release
- At Will Employment Release
- Avoid Negligent Hiring Mistakes
- Employee Referral Program Metrics
- Fair Labor Standards Act Information
- Four Interview Questions Never to Ask
- Giving Employee References
- Hiring Compliance Guidelines
- Hiring for Small Business
- Interviewing People with Disabilities
- Job Denial Letter
- Legal Issues in Interviewing
- Minimize Employment Risks: Document
- SAMPLE Employment Policy
- SAMPLE Letter: Educational Records Check
- SAMPLE Letter: Reference Check
- Ten Safe Hiring Tools
- What is Negligent Hiring
Line Manager / Recruiting Partnership
- Defending Candidates to Hiring Managers
- Interlocking Core Competency Interviews
- Internal Application Process
- Making Internships Work for You
- Making the Case for Behavioral Interviewing
- Non-Traditional Merit Pay Alternatives
- OFCCP Definition of an Internet Applicant
- Why Managers Shouldn't Do Most Hiring
- Workforce Planning: Strategic Staffing Strategy
Post-Interview
Pre-Planning & Retention
- Bonus or Incentive?
- Brand-Building on a Budget
- Build a First-Rate Hiring Process
- Closing the Deal
- Compensation Plans: An Overview
- Conducting an Exit Interview
- Good Hiring Starts with a Good Job Profile
- Improve the Quality of the Employment Function
- Interview Process Problems
- Interview the Job Before the Candidates
- Job Description Template-Link Pay to Performance
- Linking Pay to Company Performance
- Selecting and Using Salary Surveys
- Succession Planning
- Succession Planning: Identifying Top Performers
- Using a Pre-Interviewing Questionnaire
- Winning the War for Talent
- Worker Shortage by 2010: Preparation
Reading the Candidate
- Beware of Those Who Boast
- Blind Man's Bluff
- Decision, Decisions: Choosing the Better Applicant
- Detecting Deceit in Interviews
- Little White Lies on Resumes
- Suspend Judgment Until the Interview is Over
Recruiting Basics
- College Recruiting Basics
- College Recruiting Essentials
- Cut Down on Interview No Shows
- Discouraging Low Quality Applicants
- Don't Hold Too Many Interviews
- Job Descriptions: Why are they Important?
- New Strategies for Screening Job Candidates
- Preventing Resume Overload with Questionnaires
- Resume Review Basics
- Test Validation Explained
- The Value of Person-Organization Fit
- Three Companies Cut Turnover with Tests
Recruiting Best Practices
- 25 Telltale Signs of the Wrong Candidate
- 5 Overlooked Ways to Hire Winners
- Asking the Right Recruitment Questions
- Attracting Your Competitor's Employees
- BPR.......for Recruiters!
- Candidate Engagement
- Cloud Recruiting
- Evaluate Your Capture Strategy
- Hiring Best Practices
- How Do You Attract and Retain the Best People?
- How to Attract Applicants to Undesirable Jobs
- How to Attract, Develop and Retain Best People
- How to Find and Keep Valued Employees
- Ignorance and the Human Condition
- Onboarding Success Secrets
- Secrets to Non-Profit Hiring
- Selecting Top Management Talent
- Semi-Active Candidates are Best Bets
- Six Core Selling Principles
- Skills Based Recruiting: When, not How
- Smart Choices: How to Hire the Best
- Strategy for Hiring the Best This Year
- The Uses and Misuses of Personality Tests
- Top 10 Employee Selection Mistakes & Solutions
- Treat Candidates with the Carbon Rule