Psyched Out by a Psyc Exam!
In politics, there are Democrats and Republicans. In baseball, there are Yankee fans and Red Sox fans. In consumer electronics, there are Apple fanatics and Android devotees. And, in the world of statistics, there are Frequentists and Bayesians.
In his recent blog post entitled, “Frequentism and Bayesianism: A Practical Introduction“, Jake VanderPlas describes Frequentists and Bayesians as follows: “For Frequentists probabilities are fundamentally related to frequencies of events.” “For Bayesians, probabilities are fundamentally related to our knowledge about an event.” The difference may appear subtle, more philosophical than practical. In reality, however, both their application and solutions can be markedly different.
Long before I had any idea of the existence of either faction, I came upon an experience that would set me down a path of appreciation for one and a passion for the other. Note: By no means is this meant to be a rigorous introduction to statistics. I’m just relaying an interesting story with an underlying statistical interpretation. Take it for what it is.
How a night of too much fun lead to a Bayesian decision.
It’s 1981, as a University of Michigan sophomore, I’m set to enjoy my first spring in Ann Arbor, Michigan. So as not to waste the next four months by doing absolutely nothing, I decided to take a class, Introduction to Psychology. Spring term, being about half the number of weeks of a full term, packs lots of information into too little time. On the upside, however, things are much more laid back and the grading curve far more generous.
I’d never had a Psychology class. The topic interested me and I’d heard the class didn’t require a ton of work. Periodically, class was held outdoors on the grass surrounding the Diag. It didn’t do much for my concentration but it certainly was relaxing.
As it turns out, the class offered me a very memorable experience. One I’ve shared with many over the years. Most of what I retained from that class came during our first multiple choice exam. Beyond the exam, I don’t remember much about that day, though the night before hadn’t been spent properly preparing for the exam. Hey, it was spring!
I arrived at Mason Hall prepared for a mediocre performance, at best. I left with the second most surprising exam experience of my life. The test was 50 multiple choice questions. The choices were A, B , C, or D. We had about an hour in which to complete the exam. Once handed the exam I looked over its several pages and sampled a few questions. Yup, my limited command of the material seemed pretty uniform throughout. Oh well, there’s no time to waste. I charged ahead.
I started with the first question. I can’t remember what it was (or any of the others for that matter). I do recall my chosen answer, C. On to question 2, by process of elimination, I responded with C. The third question, I actually knew the answer to this one, C. Hmmm…. Next, questions 4, 5, and 6. By the time I’d gotten to the tenth question, I could see a pattern forming — ten questions, ten Cs. That can’t be right! What’s going on here? I could have studied more, but that doesn’t explain why I’d think each answer is C? Am I on to something here?
Time spent at OTB. Not such a waste after all.
As a kid growing up in New York City in the 1970s, I spent my fair share of time hanging out at a New York institution of higher learning, OTB (Off Track Betting). These were smoky, street corner betting parlors populated by horse racing aficionados who’d made significant lifetime capital investments in such attractions as Aqueduct Racetrack, Belmont Park, and Saratoga Raceway. These men (and a few women) represented New York’s equestrian elite. They smoked like chimneys, smelled like bar rooms, ate deli every meal, and cursed like sailors. OTB was an ideal place for a pre-internet kid living in the big city to learn a few of life’s important lessons: computing race track odds, expected value, and the house take, as well as the ins and outs of para-mutual betting and managing your bank roll.
Here’s where that misspent time started to pay off. Observe and adjust! Ten Cs in a row! Seems pretty unlikely. While ruminating on my situation, I felt the tension in the classroom rising. My classmates were talking to themselves, fidgeting in their chairs, looking aimlessly about, and asking the TA pointless questions. After taking it all in, I felt I was on to something. I dared to ask myself, what if the answer to all 50 questions is C?! Is the fix in?
Going into the exam, I was of the mind that no answer should be favored over any other, they’re all equally probable. The probability of any given answer (A, B, C, or D) is 1/4. Like a two-headed coin or a six-sided die, the selection of answers is assumed to “fair” and “unbiased”. After answering C to the first ten questions and taking note of the discomfort of my fellow test takers, I asked myself if I really wanted to stand by my initial belief (the test is fair) or make an adjustment. Do I hedge my bet? It’s raining, the track is wet, the smart money has abandoned the favorite for a different horse, a mudder.
At the time, I didn’t possess all the tools I’d need to compute the probability that the answer to all 50 questions is C. For the curious, assuming the test is “fair”, the probability is about 8.0 x 10-31, a pretty small number indeed! Tools aside, I had a sense of how to quickly attack the problem in a way that would satisfy me enough to reasonably conclude, one way or the other, if all the answers were C.
My plan: Sample about half a dozen of the remaining 40 questions. If my response to each is C, then I’ll assume all 40 remaining answers are C. How do I choose my sample? First, I’d answer the very last question. I figured if the powers that be were trying to mess with my mind, the 50th question was a good place to check. My answer to Q.50 was C. If the answer to the first ten questions are all C and the last one is C too, well! On to step two of the plan. Choose one question at random from each of the four groups of ten questions that comprise the remaining 40 questions (e.g., Q.11-20, Q.21-30, etc.). The answers, C, C, C, and C. Holy cow Batman! This horse will hunt!
It’s an hour exam, every question I’ve answered was C, and the class is getting more fidgety by the minute. I’ve taken so long to assess the situation that I’ve only got about 20 minutes left to answer the remaining 35 questions. My grasp of the material isn’t on the strong side. What do I do?
I’m convinced I need to reject my original hypothesis. The answer to each question isn’t’ unbiased. In fact, it’s highly biased, it’s C! Time to dive in! At this point, taking everything into account (tangible and intangible), I put the odds at better than 5 to 1 in my favor. If I’m wrong, I’d expect to get a 25% on the test. If I’m right, I should get 100%. That’s an expected value of about 88%. With those odds, I bet $100 to Win on “all 50 questions are C!”
I completed the exam and handed it in. I was the first to finish. I’m never the first to finish! The TA gave my test a quick look and then smiled at me. I KNEW IT! Time to cash in! Thanks OTB!
The payoff. And, messing with young impressionable minds.
I wasn’t prepared for the events of our next meeting. It turned out to be one of the most interesting classes I’ve ever attended. To the best of my recollection there were about 15 to 20 people in the classroom. As they arrived there was a lot of chatter about the exam. Many thought it was very difficult. Finally, the TA arrived. He launched into the next section. After a minute or so, a young lady asked if he had graded the exams. He said he had. She asked if he would pass them out and discuss the test. Almost instantly the rest of the class chimed in. The TA smiled and complied. I sat by quietly.
As he passed back the tests the murmurs got louder, tensions rose, and things began to get heated. I had expected laughter and smiles. A realization of what had gone on. Boy was I wrong. The young lady who had originally asked that the test be handed back shouted from the front row, “This is unfair!” Her voice cracked. She was visibly upset. The TA asked for an explanation. She responded, it’s not fair that all 50 questions should have the same answer! She wasn’t alone in her thinking, several others joined her in support. The TA asked, “Why isn’t it fair?”. I wondered, how many Cs would have been fair. At what point do you cross the line? Are there certain answer patterns that are acceptable while others are not? The TA tried repeatedly to extract a rational explanation from anybody with an objection. None was forthcoming. Just a whole lot of incoherent mumbling and grumbling.
The TA went on to explain that we’d been part of an experiment. The objective: To observe the effects on test takers of having every answer to a multiple choice exam be the same. Fantastic! As it turns out, he explained, they are pretty pronounced. On average, students score significantly lower, are less likely to complete the exam, and report higher levels of stress while taking the exam.
Upon hearing this, the young lady in the front row had a complete meltdown. She began to sob, continued to insist the exam was unfair, and charged out of the room. It’s been so long, I can’t remember if she ever came back, she may have dropped the class. From her reaction, I suspect this may have been the first time she hadn’t received an A on a test.
I thought the experiment was brilliant. Highly appropriate for an introductory psychology class. It was a terrific real world example of how people react to unexpected events and how they can influence our ability to make rational decisions. Look, if the answer is C, then its C! What difference does it make that the prior answer was C or that the last ten answers were C. It shouldn’t make a difference, but it does!
That was the best I did on any of the exams that spring. Had I not been ill-prepared for the exam, I probably would have fallen into the same trap as most of the rest of the class. I guess good things can come to those who stay up too late having too much fun!
The more important point! Bayesians versus Frequentists.
It took a number of years and a lot more learning before I came to appreciate the importance of this experience. Businesses face similar circumstances every day. Short time intervals, little reliable information, tight budgets, go/no-go decisions. How they respond is often the difference between success and failure. The difference between the iPhone and the Newton. Between Coke and New Coke!
As Statisticians we’re taught to embrace experimental design, random sampling, hypothesis testing, confidence intervals, and host of other techniques. How useful would any of those tools have been during my exam? Not very! The lesson to be learned, the concept to be embraced, it’s just as important to develop a good sense of the problem, the situation, and its limitations as it is to be able to design and deploy an experiment or analyze and present the data. In fact, if you want to be good at what you do, it’s even more important.
It wouldn’t be fair to say the frequentist approach is out dated. It’s been around for a long time and has contributed greatly to our understanding of our world and the universe. Though characterizing it as “B.C.”, Before Computers, seems reasonable. While the complexities of Bayesian methods can be daunting, they’ve flourished in an era of expansive software sophistication and abundantly cheap computing power. In his book, “Doing Bayesian Data Analysis“, John Kruschke states, “It has only been with the development of MCMC algorithms and software that Bayesian inference is applicable to complex data analysis, and it has only been with the production of fast and cheap computer hardware that Bayesian inference is accessible to a wide audience.”
What began as a basic psychology test quickly became an elementary exercise in Bayesian thinking. The subtle difference between perception of probability as “related to frequencies of events” or “related to our knowledge about an event” became far more pronounced when faced with an unusual situation and less than an hour to decide how to proceed. In this instance, the Bayesian path seemed to be the only viable option. If you’re unfamiliar with Bayes Theorem and the body of work that surrounds it, I’d highly recommend you take the time to familiarize yourself with the basics. You never know when it might pay off.
By the way, the most surprising exam I’d ever taken? Freshman year, WWII History midterm, the professor, an elderly veteran, very nice man, handed out a WWI exam. When asked by a bunch of confused students what was going on, he replied, “Oh! Have fun with it!” The grades were very generous.
Copyright (c) 2015 Bayes, LLC. All Right Reserved.