18 November 2009

You Know You're Right

Hello reader, I've prepared a little thought experiment for you. I am going to give you a set of 3 numbers that fit a specific rule. Next, you're going to come up with a subsequent 3 numbers that match the same rule. When you do that, I'm going to tell you whether your numbers fit the rule or not and then you're going to guess what the rule was. In order for this experiment to work properly, please don't read any further than the step you're on. Be honest and don't think too hard, it's really not tricky at all.



Step 1: Here are the numbers: 2, 4, 6



Step 2: Now, you get to fill in the next 3 numbers: (Please only use this form once, unless the experiment instructs otherwise.)

<-- click this button when you've entered the numbers.



Step 3: Based on the response you received, write down what you think the rule is that the numbers are following.




Step 3.1: This is optional. Since you have a record of what you believe to be the rule, I'll allow you to try the form once more if you're not 100% positive. If you do this, record what you think the rule is again. Otherwise, go to step 4.




Step 4: To reveal the rule, click and drag your mouse over this white block to select the text: The rule is: any number in ascending order. Does your note in Step 3 say the same thing?



Step 5: Were you right? The first time? If you were, congratulations! If you weren't, don't despair - there's a reason for this. This experiment was designed to show you a little bit about what is called the confirmation bias. It goes like this: you get an idea in your head and then you start looking for clues that confirm your idea. In this experiment, if you follow the clues that you look for, they'll be confirmed and it will reinforce your idea. Unfortunately, it only leads you further from the truth. It is usually when you doubt your initial hunch that you will be able to see clearly.

The next time you hear someone spouting off all of the reasons why they believe something, send them to this experiment. It is important to remind ourselves that even with all the evidence we can pile up that support our claims, corroboration is merely in the eye of the beholder. Just because your 8, 10 and 12 match the pattern you were looking for does not prove the rule you thought of was correct. And so it goes in all walks of life...
Posted by Andrew Gonsalves at 8:28 PM
Labels:

19 nibbles:

  1. I haven't figured out what the rule is, but it is not what you give as the answer! For example, confirming 3, 333, 343 answers "no".
    ReplyDelete
  2. Karl: The answer is correct, your numbers just don't follow the 3 that were given.
    ReplyDelete
  3. "any number in ascending order" is biased the same way. You could have picked: "Any number", it would have worked as well. What are "any numbers" that can appear after 2, 4, 6? 1, 9, 12 for example. They follow the rule, and are subsequent by virtue of the fact that you gave the first 3, and I gave the second set of 3; they are part of the same sequence, and there is no implication of mathematical order in my rule.
    ReplyDelete
  4. Sébastien: 1, 9 and 12 do not follow the rule. I'm not sure how you're coming to that conclusion. 1 does not ascend from 8. Also, I'm not sure how you can call a rule "biased" - it is just the rule you're trying to discover and you're using your own biases to figure it out.
    ReplyDelete
  5. No, 1, 9, 12 follow my rule, which is "Any Number". What I mean, is that given your 3 numbers, you can come up with several rules that will validate or invalidate another series of 3 numbers. The one that fits 8, 10, 12 is valid given "2 4 6", mine is valid, yours work too. My point was that people will most likely pick 8, 10, 12 because it's natural to favor even/odd/multiple relationships before mathematical ordering in this type of situation. And yes, I agree, it's dangerous to follow such intuition/bias. But in the same way, your choice of rule is biased as well towards ascending order, because that's how you built the problem. The "Any Number" rule though, works. I give you 2, 4, 6. You give me 1, 9, 12. They match that rule, i.e. I could have written a similar post demonstrating that your more-specific rule was biased, since you would have most likely picked "any number in ascending order" before the less-specific "any number".
    ReplyDelete
  6. Sébastien: The point of the exercise is to figure out my rule, not your rule. I realize that the numbers I gave could lead you to come up with a whole set of other rules, but that's not the point. The point is that you came up with your rule and then looked for clues to corroborate it. Your rule (thus: interpretation) did not affect the actual rule (thus: reality) even though you might find clues that lead you to believe that your rule is the one in play.
    ReplyDelete
  7. I guess I'm just not that blown away :) I'm not "convinced" my rule is in play, I'm just giving you a possible rule. Does it discard other rules? Not really, but you just asked for one :) If you are *pressed* to find a rule, then I don't see how else you can act on that test but by trying to fit a model (a rule) to the data (3 numbers), keeping in mind that it is very very sparse. I think "confirmation bias" sounded a little too fancy to me for what looks like a bad case of good old generalization. Deductive inference goes wrong all the time...
    ReplyDelete
  8. Errr. You're totally over-thinking this. It is very simple, for the purpose of generalization: 1. You see evidence and you create a hypothesis. 2. You seek evidence to confirm that hypothesis. 3. You claim knowledge because your evidence corroborates your hypothesis. 4. You were wrong all along, even though your empirical evidence said you were right.
    ReplyDelete
  9. People like to reinforce their beliefs.

    Attendance/participation with simliar minded groups (churches, clubs, anything on the internet, television programs, etc...) demonstrate a similar effect.

    They go to the groups because they share beliefs. When their bias is confirmed by others, the belief (no matter how true or false) is strongly enforced.

    This is the danger of only reinforcing ideas from the similar minded.

    I read this site because I disagree with a lot of what you say.
    ReplyDelete
  10. I really enjoyed this post BTW.
    ReplyDelete
  11. JD: You're sending mixed signals. :p
    ReplyDelete
  12. I'm confused as to what this has to do with confirmation bias.

    It says something to me about the natural predilection we have for pattern detection and matching, that is all.

    I skipped 3.1 for example. I wasn't a 100% sure but you only permitted one more try. Forgive me, I'm a literalist. And with one more try I couldn't begin to explore the enormous realm of possible rules beyond the alluringly obvious pattern. If you'd invited me to explore at depth I might have struck fired off a few more exploratory efforts to test the accuracy of my rule. With 1 try I simply felt it not worth my time.

    That itself was a bias (away from efforts judged to have a probably low return). Because with one simple test (including odd numbers) I would have confirmed my suspicion was invalid.

    A demonstration of confirmation bias would be if I simply tried another sequence of even numbers (they would confirm "a subsequent 3 numbers that match the same rule"). But as it stands the exercise revealed nothing to me about confirmation bias, only something about pattern detection and hypothesis testing.

    I tested 8, 10, 12. It failed to disprove the hypothesis, and so my confidence rose.

    To demonstrate confirmation bias I would need to find myself testing more and more samples that match my null hypothesis exclusively because the minute I test one which doesn't - which is likely to happen inside of a few tries, by my third stab I'd guess at least as I suspect I'd have tried:

    8, 10, 12
    something like 32, 34, 36
    then maybe something like 9, 10, 11
    and carried on from there.

    Nice try, but I think the test could use some refinement. Confirmation bias certainly exists, that I don't doubt, but this didn't impress me a as a demonstration of it.
    ReplyDelete
  13. Andrew,

    It sounds like the point Sebastien is making is that this example is entirely arbitrary, and you seem to have bent over backward to trick people into giving the "wrong" answer. You're presenting us with a logic question, and sure, your rule is one possible solution. But the true answer, if somewhere were presented this as a true problem with either be "there could be multiple rules" or more likely "there is not enough information".

    This is why datasets of three are not often used, and I'd like to see any scientist in any field claim that results from such a small dataset suffer from confirmation bias before they claim, oh, I don't know, lack of data.

    It seems like you wanted to make a very valid point about something that we encounter every day, but then dashed off a hasty and ill-thought out blog post. A better approach might have been to research a few studies . Or maybe just point to the Wikipedia article. This is just not a very convincing display, as we can all tell you're simply trying to be clever here.
    ReplyDelete
  14. Bernd: Compartmentalizing mathematics from judgment is one way to miss the point. Alas, no concept is an island - everything is related. You were given the option to test more if you wanted to really nail your hypothesis.

    photothis: A dataset of 3 is not the problem. This works just as well with a dataset of 2 or 1. This is because the idea is not necessarily to paint a clear picture to provide you with a good chance of guessing the rule. Instead, the dataset is short and vague, causing you to make a judgment and then try out your hypothesis. If you thought up a rule, it meant that a small dataset is still enough for you to postulate on. I'm thinking you just didn't like the purposely misleading nature of the experiment. It is a variant of one thought up by a published psychologist.
    ReplyDelete
  15. Oh, well I stand corrected, an appeal to authority makes the whole difference.

    ;)

    How about a link to the experiment you're basing this on?
    ReplyDelete
  16. Heheh. You got me, using this argument from authority fallacy to get my point across. I can't promise I'll get around to this, but I'll see if I can find an online version of the study I was reading.
    ReplyDelete
  17. "Compartmentalizing mathematics from judgment is one way to miss the point"

    I must have missed the point as I have no idea what compartmentalizing mathematics and judgment means.

    "You were given the option to test more"

    We were invited to try one more time only. Which as I described left me with a sense of "What for? I'm aware that any number of rules could produce these three numbers, and I'm not going to find the rule with two tests ...".

    Perhaps if it had made an open ended invitation to test as many solutions as we'd liked (and optionally recorded these and played them back for retrospection) or set a challenge a la mastermind, to see how few tests you need to conclude with confidence what the rule is, then I'd have been lured into trying again. 3.1 simply left me stumped. No I was not 100% sure, but nor was I going to be by trying something else.

    In retrospect a truly literal interpretation I failed to make is that trying again means going to 1, then 2, and the 3 and then 3.1 again so 3.1 can be invoked ad infinitum. The spirit of the exercise as I read it though was to have a second go if you felt it would help. It clearly wasn't going to help (the space of possible rules is too big).

    Alas, unlike photothis, I'm not so impressed by this invocation of authority. Many a publish paper comes under significant criticism post publication and indeed this is part of the intent of the whole scientific publication system. The pre-publication peer review is intended simply to reduce the amount of broader peer rejection. But broader rejection of methods is far from uncommon still, and I'd have to read the paper to judge it, but cannot rule out that I have similar complaints against the conclusions it draws from the methods it employs. It means little to me until I see the paper (it falls into the category of dinner table conversation I described in an earlier sub-blog, in which hand-wavy references to "I read it somewhere reputable" are made, but the effects of memory and paraphrasing may well be dominant in the recounting, and the source not subject to scrutiny either - authority inovoke ...). But then Andrew already admitted as much "the argument from authority fallacy".
    ReplyDelete
  18. > Alas, unlike photothis, I'm not so impressed by this invocation of authority.

    That was sarcasm...
    ReplyDelete
  19. Ha ha, thanks Sebastian, detecting sarcasm is a known weakness I mine I admit. Thanks for the headsup. I'm guessing the ;) on following line was the cue I missed. Perhaps Andrew will write us a piece on sarcasm some time ;-).
    ReplyDelete

Subscribe to: Post Comments (Atom)