Comments on Don't Feed the Animals: You Know You're Right

Ha ha, thanks Sebastian, detecting sarcasm is a kn...

2009-11-22T20:57:13.886-05:00

Ha ha, thanks Sebastian, detecting sarcasm is a known weakness I mine I admit. Thanks for the headsup. I'm guessing the ;) on following line was the cue I missed. Perhaps Andrew will write us a piece on sarcasm some time ;-).

> Alas, unlike photothis, I'm not so impres...

2009-11-22T18:58:37.719-05:00

> Alas, unlike photothis, I'm not so impressed by this invocation of authority.

That was sarcasm...

"Compartmentalizing mathematics from judgment...

2009-11-22T18:55:21.775-05:00

"Compartmentalizing mathematics from judgment is one way to miss the point"

I must have missed the point as I have no idea what compartmentalizing mathematics and judgment means.

"You were given the option to test more"

We were invited to try one more time only. Which as I described left me with a sense of "What for? I'm aware that any number of rules could produce these three numbers, and I'm not going to find the rule with two tests ...".

Perhaps if it had made an open ended invitation to test as many solutions as we'd liked (and optionally recorded these and played them back for retrospection) or set a challenge a la mastermind, to see how few tests you need to conclude with confidence what the rule is, then I'd have been lured into trying again. 3.1 simply left me stumped. No I was not 100% sure, but nor was I going to be by trying something else.

In retrospect a truly literal interpretation I failed to make is that trying again means going to 1, then 2, and the 3 and then 3.1 again so 3.1 can be invoked ad infinitum. The spirit of the exercise as I read it though was to have a second go if you felt it would help. It clearly wasn't going to help (the space of possible rules is too big).

Alas, unlike photothis, I'm not so impressed by this invocation of authority. Many a publish paper comes under significant criticism post publication and indeed this is part of the intent of the whole scientific publication system. The pre-publication peer review is intended simply to reduce the amount of broader peer rejection. But broader rejection of methods is far from uncommon still, and I'd have to read the paper to judge it, but cannot rule out that I have similar complaints against the conclusions it draws from the methods it employs. It means little to me until I see the paper (it falls into the category of dinner table conversation I described in an earlier sub-blog, in which hand-wavy references to "I read it somewhere reputable" are made, but the effects of memory and paraphrasing may well be dominant in the recounting, and the source not subject to scrutiny either - authority inovoke ...). But then Andrew already admitted as much "the argument from authority fallacy".

Heheh. You got me, using this argument from author...

2009-11-20T09:57:40.842-05:00

Heheh. You got me, using this argument from authority fallacy to get my point across. I can't promise I'll get around to this, but I'll see if I can find an online version of the study I was reading.

Oh, well I stand corrected, an appeal to authority...

2009-11-20T09:34:57.585-05:00

Oh, well I stand corrected, an appeal to authority makes the whole difference.

;)

How about a link to the experiment you're basing this on?

Bernd: Compartmentalizing mathematics from judgmen...

2009-11-20T08:17:57.405-05:00

Bernd: Compartmentalizing mathematics from judgment is one way to miss the point. Alas, no concept is an island - everything is related. You were given the option to test more if you wanted to really nail your hypothesis.

photothis: A dataset of 3 is not the problem. This works just as well with a dataset of 2 or 1. This is because the idea is not necessarily to paint a clear picture to provide you with a good chance of guessing the rule. Instead, the dataset is short and vague, causing you to make a judgment and then try out your hypothesis. If you thought up a rule, it meant that a small dataset is still enough for you to postulate on. I'm thinking you just didn't like the purposely misleading nature of the experiment. It is a variant of one thought up by a published psychologist.

Andrew, It sounds like the point Sebastien is mak...

2009-11-19T20:14:21.883-05:00

Andrew,

It sounds like the point Sebastien is making is that this example is entirely arbitrary, and you seem to have bent over backward to trick people into giving the "wrong" answer. You're presenting us with a logic question, and sure, your rule is one possible solution. But the true answer, if somewhere were presented this as a true problem with either be "there could be multiple rules" or more likely "there is not enough information".

This is why datasets of three are not often used, and I'd like to see any scientist in any field claim that results from such a small dataset suffer from confirmation bias before they claim, oh, I don't know, lack of data.

It seems like you wanted to make a very valid point about something that we encounter every day, but then dashed off a hasty and ill-thought out blog post. A better approach might have been to research a few studies . Or maybe just point to the Wikipedia article. This is just not a very convincing display, as we can all tell you're simply trying to be clever here.

I'm confused as to what this has to do with co...

2009-11-19T19:08:11.853-05:00

I'm confused as to what this has to do with confirmation bias.

It says something to me about the natural predilection we have for pattern detection and matching, that is all.

I skipped 3.1 for example. I wasn't a 100% sure but you only permitted one more try. Forgive me, I'm a literalist. And with one more try I couldn't begin to explore the enormous realm of possible rules beyond the alluringly obvious pattern. If you'd invited me to explore at depth I might have struck fired off a few more exploratory efforts to test the accuracy of my rule. With 1 try I simply felt it not worth my time.

That itself was a bias (away from efforts judged to have a probably low return). Because with one simple test (including odd numbers) I would have confirmed my suspicion was invalid.

A demonstration of confirmation bias would be if I simply tried another sequence of even numbers (they would confirm "a subsequent 3 numbers that match the same rule"). But as it stands the exercise revealed nothing to me about confirmation bias, only something about pattern detection and hypothesis testing.

I tested 8, 10, 12. It failed to disprove the hypothesis, and so my confidence rose.

To demonstrate confirmation bias I would need to find myself testing more and more samples that match my null hypothesis exclusively because the minute I test one which doesn't - which is likely to happen inside of a few tries, by my third stab I'd guess at least as I suspect I'd have tried:

8, 10, 12
something like 32, 34, 36
then maybe something like 9, 10, 11
and carried on from there.

Nice try, but I think the test could use some refinement. Confirmation bias certainly exists, that I don't doubt, but this didn't impress me a as a demonstration of it.

JD: You're sending mixed signals. :p

2009-11-19T15:48:33.724-05:00

JD: You're sending mixed signals. :p

I really enjoyed this post BTW.

2009-11-19T15:18:51.407-05:00

I really enjoyed this post BTW.

People like to reinforce their beliefs. Attendanc...

2009-11-19T15:16:34.000-05:00

People like to reinforce their beliefs.

Attendance/participation with simliar minded groups (churches, clubs, anything on the internet, television programs, etc...) demonstrate a similar effect.

They go to the groups because they share beliefs. When their bias is confirmed by others, the belief (no matter how true or false) is strongly enforced.

This is the danger of only reinforcing ideas from the similar minded.

I read this site because I disagree with a lot of what you say.

Errr. You're totally over-thinking this. It is...

2009-11-19T14:51:44.552-05:00

Errr. You're totally over-thinking this. It is very simple, for the purpose of generalization: 1. You see evidence and you create a hypothesis. 2. You seek evidence to confirm that hypothesis. 3. You claim knowledge because your evidence corroborates your hypothesis. 4. You were wrong all along, even though your empirical evidence said you were right.

I guess I'm just not that blown away :) I'...

2009-11-19T14:39:35.230-05:00

I guess I'm just not that blown away :) I'm not "convinced" my rule is in play, I'm just giving you a possible rule. Does it discard other rules? Not really, but you just asked for one :) If you are *pressed* to find a rule, then I don't see how else you can act on that test but by trying to fit a model (a rule) to the data (3 numbers), keeping in mind that it is very very sparse. I think "confirmation bias" sounded a little too fancy to me for what looks like a bad case of good old generalization. Deductive inference goes wrong all the time...

Sébastien: The point of the exercise is to figure ...

2009-11-19T13:21:32.972-05:00

Sébastien: The point of the exercise is to figure out my rule, not your rule. I realize that the numbers I gave could lead you to come up with a whole set of other rules, but that's not the point. The point is that you came up with your rule and then looked for clues to corroborate it. Your rule (thus: interpretation) did not affect the actual rule (thus: reality) even though you might find clues that lead you to believe that your rule is the one in play.

No, 1, 9, 12 follow my rule, which is "Any Nu...

2009-11-19T13:10:55.485-05:00

No, 1, 9, 12 follow my rule, which is "Any Number". What I mean, is that given your 3 numbers, you can come up with several rules that will validate or invalidate another series of 3 numbers. The one that fits 8, 10, 12 is valid given "2 4 6", mine is valid, yours work too. My point was that people will most likely pick 8, 10, 12 because it's natural to favor even/odd/multiple relationships before mathematical ordering in this type of situation. And yes, I agree, it's dangerous to follow such intuition/bias. But in the same way, your choice of rule is biased as well towards ascending order, because that's how you built the problem. The "Any Number" rule though, works. I give you 2, 4, 6. You give me 1, 9, 12. They match that rule, i.e. I could have written a similar post demonstrating that your more-specific rule was biased, since you would have most likely picked "any number in ascending order" before the less-specific "any number".

Sébastien: 1, 9 and 12 do not follow the rule. I&#...

2009-11-19T12:50:56.822-05:00

Sébastien: 1, 9 and 12 do not follow the rule. I'm not sure how you're coming to that conclusion. 1 does not ascend from 8. Also, I'm not sure how you can call a rule "biased" - it is just the rule you're trying to discover and you're using your own biases to figure it out.

"any number in ascending order" is biase...

2009-11-19T12:16:58.264-05:00

"any number in ascending order" is biased the same way. You could have picked: "Any number", it would have worked as well. What are "any numbers" that can appear after 2, 4, 6? 1, 9, 12 for example. They follow the rule, and are subsequent by virtue of the fact that you gave the first 3, and I gave the second set of 3; they are part of the same sequence, and there is no implication of mathematical order in my rule.

Karl: The answer is correct, your numbers just don...

2009-11-19T07:51:55.163-05:00

Karl: The answer is correct, your numbers just don't follow the 3 that were given.

I haven't figured out what the rule is, but it...

2009-11-18T23:46:36.471-05:00

I haven't figured out what the rule is, but it is not what you give as the answer! For example, confirming 3, 333, 343 answers "no".