Clutch Munny
01-07-2007, 05:36 AM
What is it to think critically? Many people seem to believe that thinking critically amounts to thinking logically, or thinking scientifically. It is not my aim in these brief remarks to prove that these conceptions of critical thought are mistaken, though I believe the former is badly mistaken, while the latter is true only in a sense of “scientific” that is not widely adopted. Instead, I want to explain a different understanding of critical thought.
I contend that the best way to master skeptical or critical thought is, first, by understanding the biases that affect human thinking; second, by attending to the behaviour of oneself and others, as a matter of sheer reflexive habit, in order to recognize these biases at work; and third, to leave open always the element of doubt introduced by the knowledge that many such biases are invisible. Cognitive virtue, I contend, is essentially a matter of thinking in accordance with these habits. Acquiring this virtue is a process that is, in its way, as long-term and demanding as the process of learning how to swim. An essay of this sort can no more impart cognitive virtue than it can impart the ability to swim the Butterfly, but I hope to convince you both of the ubiquity of the biases in question, and of the importance of habitual cognitive virtue.
In order to monitor your own reasoning and that of others for factors that can bias it, the most important thing is to have the fullest possible understanding of just what kinds of biases regularly condition our ways of thought. Most of the following remarks are devoted to cataloging and explaining these different biases, and the distinct levels at which they operate. Some of these are obvious and commonsense kinds of cognitive bias, while others may surprise you.
First, however, a note on how I will be using the term "bias". In common parlance this term has universally negative connotations. I will not be using it in that way. A bias, as I use the term, is simply a factor that predisposes one to reach a particular kind of judgment. In itself, this is neither good nor bad. Most often, though, it is good. We could hardly be the thinking machines we are, were we not biased to pare away the huge number of irrelevant judgements and inferences surrounding any daily action or decision, inferences that a more open-minded system would have to crunch through, somehow, in real time. (Which is why it is so incredibly difficult to build computers that mimic our reasoning skills, even though their inference-crunching capacity is, in some obvious respects, vastly superior to our own. The trick is to engineer the right, and enough, overlapping biases to restrict the class of relevant inferences in any particular case. And it turns out to be a very difficult trick indeed.)
Biases get a bad reputation because they tend not to come to our attention unless something has gone wrong with them. In this way the term has acquired the connotation of a pure prejudice, often with overtones of racism, misogyny, and the like. Since I am discussing critical thinking, I too will be disproportionately interested in the cases in which something goes wrong. But understanding those cases, understanding how they go wrong, requires a broader conception of biases as systematic shapers of our thought, normally for better and occasionally for worse.
[B]Biases of situation: contingency and embedding
I’ll begin with a commonsense kind of bias arising from the fact that everyone comes from somewhere. Each person's perspective is in large measure shaped by where they come from, the lifestyle and culture in which they were raised. So why is it typically derogatory to point this out with respect to a single person? In short, why is it an insult to call someone provincial?
Provincialism consists in one's failing to recognize the extent to which one's beliefs and judgements are a function of the contingent way one is embedded in a culture, a community, a history. We like to think that our beliefs and values, and especially those most important in defining us relative to a community, are carefully considered. (This is a facet of Optimistic Self-Assessment, discussed below.) That is, we are tempted to characterize our most definitive beliefs as based upon a weighing of the evidence. In fact this is only rarely the case. To choose just one example, suppose we are told only that some person is religious. We have to make a guess about which religion she practices. Which single piece of information is most predictive for us: her reasoning abilities, or her culture of origin? By far the best predictor of her particular religion is not any measurement of her evidential powers, but is information about the culture in which she was raised.
People born and raised in Egypt tend, on balance, to be Muslim. People born and raised in India tend, on balance, to be Hindu. People born and raised in Mississippi tend, on balance, to be Christian. And yet people born and raised in all three places tend, if questioned, to describe themselves as choosing their beliefs quite deliberately, on the basis of a thorough evaluation of the evidence. Perhaps there is a very restricted sense in which they are right. But at a minimum it is clear that the contingent or accidental property of being born and raised in a particular place powerfully constrains what we understand as a live options in our reasoning.
One important aspect of critical thought, then, is an ongoing sensitivity to the respects in which our thinking may be constrained or biased merely by our upbringing. This does not mean that every habit of thought stemming from our upbringing is suspect. Sometimes the peculiarities of our background provide genuine information that many other people may lack, so that an idiosyncrasy in reasoning based on one's background sometimes indicates nothing other than one's superior grasp of the situation. This can apply to anything from special regional knowledge to knowledge of ethnic or gender experience; indeed, the idea that an underclass has a kind of knowledge invisible to an overclass is an old Marxian notion with a good deal to recommend it. Nevertheless, we ought to be alive to the possibility that when we find something to be obviously right or obviously wrong, we are as much rehearsing the contingencies of our upbringing as evaluating some current circumstance.
[B]Sensory biases: "I saw it with my own eyes!"
The brain is a pattern-recognizer par excellence. It has a deep fondness for patterns of most any sort, and a particular obsession with some specific ones: vertically symmetrical visual patterns, and especially faces, for example . It loves them so much that, deprived of genuine ones, it will make up its own out of ambiguous or outright random inputs. For at least this reason, personal experience and (for further reasons as well) sincere reports thereof do not merit the trust they are often granted. Seeing it with your own two eyes, and hearing it with your own two ears, are only default evidence that it really happened – or at least, that it really happened that way. Many reported events are vastly less probable than that the reporter’s brain gave reality a helping hand of one sort or another. For this does not just happen – it happens non-stop. Your brain is, to some extent, doing it right now.
Characteristic of the most flatfooted philosophical empiricism was the idea that the senses deliver information that, however ambiguous with respect to the true nature of the world, is at least an accurate representation of events at the interface of our bodies with the world. This, it turns out, is a serious oversimplification, one that is more serious still when applied to perceptual faculties in turn. Whether in its far flung sensory outposts or in the unimaginable urban sprawl of the brain, the human nervous system has at most a passing interest in truth or verisimilitude. Critical thinking about sincere claims (one's own or others') to have experienced this or that requires taking seriously the extent to which our brains are prepared to sacrifice accuracy for a kind of local coherence. This tendency begins at the periphery, in the sensory organs themselves and in their informational traffic with systems further downstream.
Take the visual system, for example. Different layers of cells within the eye itself begin the processing of information that continues through much of the brain. The way that cells in one layer are wired up to cells in the next constitutes a bias in favor of, for instance, detecting edges in the visual field. Relatively smooth changes in shading at the surface of the retina are sharpened and exaggerated before the signal even leaves the eye. All in all, this is a fabulously useful property of the eye. But, like most biases, it leaves the eye open to being spoofed about edges on occasion. The point generalizes to other properties of vision, and to other sensory modalities as well. The “raw materials” delivered by the sensory systems are not raw. They represent the effects of structures biased towards a particular style of representation.
On the slightly bigger picture, there is also the question of exactly what sensory information gets sent downstream. Here too the system is not primarily concerned with truth. Or rather, if the role of the sensory systems is to generate a complete and accurate representation of one's environment, the only conclusion to draw is that these systems do their jobs poorly indeed. An accurate representation of one's environment seems instead to be a side effect of the sensory systems’ doing a quite different job: namely, detecting change in one's environment. The mechanisms in play in detecting change are good enough that we normally have a relatively accurate idea of our surroundings. But should our surroundings change undetected, these mechanisms may well fail to update the representation accordingly.
A wide range of fascinating and hilarious experimental results illustrate how astonishingly tolerant we are of large-scale changes in our environment when they are introduced so that we do not notice them when they first occur. For example, sticking with the visual system, most people know that the human eye, even when attending to some point in the visual field, “jumps around” a bit. These jumps are called saccades. They take anywhere from a few hundredths of a second to nearly a full second to occur, and while they are occurring, there is a reduction in the useful signal from the eye. In the lab, psychologists can use head restraints and laser monitors to predict when a saccade will occur, and they can arrange for some change to be introduced to an observed photograph while the saccade is underway. The changed elements can be quite extreme – things appearing or disappearing, the sun turning black, etc. – yet, even though these are plainly visible, if the change itself was not observed, people will not volunteer any report of the anomaly. Using a visual distractor, and counting on natural saccades to induce the effect, one can get something of a feel for this effect outside the lab as well, as these folks have nicely done for us: Movie Demonstrations for article submitted to Nature Aug. 2, 1998 "Change blindness as a result of mudsplashes" (http://nivea.psycho.univ-paris5.fr/Mudsplash/Nature_Supp_Inf/Movies/Movie_List.html).
Experimentally, this phenomenon is called "change blindness"; that is, we tend to be blind to changes we do not actually see occur. This certainly ought to inform our assessment of phenomena like someone's claiming not to have noticed something right in front of his face. The importance of the phenomenon is magnified, however, when we note what can happen when change-blind details are brought to a subject's attention. In these situations, people sometimes simply profess surprise. Sometimes, however, what they do is confabulate an explanation for their having failed to remark on the detail. The crucial thing to understand about confabulations of this sort is that the speaker is not lying; indeed, this is not even lying to oneself, in any useful sense of the phrase. The construction of the explanation occurs at too deep a level for there to be any plausible deliberate intent. Whether it is your own post hoc explanation or someone else's obviously sincere story, we have here just one example of the many respects in which simple sensory functions are biased in a way that ought to modify our judgments of strange or surprising eyewitness claims.
[B]Perceptual biases: The Circus McGurkus
Broadening the picture still further to consider perceptual cases, we see further and stranger prospects for biases to interfere with the accuracy of our representations. The distinction between sensation and perception can be roughly captured in terms of a distinction between inputs that have not, any inputs that have, been conceptualized. (Alternatively, one may think of perceptual outputs as apt to be judged, or to fix the content of beliefs.) Basically, perception is the further processing of sensation; it is not surprising that this further processing provides opportunities for further biases to make themselves felt. A very wide range of biases operate on perception: these range from high-level cognitive factors, such as the power of suggestion, to quite low-level factors stemming from the structure of our perceptual systems and, in some degenerate sense, the power of expectation.
An example of the latter is the McGurk Effect. Here is an example (http://www.freethought-forum.com/video/circusmcgurkus.mov) (Quicktime file) – one I made myself by filming my colleague.
This is an audio-visual illusion, in which altogether unconscious processes reconcile an audio-visual input that defies the brain's equally unconscious expectation of how to interpret speech. The illusion is generated by filming someone looking into a camera and clearly enunciating "gah, gah". The audio portion of the tape is erased, and overtop is recorded someone of the same gender saying "bah, bah". The sound of a hard "g" in English is velar: it is made with the tongue contacting the roof of the mouth fairly near the back of the mouth. This is visible when someone is speaking clearly, though it is not something of which we ordinarily take much note. In contrast, the sound of a "b" is bilabial, made between both lips. That's as far forward as you can get. What makes the McGurk Effect so nifty is the way the brain resolves the tension between the audio sensory input and the the visual sensory input: at the perceptual level – what you’re conscious of – the output is of someone saying, "dah, dah". (In fact for most people it's more of a Greek delta sound, like "thah, thah" .) That is, the brain helpfully splits the difference and locates the sound about halfway between the velar and the bilabial, in the form of the interdental sound "dthah". By closing your eyes or looking away you can change the sound back to “bah”, but knowing the illusion doesn’t help you to hear it correctly as long as you watch the video. (This effect can be achieved with other sounds too.)
It makes perfect sense to call this the result of a bias: the brain is biased toward coherent audio-visual inputs, and when the world does not cooperate by delivering coherent input, the brain is inclined to clean things up in a way that makes perception, at least, fit with the expectation. Notice, though, how low-level a phenomenon this is. It would be a travesty to call this self-deception or wishful thinking; it is really just a matter of how different perceptual subsystems interact. Can this sort of thing also happen without the aid of impish psycholinguists? Certainly. As you attend to one speaker in a crowded room, with multiple background voices of different intensities layered overtop of the single visual input, it would be utterly unsurprising for nature to McGurk you on occasion. You would normally assume that the speaker mispronounced or somesuch. But if the effect resulted in an interpretation that seemed both significant and extremely surprising, it would be important to understand, and hence be able to evaluate, the possibility that your brain had simply made a poor editing decision. You heard it with your own two ears? Sure. So what?
At cognitively higher levels, the effects of bias are again familiar and commonsense: how something tastes can be radically influenced by how one expects it to taste, for instance. I like smoked oysters. Yum. But my gustatory perception will be very different if I am blindfolded, told that I am to be spoon-fed a strawberry milkshake, and am instead given a smoked oyster. Yuck!
Greater potential for cognitive mischief comes in the form of phenomena like conscious priming, in which an explicit expectation can conjure some perception into existence. For example, listen once to this audio sample before continuing to read (http://www.freethought-forum.com/sounds/backwards.mov).
If you’re like most people, that sounded tantalizingly like it was about to become intelligible, but never quite did.
Now I’d like for you to read the following out loud: If you were walking away, and I asked how fast.
Yes, I know, it’s a sentence fragment. Read it again anyhow, if you don’t mind. Then listen to the sample again.
And now you can hear it, right there in the sample. In fact, you almost certainly can’t go back to not hearing it. Your phono-semantic perceptual systems have been given a “top-down” nudge – i.e., a higher-level influence on a low-level process – and that low-level process is not subject to conscious control. This kind of priming can apply in many different contexts to many different systems, which explains why telling someone what evidence to look for tends to increase the probability that they will find it – irrespective of whether it’s there. Priming is also known to work at unconscious levels, so there is no reason to think that one will be well-positioned to judge whether one’s own perception really was biased by priming. It’s just a prospect that responsible reasoning must bear in mind anyhow.
That audio file, by the way, is the first ten or so seconds of the Refreshments’ song “Banditos”, played backwards. With a simple freeware audio editor, you too can construct as many of these little Jedi mind tricks as you please. Even when played backwards human voices have the pitch and timbre of human voices, and this sound gets your brain really worked up, as it looks for something of semantic significance to extract from the signal. All that’s needed is a rough and ready fit with a semantically significant string of sounds, and priming will serve to close the gap. The prime can be thought of as a phono-semantic attractor – a big, powerful magnet in the space of possible interpretations, which pulls in otherwise ambiguous inputs that are only roughly in the phonological neighborhood. The priming strongly biases you to perceive something that isn't really there.
[B]Cognitive biases: You’ve been framed.
There are many weird and wonderful biases that act and interact at the higher cognitive levels. Many of these are related to our remarkably awful factory-preset abilities to reason about probabilities, and most fall under the label of confirmation bias in one respect or another. That is, there are many cognitive mechanisms by which we are led to selectively attend to evidence that supports some hypothesis – indeed, even when we are explicitly sceptical of it!
Do you believe that black cats crossing your path leads to misfortune? Of course not. Nevertheless, just because you are aware of the superstition, if you notice a black cat crossing your path you are much likelier to attend to, and to recall, any immediately subsequent small misfortune. You might bump your arm on a sign, and not remember that evening where the bruise came from; but if you bumped it five seconds after seeing the black cat, you are much likelier to recall where and when it happened, even if only from the irony.
When explicit scepticism is not involved, though, this and similar kinds of confirmation bias can have an enormous effect in skewing one’s sense of the evidential support for some hypothesis or other. These biases are so common as to be notable only by their absence, and can be invisible to large groups of people, to trained experimentalists, and to critical thinking essayists. They work in many ways, along many dimensions: for example, not only are you unlikely to see any significance to phone calls you get from people about whom you did not recently dream gave you a phone call; but you are unlikely even to remember having such a dream unless you receive a call from the person about whom you dreamed. Countervailing cases go unnoticed, unrecalled, while supporting cases are confabulated in perception, noted with greater significance, and recalled with greater probability. These general remarks are all I will say about confirmation biases, though they are the largest sub-topic of this article.
The main example I will consider as cognitive bias is somewhat different, though it too is really a wide class of related examples. It is called the framing effect. Its greatest exploration began with the (only recently Nobel Prize-winning) research of Kahneman and Tversky in the 1970s, work that overturned some fundamental presuppositions of classical economics. What they showed is that the way people judge information is strongly influenced by the way the information is presented. This much may seem obvious in the age of media spin, but the contexts in question were ones of careful consideration, with university students having time to weigh situations pretty thoroughly. So the biasing effects of the way information was framed were still quite surprising. For instance, a scenario was described on which 600 people are sick, and there is only enough medication either to give an under-dose to everyone, in which case there is a 2/3 chance that everyone will die, or to give a full dose to 200 people, in which case they will all certainly live and everyone else will certainly die. (From the perspective of classical decision theory these two scenarios are equivalent, though that doesn’t matter right now.) It turned out that subjects’ response to the latter action varied wildly depending on how it was described. Framed in terms like, “Exactly 200 people will definitely be saved”, this option was strongly viewed as an acceptable course of action. Framed in terms like, “Exactly 400 people will definitely be lost”, it was equally strongly viewed as an unacceptable course of action. Of course, given the scenario presented to the subjects, these two descriptions convey precisely the same information. So the difference is not in the information, but in the way it is framed; hence, “framing effect”. Even smart people in deliberative contexts are extremely sensitive to this.
Many other cognitive biases are either innate or are socially inculcated in less explicit ways. Optimistic self-assessment, also known as the Lake Woebegon Effect (from Garrison Keillor’s fictional town in which all the children are above average), is a bias to judge oneself favourably relative to one’s cohort, with respect to a limited class of virtues. (The tendency to produce more realistic self-assessments seems to be statistically correlated with chronic depression. Make of that what you will.) That is, people do not tend to overrate themselves on measurable qualities like mathematical ability, but on ill-defined virtues like leadership or honesty our self-assessments tend to be impossibly high. And “impossible” here means mathematically impossible: one massive survey (over a million high-school seniors) found that 100% of respondents ranked themselves in the top 10% of their peers for ability to get along with others, while a full 25% ranked themselves in top 1%. As Mark Knopfler sings about two men claiming to be Jesus: one of them must be wrong!
[B]Biases of memory: Who controls the present, controls the past
No discovery of experimental psychology is likely to be as potentially destabilizing to social and legal institutions as the recent discoveries about the frailty of human memory. One might simply note that all cognitive and perceptual biases noted above (and all the others too) apply to memory in all its guises: short-term, long-term, episodic, and what haven’t you. This is roughly true, but it’s worth getting a feel for just how significant this fact is.
Framing effects, to continue the example just canvassed, are known to have a powerful effect on the contents of memory – contents that appear to subjects to have all the detail and vivacity of the most reliable memory. In one famous experiment, two groups were shown video of one car running into another; one group was asked to estimate the speed of impact when one car hit the other, while the second group was asked to estimate the speed when one car smashed into the other. The second group systematically recalled the crash as occurring at a higher speed. While perhaps predictable, this effect in itself undermines the idea of memory as having its content fixed independently of the means of recall. Several days later, however, both groups were asked whether broken glass was visible in the video. Even though no broken glass was visible, members of the second “smashed” group had a significantly higher incidence of reporting a clear recollection of broken glass. They could see it in their minds, as clear as day.
Another well-known experiment involved inserting fictitious memories into the recall of young children and teens. (Innocuous memories, it should be noted!) These too were eventually recalled with a detail and phenomenology that made them indistinguishable from veridical, or at least veridically-based, memories. What exotic methods were employed to effect such a remarkable result?
It was this: repetition. Nothing more. In short, exactly the sort of thing that family and friends’ storytelling is apt present to any of us over many years. (Bear in mind that repetition is only a sufficient condition for false memories; it is unlikely to be necessary as well.)
Of course, certain constraints apply. It’s more or less impossible to convince a normal subject that she’s been to the Moon, since memory-fixation processes interact with general knowledge quite robustly. But within those broad constraints, the interaction with general knowledge is far from perfect; in one experiment, a subject was successfully implanted with the memory of having met Bugs Bunny during a childhood trip to Disneyland – even though it’s hardly a state secret that Bugs Bunny is not a Disney character.
The implications of such research for “eyewitness testimony” in legal contexts are hard to overestimate, for reasons that are not difficult to see. For our more humble purposes, however, such institutional concerns are set aside in favour of a deep personal caution towards our own memories, when they are in tension with the balance of independent evidence, and with the testimony of others, however sincere, when similar conditions obtain.
[B]Biases of social interaction
A crucial thing to understand about the kinds of biases we have been considering is how they interact with and reinforce one another. This is particularly true at the level of “social cognition”. This term picks out at least two different kinds of phenomena: cognition about social situations in particular, and cognition (about whatever) that has emergent effects at the social level.
As an example of a bias in the first sort of social cognition, consider the tendency so basic to our social interactions that psychologists flatter it with the label, The Fundamental Attribution Error. This denotes our systematic tendency to read personality traits off single events, actions, or outcomes of another person. Seeing a stranger with obviously unwashed hair, for example, we are inclined to attribute poor hygiene or a lack of care about personal appearance, rather than considering the vast number of respects in which a single such event could be random, situational, accidental, or otherwise explained by circumstances rather than the character of the person. The old saw about first impressions is right on the money as a description of our attribution tendencies. Such inferences are largely unwarranted, however. (A sense of one’s ability to make accurate judgements from first impressions would probably be a fine case study in confirmation bias!)
The second sort of social cognition bias is more complex and systemic, being an emergent effect. Independent and isolated cognitive biases: sensitivity to repetition; emotional attachment to the admiration, however conversationally fleeting, of having a Good Story; sensitivity to pressure to make one’s utterances as relevant as possible – these can interact in social contexts to bias things like the way one chooses to describe an event. The cumulative effects of even very small degrees of such description-biasing can become very large, very fast. In aggregate the effects are known as leveling and sharpening. Leveling is the phenomenon of mitigating, explanatory, and situating details dropping out of some story as it gets told and retold, while sharpening denotes the tendency of the key, salient, or surprising details to become exaggerated. Sometimes these processes are effected by outright lying, but they certainly need not be. Only a few re-tellings are required in order for a story to go from being a mildly surprising account of a familiar event, to being an account of something miraculous, astounding, another entry in the catalogue of Mysteries of the Unexplained. And, of course, at each stage the teller speaks with the compelling air of entitlement belonging to one who has consciously made no significant change – perhaps no change whatever – to the story, and who inherited the story from someone who spoke with an equally obvious sincerity and entitlement.
The fact is, the cumulative effects of our brains in social communicative contexts make our individual brains look like sacred keepers of truth by comparison. This is because social communicative contexts are primarily oriented towards other things – in fact, damned near anything other – than the promulgation of strict truths. They are about entertainment, communal belonging, empathy, mating, bullying, and a host of other goals, with the group commitment to canons of truth being thin indeed.
This is, to be sure, a Good Thing. I am not advocating that we all become Science Officer Spock; there is a lot to be said for bullshitting, and I am personally an enthusiastic devotee of the practice. The point, rather, is to bear in mind that information received through social channels – which amounts to a huge proportion of the information we receive, whether we note it or not – is inherently dubious. Each such piece of information arrives at the end of a chain of transmission-events that were (usually) unconstrained by any useful standard of accuracy or warrant.
Indeed, not only is there no requirement or social pressure in favour of strict accuracy in social transmission, there are powerful norms against this in virtually all social contexts. There is a word for someone who, in the normal run of conversation, is prone to saying, “What’s your evidence for that?” The word is asshole. And quite properly so. Who wants to be justifying themselves all the time? “Always exaggerate,” ran an old New Yorker cartoon. “It makes life more interesting.” So true. The important thing is to find ways of policing one’s own credulity in such contexts, to be aware that even though one would never consciously describe such exchanges primarily as informative, still the effects of repetition and testimonial bias can be powerful.
This policing is exhaustingly difficult if done consciously and deliberately. Acquiring the habit of doing it automatically, almost unconsciously, is, like any good habit, hard work that begins with the exhausting part. And because few of us have any such habits, and none of us has enough, we find exactly what we should expect as a result: A widespread serious attachment to unfecund, dubious, and occasionally plain old crazy beliefs – originating in many cases from biased perception, shored up by individual confirmation biases, and both spread and magnified by biases in social cognition.
[B]Closing thoughts: Media and other evil people
The main fault with this article – setting aside lack of references, no bibliography, a healthy degree of leveling and sharpening (caveat lector!), and sundry infelicities of prose – is its failure to consider the way that operations of the media shape and are shaped by the sum of these biases. I will simply close by pointing out that all of these facts are familiar to advertisers, political parties, and others who are, quite literally, competitors for control of your mind. I have focused on the vagaries of human reasoning as it occurs apart from deliberate culture-wide attempts to inculcate false or unwarranted beliefs. But if there is any overriding prudential reason to take seriously the project of finding out how your own mind works, it is the certain knowledge that this information is being used on you, with no regard for your best interests, by people and institutions to whom you are nothing more than an available vote or an open wallet.
Me, I study it because it’s some of the most interesting stuff I’ve ever encountered.
*Thanks and more thanks to livius drusus, for technical know-how, keen editing, and much initiative.*
I contend that the best way to master skeptical or critical thought is, first, by understanding the biases that affect human thinking; second, by attending to the behaviour of oneself and others, as a matter of sheer reflexive habit, in order to recognize these biases at work; and third, to leave open always the element of doubt introduced by the knowledge that many such biases are invisible. Cognitive virtue, I contend, is essentially a matter of thinking in accordance with these habits. Acquiring this virtue is a process that is, in its way, as long-term and demanding as the process of learning how to swim. An essay of this sort can no more impart cognitive virtue than it can impart the ability to swim the Butterfly, but I hope to convince you both of the ubiquity of the biases in question, and of the importance of habitual cognitive virtue.
In order to monitor your own reasoning and that of others for factors that can bias it, the most important thing is to have the fullest possible understanding of just what kinds of biases regularly condition our ways of thought. Most of the following remarks are devoted to cataloging and explaining these different biases, and the distinct levels at which they operate. Some of these are obvious and commonsense kinds of cognitive bias, while others may surprise you.
First, however, a note on how I will be using the term "bias". In common parlance this term has universally negative connotations. I will not be using it in that way. A bias, as I use the term, is simply a factor that predisposes one to reach a particular kind of judgment. In itself, this is neither good nor bad. Most often, though, it is good. We could hardly be the thinking machines we are, were we not biased to pare away the huge number of irrelevant judgements and inferences surrounding any daily action or decision, inferences that a more open-minded system would have to crunch through, somehow, in real time. (Which is why it is so incredibly difficult to build computers that mimic our reasoning skills, even though their inference-crunching capacity is, in some obvious respects, vastly superior to our own. The trick is to engineer the right, and enough, overlapping biases to restrict the class of relevant inferences in any particular case. And it turns out to be a very difficult trick indeed.)
Biases get a bad reputation because they tend not to come to our attention unless something has gone wrong with them. In this way the term has acquired the connotation of a pure prejudice, often with overtones of racism, misogyny, and the like. Since I am discussing critical thinking, I too will be disproportionately interested in the cases in which something goes wrong. But understanding those cases, understanding how they go wrong, requires a broader conception of biases as systematic shapers of our thought, normally for better and occasionally for worse.
[B]Biases of situation: contingency and embedding
I’ll begin with a commonsense kind of bias arising from the fact that everyone comes from somewhere. Each person's perspective is in large measure shaped by where they come from, the lifestyle and culture in which they were raised. So why is it typically derogatory to point this out with respect to a single person? In short, why is it an insult to call someone provincial?
Provincialism consists in one's failing to recognize the extent to which one's beliefs and judgements are a function of the contingent way one is embedded in a culture, a community, a history. We like to think that our beliefs and values, and especially those most important in defining us relative to a community, are carefully considered. (This is a facet of Optimistic Self-Assessment, discussed below.) That is, we are tempted to characterize our most definitive beliefs as based upon a weighing of the evidence. In fact this is only rarely the case. To choose just one example, suppose we are told only that some person is religious. We have to make a guess about which religion she practices. Which single piece of information is most predictive for us: her reasoning abilities, or her culture of origin? By far the best predictor of her particular religion is not any measurement of her evidential powers, but is information about the culture in which she was raised.
People born and raised in Egypt tend, on balance, to be Muslim. People born and raised in India tend, on balance, to be Hindu. People born and raised in Mississippi tend, on balance, to be Christian. And yet people born and raised in all three places tend, if questioned, to describe themselves as choosing their beliefs quite deliberately, on the basis of a thorough evaluation of the evidence. Perhaps there is a very restricted sense in which they are right. But at a minimum it is clear that the contingent or accidental property of being born and raised in a particular place powerfully constrains what we understand as a live options in our reasoning.
One important aspect of critical thought, then, is an ongoing sensitivity to the respects in which our thinking may be constrained or biased merely by our upbringing. This does not mean that every habit of thought stemming from our upbringing is suspect. Sometimes the peculiarities of our background provide genuine information that many other people may lack, so that an idiosyncrasy in reasoning based on one's background sometimes indicates nothing other than one's superior grasp of the situation. This can apply to anything from special regional knowledge to knowledge of ethnic or gender experience; indeed, the idea that an underclass has a kind of knowledge invisible to an overclass is an old Marxian notion with a good deal to recommend it. Nevertheless, we ought to be alive to the possibility that when we find something to be obviously right or obviously wrong, we are as much rehearsing the contingencies of our upbringing as evaluating some current circumstance.
[B]Sensory biases: "I saw it with my own eyes!"
The brain is a pattern-recognizer par excellence. It has a deep fondness for patterns of most any sort, and a particular obsession with some specific ones: vertically symmetrical visual patterns, and especially faces, for example . It loves them so much that, deprived of genuine ones, it will make up its own out of ambiguous or outright random inputs. For at least this reason, personal experience and (for further reasons as well) sincere reports thereof do not merit the trust they are often granted. Seeing it with your own two eyes, and hearing it with your own two ears, are only default evidence that it really happened – or at least, that it really happened that way. Many reported events are vastly less probable than that the reporter’s brain gave reality a helping hand of one sort or another. For this does not just happen – it happens non-stop. Your brain is, to some extent, doing it right now.
Characteristic of the most flatfooted philosophical empiricism was the idea that the senses deliver information that, however ambiguous with respect to the true nature of the world, is at least an accurate representation of events at the interface of our bodies with the world. This, it turns out, is a serious oversimplification, one that is more serious still when applied to perceptual faculties in turn. Whether in its far flung sensory outposts or in the unimaginable urban sprawl of the brain, the human nervous system has at most a passing interest in truth or verisimilitude. Critical thinking about sincere claims (one's own or others') to have experienced this or that requires taking seriously the extent to which our brains are prepared to sacrifice accuracy for a kind of local coherence. This tendency begins at the periphery, in the sensory organs themselves and in their informational traffic with systems further downstream.
Take the visual system, for example. Different layers of cells within the eye itself begin the processing of information that continues through much of the brain. The way that cells in one layer are wired up to cells in the next constitutes a bias in favor of, for instance, detecting edges in the visual field. Relatively smooth changes in shading at the surface of the retina are sharpened and exaggerated before the signal even leaves the eye. All in all, this is a fabulously useful property of the eye. But, like most biases, it leaves the eye open to being spoofed about edges on occasion. The point generalizes to other properties of vision, and to other sensory modalities as well. The “raw materials” delivered by the sensory systems are not raw. They represent the effects of structures biased towards a particular style of representation.
On the slightly bigger picture, there is also the question of exactly what sensory information gets sent downstream. Here too the system is not primarily concerned with truth. Or rather, if the role of the sensory systems is to generate a complete and accurate representation of one's environment, the only conclusion to draw is that these systems do their jobs poorly indeed. An accurate representation of one's environment seems instead to be a side effect of the sensory systems’ doing a quite different job: namely, detecting change in one's environment. The mechanisms in play in detecting change are good enough that we normally have a relatively accurate idea of our surroundings. But should our surroundings change undetected, these mechanisms may well fail to update the representation accordingly.
A wide range of fascinating and hilarious experimental results illustrate how astonishingly tolerant we are of large-scale changes in our environment when they are introduced so that we do not notice them when they first occur. For example, sticking with the visual system, most people know that the human eye, even when attending to some point in the visual field, “jumps around” a bit. These jumps are called saccades. They take anywhere from a few hundredths of a second to nearly a full second to occur, and while they are occurring, there is a reduction in the useful signal from the eye. In the lab, psychologists can use head restraints and laser monitors to predict when a saccade will occur, and they can arrange for some change to be introduced to an observed photograph while the saccade is underway. The changed elements can be quite extreme – things appearing or disappearing, the sun turning black, etc. – yet, even though these are plainly visible, if the change itself was not observed, people will not volunteer any report of the anomaly. Using a visual distractor, and counting on natural saccades to induce the effect, one can get something of a feel for this effect outside the lab as well, as these folks have nicely done for us: Movie Demonstrations for article submitted to Nature Aug. 2, 1998 "Change blindness as a result of mudsplashes" (http://nivea.psycho.univ-paris5.fr/Mudsplash/Nature_Supp_Inf/Movies/Movie_List.html).
Experimentally, this phenomenon is called "change blindness"; that is, we tend to be blind to changes we do not actually see occur. This certainly ought to inform our assessment of phenomena like someone's claiming not to have noticed something right in front of his face. The importance of the phenomenon is magnified, however, when we note what can happen when change-blind details are brought to a subject's attention. In these situations, people sometimes simply profess surprise. Sometimes, however, what they do is confabulate an explanation for their having failed to remark on the detail. The crucial thing to understand about confabulations of this sort is that the speaker is not lying; indeed, this is not even lying to oneself, in any useful sense of the phrase. The construction of the explanation occurs at too deep a level for there to be any plausible deliberate intent. Whether it is your own post hoc explanation or someone else's obviously sincere story, we have here just one example of the many respects in which simple sensory functions are biased in a way that ought to modify our judgments of strange or surprising eyewitness claims.
[B]Perceptual biases: The Circus McGurkus
Broadening the picture still further to consider perceptual cases, we see further and stranger prospects for biases to interfere with the accuracy of our representations. The distinction between sensation and perception can be roughly captured in terms of a distinction between inputs that have not, any inputs that have, been conceptualized. (Alternatively, one may think of perceptual outputs as apt to be judged, or to fix the content of beliefs.) Basically, perception is the further processing of sensation; it is not surprising that this further processing provides opportunities for further biases to make themselves felt. A very wide range of biases operate on perception: these range from high-level cognitive factors, such as the power of suggestion, to quite low-level factors stemming from the structure of our perceptual systems and, in some degenerate sense, the power of expectation.
An example of the latter is the McGurk Effect. Here is an example (http://www.freethought-forum.com/video/circusmcgurkus.mov) (Quicktime file) – one I made myself by filming my colleague.
This is an audio-visual illusion, in which altogether unconscious processes reconcile an audio-visual input that defies the brain's equally unconscious expectation of how to interpret speech. The illusion is generated by filming someone looking into a camera and clearly enunciating "gah, gah". The audio portion of the tape is erased, and overtop is recorded someone of the same gender saying "bah, bah". The sound of a hard "g" in English is velar: it is made with the tongue contacting the roof of the mouth fairly near the back of the mouth. This is visible when someone is speaking clearly, though it is not something of which we ordinarily take much note. In contrast, the sound of a "b" is bilabial, made between both lips. That's as far forward as you can get. What makes the McGurk Effect so nifty is the way the brain resolves the tension between the audio sensory input and the the visual sensory input: at the perceptual level – what you’re conscious of – the output is of someone saying, "dah, dah". (In fact for most people it's more of a Greek delta sound, like "thah, thah" .) That is, the brain helpfully splits the difference and locates the sound about halfway between the velar and the bilabial, in the form of the interdental sound "dthah". By closing your eyes or looking away you can change the sound back to “bah”, but knowing the illusion doesn’t help you to hear it correctly as long as you watch the video. (This effect can be achieved with other sounds too.)
It makes perfect sense to call this the result of a bias: the brain is biased toward coherent audio-visual inputs, and when the world does not cooperate by delivering coherent input, the brain is inclined to clean things up in a way that makes perception, at least, fit with the expectation. Notice, though, how low-level a phenomenon this is. It would be a travesty to call this self-deception or wishful thinking; it is really just a matter of how different perceptual subsystems interact. Can this sort of thing also happen without the aid of impish psycholinguists? Certainly. As you attend to one speaker in a crowded room, with multiple background voices of different intensities layered overtop of the single visual input, it would be utterly unsurprising for nature to McGurk you on occasion. You would normally assume that the speaker mispronounced or somesuch. But if the effect resulted in an interpretation that seemed both significant and extremely surprising, it would be important to understand, and hence be able to evaluate, the possibility that your brain had simply made a poor editing decision. You heard it with your own two ears? Sure. So what?
At cognitively higher levels, the effects of bias are again familiar and commonsense: how something tastes can be radically influenced by how one expects it to taste, for instance. I like smoked oysters. Yum. But my gustatory perception will be very different if I am blindfolded, told that I am to be spoon-fed a strawberry milkshake, and am instead given a smoked oyster. Yuck!
Greater potential for cognitive mischief comes in the form of phenomena like conscious priming, in which an explicit expectation can conjure some perception into existence. For example, listen once to this audio sample before continuing to read (http://www.freethought-forum.com/sounds/backwards.mov).
If you’re like most people, that sounded tantalizingly like it was about to become intelligible, but never quite did.
Now I’d like for you to read the following out loud: If you were walking away, and I asked how fast.
Yes, I know, it’s a sentence fragment. Read it again anyhow, if you don’t mind. Then listen to the sample again.
And now you can hear it, right there in the sample. In fact, you almost certainly can’t go back to not hearing it. Your phono-semantic perceptual systems have been given a “top-down” nudge – i.e., a higher-level influence on a low-level process – and that low-level process is not subject to conscious control. This kind of priming can apply in many different contexts to many different systems, which explains why telling someone what evidence to look for tends to increase the probability that they will find it – irrespective of whether it’s there. Priming is also known to work at unconscious levels, so there is no reason to think that one will be well-positioned to judge whether one’s own perception really was biased by priming. It’s just a prospect that responsible reasoning must bear in mind anyhow.
That audio file, by the way, is the first ten or so seconds of the Refreshments’ song “Banditos”, played backwards. With a simple freeware audio editor, you too can construct as many of these little Jedi mind tricks as you please. Even when played backwards human voices have the pitch and timbre of human voices, and this sound gets your brain really worked up, as it looks for something of semantic significance to extract from the signal. All that’s needed is a rough and ready fit with a semantically significant string of sounds, and priming will serve to close the gap. The prime can be thought of as a phono-semantic attractor – a big, powerful magnet in the space of possible interpretations, which pulls in otherwise ambiguous inputs that are only roughly in the phonological neighborhood. The priming strongly biases you to perceive something that isn't really there.
[B]Cognitive biases: You’ve been framed.
There are many weird and wonderful biases that act and interact at the higher cognitive levels. Many of these are related to our remarkably awful factory-preset abilities to reason about probabilities, and most fall under the label of confirmation bias in one respect or another. That is, there are many cognitive mechanisms by which we are led to selectively attend to evidence that supports some hypothesis – indeed, even when we are explicitly sceptical of it!
Do you believe that black cats crossing your path leads to misfortune? Of course not. Nevertheless, just because you are aware of the superstition, if you notice a black cat crossing your path you are much likelier to attend to, and to recall, any immediately subsequent small misfortune. You might bump your arm on a sign, and not remember that evening where the bruise came from; but if you bumped it five seconds after seeing the black cat, you are much likelier to recall where and when it happened, even if only from the irony.
When explicit scepticism is not involved, though, this and similar kinds of confirmation bias can have an enormous effect in skewing one’s sense of the evidential support for some hypothesis or other. These biases are so common as to be notable only by their absence, and can be invisible to large groups of people, to trained experimentalists, and to critical thinking essayists. They work in many ways, along many dimensions: for example, not only are you unlikely to see any significance to phone calls you get from people about whom you did not recently dream gave you a phone call; but you are unlikely even to remember having such a dream unless you receive a call from the person about whom you dreamed. Countervailing cases go unnoticed, unrecalled, while supporting cases are confabulated in perception, noted with greater significance, and recalled with greater probability. These general remarks are all I will say about confirmation biases, though they are the largest sub-topic of this article.
The main example I will consider as cognitive bias is somewhat different, though it too is really a wide class of related examples. It is called the framing effect. Its greatest exploration began with the (only recently Nobel Prize-winning) research of Kahneman and Tversky in the 1970s, work that overturned some fundamental presuppositions of classical economics. What they showed is that the way people judge information is strongly influenced by the way the information is presented. This much may seem obvious in the age of media spin, but the contexts in question were ones of careful consideration, with university students having time to weigh situations pretty thoroughly. So the biasing effects of the way information was framed were still quite surprising. For instance, a scenario was described on which 600 people are sick, and there is only enough medication either to give an under-dose to everyone, in which case there is a 2/3 chance that everyone will die, or to give a full dose to 200 people, in which case they will all certainly live and everyone else will certainly die. (From the perspective of classical decision theory these two scenarios are equivalent, though that doesn’t matter right now.) It turned out that subjects’ response to the latter action varied wildly depending on how it was described. Framed in terms like, “Exactly 200 people will definitely be saved”, this option was strongly viewed as an acceptable course of action. Framed in terms like, “Exactly 400 people will definitely be lost”, it was equally strongly viewed as an unacceptable course of action. Of course, given the scenario presented to the subjects, these two descriptions convey precisely the same information. So the difference is not in the information, but in the way it is framed; hence, “framing effect”. Even smart people in deliberative contexts are extremely sensitive to this.
Many other cognitive biases are either innate or are socially inculcated in less explicit ways. Optimistic self-assessment, also known as the Lake Woebegon Effect (from Garrison Keillor’s fictional town in which all the children are above average), is a bias to judge oneself favourably relative to one’s cohort, with respect to a limited class of virtues. (The tendency to produce more realistic self-assessments seems to be statistically correlated with chronic depression. Make of that what you will.) That is, people do not tend to overrate themselves on measurable qualities like mathematical ability, but on ill-defined virtues like leadership or honesty our self-assessments tend to be impossibly high. And “impossible” here means mathematically impossible: one massive survey (over a million high-school seniors) found that 100% of respondents ranked themselves in the top 10% of their peers for ability to get along with others, while a full 25% ranked themselves in top 1%. As Mark Knopfler sings about two men claiming to be Jesus: one of them must be wrong!
[B]Biases of memory: Who controls the present, controls the past
No discovery of experimental psychology is likely to be as potentially destabilizing to social and legal institutions as the recent discoveries about the frailty of human memory. One might simply note that all cognitive and perceptual biases noted above (and all the others too) apply to memory in all its guises: short-term, long-term, episodic, and what haven’t you. This is roughly true, but it’s worth getting a feel for just how significant this fact is.
Framing effects, to continue the example just canvassed, are known to have a powerful effect on the contents of memory – contents that appear to subjects to have all the detail and vivacity of the most reliable memory. In one famous experiment, two groups were shown video of one car running into another; one group was asked to estimate the speed of impact when one car hit the other, while the second group was asked to estimate the speed when one car smashed into the other. The second group systematically recalled the crash as occurring at a higher speed. While perhaps predictable, this effect in itself undermines the idea of memory as having its content fixed independently of the means of recall. Several days later, however, both groups were asked whether broken glass was visible in the video. Even though no broken glass was visible, members of the second “smashed” group had a significantly higher incidence of reporting a clear recollection of broken glass. They could see it in their minds, as clear as day.
Another well-known experiment involved inserting fictitious memories into the recall of young children and teens. (Innocuous memories, it should be noted!) These too were eventually recalled with a detail and phenomenology that made them indistinguishable from veridical, or at least veridically-based, memories. What exotic methods were employed to effect such a remarkable result?
It was this: repetition. Nothing more. In short, exactly the sort of thing that family and friends’ storytelling is apt present to any of us over many years. (Bear in mind that repetition is only a sufficient condition for false memories; it is unlikely to be necessary as well.)
Of course, certain constraints apply. It’s more or less impossible to convince a normal subject that she’s been to the Moon, since memory-fixation processes interact with general knowledge quite robustly. But within those broad constraints, the interaction with general knowledge is far from perfect; in one experiment, a subject was successfully implanted with the memory of having met Bugs Bunny during a childhood trip to Disneyland – even though it’s hardly a state secret that Bugs Bunny is not a Disney character.
The implications of such research for “eyewitness testimony” in legal contexts are hard to overestimate, for reasons that are not difficult to see. For our more humble purposes, however, such institutional concerns are set aside in favour of a deep personal caution towards our own memories, when they are in tension with the balance of independent evidence, and with the testimony of others, however sincere, when similar conditions obtain.
[B]Biases of social interaction
A crucial thing to understand about the kinds of biases we have been considering is how they interact with and reinforce one another. This is particularly true at the level of “social cognition”. This term picks out at least two different kinds of phenomena: cognition about social situations in particular, and cognition (about whatever) that has emergent effects at the social level.
As an example of a bias in the first sort of social cognition, consider the tendency so basic to our social interactions that psychologists flatter it with the label, The Fundamental Attribution Error. This denotes our systematic tendency to read personality traits off single events, actions, or outcomes of another person. Seeing a stranger with obviously unwashed hair, for example, we are inclined to attribute poor hygiene or a lack of care about personal appearance, rather than considering the vast number of respects in which a single such event could be random, situational, accidental, or otherwise explained by circumstances rather than the character of the person. The old saw about first impressions is right on the money as a description of our attribution tendencies. Such inferences are largely unwarranted, however. (A sense of one’s ability to make accurate judgements from first impressions would probably be a fine case study in confirmation bias!)
The second sort of social cognition bias is more complex and systemic, being an emergent effect. Independent and isolated cognitive biases: sensitivity to repetition; emotional attachment to the admiration, however conversationally fleeting, of having a Good Story; sensitivity to pressure to make one’s utterances as relevant as possible – these can interact in social contexts to bias things like the way one chooses to describe an event. The cumulative effects of even very small degrees of such description-biasing can become very large, very fast. In aggregate the effects are known as leveling and sharpening. Leveling is the phenomenon of mitigating, explanatory, and situating details dropping out of some story as it gets told and retold, while sharpening denotes the tendency of the key, salient, or surprising details to become exaggerated. Sometimes these processes are effected by outright lying, but they certainly need not be. Only a few re-tellings are required in order for a story to go from being a mildly surprising account of a familiar event, to being an account of something miraculous, astounding, another entry in the catalogue of Mysteries of the Unexplained. And, of course, at each stage the teller speaks with the compelling air of entitlement belonging to one who has consciously made no significant change – perhaps no change whatever – to the story, and who inherited the story from someone who spoke with an equally obvious sincerity and entitlement.
The fact is, the cumulative effects of our brains in social communicative contexts make our individual brains look like sacred keepers of truth by comparison. This is because social communicative contexts are primarily oriented towards other things – in fact, damned near anything other – than the promulgation of strict truths. They are about entertainment, communal belonging, empathy, mating, bullying, and a host of other goals, with the group commitment to canons of truth being thin indeed.
This is, to be sure, a Good Thing. I am not advocating that we all become Science Officer Spock; there is a lot to be said for bullshitting, and I am personally an enthusiastic devotee of the practice. The point, rather, is to bear in mind that information received through social channels – which amounts to a huge proportion of the information we receive, whether we note it or not – is inherently dubious. Each such piece of information arrives at the end of a chain of transmission-events that were (usually) unconstrained by any useful standard of accuracy or warrant.
Indeed, not only is there no requirement or social pressure in favour of strict accuracy in social transmission, there are powerful norms against this in virtually all social contexts. There is a word for someone who, in the normal run of conversation, is prone to saying, “What’s your evidence for that?” The word is asshole. And quite properly so. Who wants to be justifying themselves all the time? “Always exaggerate,” ran an old New Yorker cartoon. “It makes life more interesting.” So true. The important thing is to find ways of policing one’s own credulity in such contexts, to be aware that even though one would never consciously describe such exchanges primarily as informative, still the effects of repetition and testimonial bias can be powerful.
This policing is exhaustingly difficult if done consciously and deliberately. Acquiring the habit of doing it automatically, almost unconsciously, is, like any good habit, hard work that begins with the exhausting part. And because few of us have any such habits, and none of us has enough, we find exactly what we should expect as a result: A widespread serious attachment to unfecund, dubious, and occasionally plain old crazy beliefs – originating in many cases from biased perception, shored up by individual confirmation biases, and both spread and magnified by biases in social cognition.
[B]Closing thoughts: Media and other evil people
The main fault with this article – setting aside lack of references, no bibliography, a healthy degree of leveling and sharpening (caveat lector!), and sundry infelicities of prose – is its failure to consider the way that operations of the media shape and are shaped by the sum of these biases. I will simply close by pointing out that all of these facts are familiar to advertisers, political parties, and others who are, quite literally, competitors for control of your mind. I have focused on the vagaries of human reasoning as it occurs apart from deliberate culture-wide attempts to inculcate false or unwarranted beliefs. But if there is any overriding prudential reason to take seriously the project of finding out how your own mind works, it is the certain knowledge that this information is being used on you, with no regard for your best interests, by people and institutions to whom you are nothing more than an available vote or an open wallet.
Me, I study it because it’s some of the most interesting stuff I’ve ever encountered.
*Thanks and more thanks to livius drusus, for technical know-how, keen editing, and much initiative.*