Post #104: Confidence Issues

24 March 2024

“In an enormous number of cases the part of wisdom and courage is to believe what is in the line of your needs: refuse to believe, and you shall indeed be right, for you shall irretrievably perish; but believe, and again you shall be right, for you shall save yourself. You make one or the other of two possible universes true by your trust or mistrust.”

—William James, “Is Life Worth Living?”*

     The blessings of confidence do not, among Americans especially, seem to call for much discussion, they look so very straightforward: obviously you would want to be as self-assured as possible, goes the refrain, since if you didn’t believe in your own worth and prospects, then who else would do it for you? Whether in the domain of dating, of applying for a job, or of carrying out a project, it can look pretty much a truism that confidence is the ticket to success, its absence, a sure precursor to failure.

     Indeed so pervasive is this bedrock belief in the power and legitimacy of confidence that it does not even have a widely known antonym, though there is certainly one on the books: as con-fidence is faith in oneself and one’s purposes, so dif-fidence is a lack thereof or deficiency therein. In the oeuvre of Thomas Hobbes, the greatest champion of diffidence, it makes its appearance as the twin of fear, which in turn is understood not as the bane of our human condition, but as an indispensable lever for keeping us from tearing each other to pieces—via the dread of the retribution we would invite by our transgressions in a social world reasonably pacified and well-ordered.

     In the sophisticated version of the argument for mustering our mental resources, as advanced by William James, I find much to agree with and admire. But everything in James turns on his insistence that we must make our home in a world of maybes, whatever we do: “So far as man stands for anything, his entire vital function may be said to have to deal with maybes. Not a victory is gained, not a deed of faithfulness or courage is done, except upon a maybe; not a service, not a sally of generosity, not a scientific exploration or experiment, that may not be a mistake. It is only by risking our persons from one hour to another that we live at all. And often enough our faith beforehand in an uncertified result is the only thing that makes the result come true.” Thus James’s vision amounts to a hybrid take on diffidence, as does David Hume’s at a higher level still, where the recognition that we can know so little is supplemented with allowances made for putting our trust, faute de mieux, in our ordinary perceptions, or our faith in maybes.

     The everyday overconfidence that I am mostly concerned with here may be a distant relative of Jamesian pragmatism, but with all the compelling subtlety subtracted for easier chewing and digestion, to say nothing of what comes next. The trouble with the everyday variety, to put it bluntly, is that it is almost invariably mistaken as to the facts—that is to say, it is usually held upon blatantly inadequate foundations, or even in utter disregard of the evidence. Hence it borders on cliché how practically all drivers, even those far from skilled when put to the test, take themselves to be at least above average performers behind the wheel, if not outright masters of the road. Founders of small businesses invariably believe themselves, in a similar spirit of self-delusion, to be overwhelmingly likely to succeed, and often never bother with considering the possibility of failure at all, when in cold fact about two thirds of such enterprises go bust within the first five years. Meanwhile the ravages of planning fallacies in major projects that devour untold billions are so rife and regular a feature of our public life that it is the stuff of near-despair to linger over the disaster sites and contemplate their sheer range and scale.**

     CFOs of major corporations are unable to beat a randomizer reliably when predicting short-term market movements, yet would never say so out loud, even if they were dimly aware of their own limitations. Just how poorly highly-paid stock-pickers do for their investors qualifies as a scandal mostly in books discussing the notorious phenomenon; the rest of the world seems happy to look the other way by focusing on the best performances from year to year, which easily anoints winners simply as a function of random variation, while too often losing sight of the correlation between years over a longer period, which typically hovers around zero. What is true of individual stock-pickers applies equally to entire funds: some will easily look winners by the rules of random variation, while the losers will quietly close shop. Here as there, the longer the time-range, the fewer outstanding winners remain, on account of their lucky years regressing to the mean, making for the disturbing fact that mutual funds, while able to draw on the mental resources of some of our best and brightest, end up, as a group, underperforming the market, at least when fees are figured in.

     To make this embarrassing situation more palatable, recourse will almost certainly be had to dark caricatures of pretend-professionals profiteering at their clients’ expense. But this is the triumph of story over evidence (a point I shall return to), a misconstrual of the problem as if it were one of scheming incompetents or nefarious characters acting in bad faith, when in reality it afflicts all of us who are constantly tackling complexities, in modern life especially, for which we are not (or only very in adequately) equipped by our mental apparatuses, formidable as they are. The belief in reliable investment acumen may nonetheless be sustained, though more on faith than evidence (how could such expertise count for nothing?), or alternatively, it may be dismissed as irrelevant or misplaced, in which case one’s own crash-course capacity is often substituted on the model of drivers not overly burdened by self-doubt. Once again, there will be gloating winners aplenty in this game of chance disguised as skill, and not a few who lose their shirts and keep their mouths shut.

     Alongside this raging bonfire of vanities, fancies, and follies, we can also observe massive flows of capital into index funds, which might be taken for signs of a belated awakening to our mental limitations—if only it were not almost always done in a spirit of risk-aversion, profit-seeking, distrust of financial institutions, and a million other motivations except the kind of universal doubt in our human faculties, even at their best, that is at issue here. The explanation is not far to seek: we may easily come to question how competent others are to pick stocks for us, no matter how seemingly qualified; doubting our own ability to do so, as a matter of cognitive principle, is quite another matter. (For anyone to imagine that he could sort out this astronomically complicated mess in a short text like mine would itself be pure hubris, so please note my disclaimer, as usual, that I am reflecting, not proposing to resolve anything.)

     So-called pundits in other fields of social endeavor and inquiry differ not so much in the greater validity of their predictions as in that their pronouncements do not lend themselves to equally ready quantification, which will sometimes shield them more from scrutiny, sometimes less; either way, prosecutions for charlatanism are as rare as demonstrations of the emperor’s new clothes are common. Even the most trusted and admired of expert specialists—medical doctors—cannot be nearly as sure of their diagnoses, if they are true to the science of their profession, as they are expected to pretend if they do not wish to be taken for duffers or quacks.***

     How such illusions of validity arise and spread in a world that is much more random than we are usually willing to consider would require a chapter or book of its own.† What is important for our purposes here is how very readily they do arise and proliferate. False confidence seems to be far more the rule than the exception, to the point of being nearly inescapable and ineradicable. Indeed it has been suggested that it may be something human beings positively require, both in order put their trust in others, and to be comfortable with themselves. Kahneman calls it the very “engine of capitalism” in the title of his especially disconcerting chapter 24, after noting, a little earlier, how the confidence we have in our beliefs is nothing short of preposterous—yet it is not therefore any less essential.††

     It has even been suggested, not with a jesting so much as a sobering intention, that those rare human beings who take the most approximately realistic view of things will often strike others as almost intolerably pessimistic, even misanthropic—and perhaps earn themselves a medical verdict of depression into the bargain. (A theme I would rather not belabor again as I have already discussed it at some length in #32.) The great comparative attractiveness of going in the opposite direction, building up one’s confidence and good cheer by taking full credit for successes and seeking the blame for failures exclusively elsewhere, is attested to by the eminent marketability of sundry regimens of “positive psychology,” whether subtle or crude. That such approaches have almost everything to recommend them as feel-good strategies (and perhaps as real avenues to success in life, conventionally conceived) need not be denied—everything, that is, except a truthful relationship to how things really are, as opposed to how we would like them to be (our old friend yatha-bhuta).

     The everyday mechanisms that makes such wild flowerings of confidence not only possible but nearly unavoidable, except by laborious effort of a sort that may feel nothing short of emotionally twisted, is disturbingly straightforward. The simpler the stories we tell, the more coherent they will appear; the more coherent, the more credible; and the more credible, the more compelling as grounds for confidence. The range and quality of the evidence is not only largely irrelevant, it may even be detrimental inasmuch as it gets in the way of the coherence that produces confidence, not only in others, but even in oneself.††† And there things are likely to end unless we train ourselves, against the clamor of our instincts, to break the pleasing fiction and allow perplexity, or cognitive dissonance, to disturb our mental peace and undermine our cherished self-confidence. Whether doing so will make us happier may seem doubtful; that it will make us wiser, if sadder, is certain.

     What set apart Socrates’ lifelong quest for understanding as described in Plato’s Apology is not the great wisdom of a sage, since it remains dubious throughout the text whether human wisdom can ever amount to very much. What made Socrates special was not the rumors, fostered by others, of insights unrivaled into the secrets of the universe, but mostly that he did not, in contrast to his contemporaries, care to vaunt himself on knowledge that he did not, by his own insistent professions, in fact possess. What is so wrong, likewise, with the bottom of the Cave as Plato characterizes it in his famous allegory, is not that life there must be necessarily bad, either in a moral or a material sense, but that its denizens must remain, by dint of their orientation towards the Cave walls, fundamentally ignorant of their own true condition, and falsely confident of its superiority at the same time.

     By the logic of Plato’s dialectical ascent (not unlike that of Millian debate or Popperian falsification, compare Republic 534C), we can be sure only of our errors, namely when the best account we can give of our beliefs collapses under examination. When such an account comes through all our argumentative challenges more or less intact, we are still very far from knowing that we are correct. The ubiquitous talk of this is that widely held belief having been “proved right” is just that, loose talk; all we can say, speaking more properly, is that we seem to be on the right track whenever we have, despite our unrelenting efforts to the contrary, been unable to prove ourselves wrong. A far less satisfying position to be in, to say the least, for who would be willing to side so energetically, so perversely we might feel, against his own beliefs? It hardly looks like common sense, and yet it alone can lead us to a tentative truthfulness that will otherwise elude us.

     Not for nothing did Plato stress how unappealing the upward journey must look from the perspective of the Cave, and that no one would undertake it unless he had been shaken out of his complacency by the force of arguments he could not in good conscience resist. Whether the upward journey can ever lead, even at its uppermost reaches, to the same easy confidence that ordinary wordings feel at the slightest stirrings of their fanciful and incoherent opinions, is a troubling question not to be answered except in light of the great illumination that Plato likened to the Sun. Until then, it is, much like the Buddha’s Path, an arduous endeavor to which to commit one’s life, not a summer’s stroll through the park, started on a whim and ended just as easily.

     How much we are really, deep down, taken in by all the bluster and bravado of the outwardly confident, our own strained demonstrations to the world included, is another open question. Somewhere buried within us, we may all have an inkling that our shows of confidence are not only designed to impress others, but to cover up and keep hidden from our own view the dread of insignificance that we are all wont to feel, at bottom, and that we so desperately struggle to compensate for, unbeknownst to ourselves, with all means at our disposal (pace Adler, see especially #43). This may help to explain why sages impress us with being so at peace with themselves, even while they may not sound very humble by our ordinary standards. They are done with the conventional pretenses, the compensation, and the straining for effect that together rule our world.

     Short of such high sagacity, perhaps what is needed in the interim is another angle on the middle way: an ability to put yourself forward freely, but without therefore taking yourself for something you are not, or your faulty and inadequate ideas about the world for anything more than groping in the dark. This may sound simply reasonable and straightforward, as the middle way usually does. If only that meant one could count on it prevailing in a world where the noisy jeers and bellowing laughter of fools, as per the Tao (ch. 41), is often the surest indication that one is on the right track.

*From the same 1895 talk that I have drawn on several times before (see #71, #73, and #100), in a slightly condensed version. The longer quotation below is taken from the same text.

**All illustrations are mentioned by Daniel Kahneman in his Thinking, Fast and Slow, pp. 256–61 in the 2011 Penguin (UK) edition. For the planning fallacy, see pp. 249–53; also the hair-raising look, in the Economist of 17 Feb. 2024,  at the “ur-snafu” that is Britain’s high-speed rail fiasco (“How not to do a megaproject” and “The horror story of HS2”). This in a country that felt compelled, a handful years ago, to sell the plot of one of its most spectacular properties abroad, the Bangkok embassy, for GBP 420 million in order to raise money for the renovation, upkeep, and pencil-supply of other embassies around the world.

***For brief discussions of these phenomena, see Kahneman, pp. 261–62, 214–17, 218–19, and 263.

†In addition to Kahneman’s book, consider the classics by Leonard Mlodinow, The Drunkard’s Walk: How Randomness Rules Our Lives (2008), and by Nassim Nicholas Taleb, Fooled by Randomness (2001), a contribution as intemperate and uneven in style as it is insightful in substance.

††See Kahneman, p. 209 for the quotation, pp. 255–56 and 263–64 for some further reflections on its meaning.

†††Kahneman’s equally merciless and incisive summary on pp. 85–87 makes for harrowing reading when you think through the implications.