It's unfortunate to read such a workmanlike book when there are so many hilarious anecdotes about "metric fixation" that would have (and should have) made a book like this fun, and far more memorable. See for example C. Northcote Parkinson's Parkinson's Law as just such a book--one that teaches useful principles with memorable examples, irony and good storytelling.
This book, in contrast, offers diminishing returns beyond the first fifty pages or so. The author becomes increasingly repetitive, and some of the later chapters seem phoned in, as if he decided it wasn't worth shaping them into something more engaging.
Notes:
1) I don't know how many times (certainly three or four separate times throughout the book) the author cites the show The Wire as an example of "juking the stats" (gaming metrics such that it subverts the metrics' entire purpose). As powerful an "example" as this might be--especially to rabid fans like myself!--one has to keep in mind this is a show, it's not a real example, and it's certainly not actual evidence of an assertion.
2) "Creaming" examples:
* Surgeons avoiding risky surgery patients (even when surgery is indicated) to keep their numbers up (this can lead to indicated care being withheld from patients who really need it)
* Detectives going after crimes they know they can solve
* Prosecutors only taking the cases where they know they'll get a conviction, etc.
3) The author was heavily influenced by Robert Merton (one of the key thinkers behind certain modern equity and options pricing models). The irony is rich here, because these models led to a perfect example of the tyranny of metrics: the infamous blowup of Long Term Capital Management (which Merton co-founded with John Meriweather).
3) "Metric fixation" is based on a sort of stack of fallacies:
* that it can replace (presumed faulty and inexact) human judgment
* that numerical indicators of performance (based on standardized data) combined with a false belief that making such metrics public assures "accountability" for the performance of an institution
* that the best way to motivate people is to attach rewards and penalties to this measured performance
4) "Not everything that can be counted counts, and not everything that counts can be counted." This is a glorious quote, often mis-attributed to Albert Einstein.
5) Note also Campbell's Law: "The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." William Strunk must be spinning in his grave at this "law" which should be tightened into something a lot less forgettable. Nevertheless the underlying point is sound.
6) Think of metrics fixation as a modern cult or even a religion.
7) Problems of information distortion:
* Measuring the most easily measurable (see "the light's better here!" fallacy at the end of this post)
* Measuring the simple when the desired outcome is complex
* Measuring inputs rather than outcomes (e.g.: measuring money spent on a project or on "education" rather than the results)
* Degrading information quality through standardization
* Gaming of metrics: creaming, lowering standards, omitting or distorting data, cheating/juking
8) Robert Lowe, attempting to standardize education in Victorian England in 1862, criticized by Matthew Arnold for unintended costs and distortions; see also Frederick Winslow Taylor (ca 1910) attempting to standardize skilled manufacturing to produce greater efficiency; "Taylorism" ultimately made many manufacturing jobs (and then later service jobs, see McDonald's et al) into utterly numbing, mindless work.
9) See also Robert McNamara moving from optimizing manufacturing at Ford to fetishizing metrics like "body counts" in Vietnam.
10) Later on came a sort of worship of the managerial class; see also the rise of highly paid "management consultants" who are not experts in any individual business domain but are good at counting those beans that can be counted. [Note as a counterpoint here: one of the insights I've discovered (in my own slow way) over the course of my investing career is there actually are many strikingly similar problems and issues that can manifest in widely different business domains--and if you can think in analogies and metaphors you can find far more commonalities than you'd ever expect across a ridiculously wide array of industries.]
11) The "spreadsheet" as a worldview. An unconscious paradigm that skews how you look at reality. Just like the map is not the territory, a spreadsheet is not a business.
12) The author, an academic, appears to step somewhat outside his circle of competence discussing principal-agent theory in corporate America. Likewise see the medicine chapter (see #19 below).
13) Abstract, formulaic knowledge (a recipe, or management "best practices") vs tacit, practical knowledge (the ability to cook a recipe such that it tastes great). Some forms of knowledge do not render well in abstract formulaic form, and it can be totally impossible to convey tacit/practical knowledge in this form (see also the difference between declarative and procedural knowledge). The problem here is that formulaic/declarative knowledge can easily be conveyed, so that's what gets conveyed. Worse, it can be mistaken for these other forms of knowledge. [This is a type of "the light's better here!" problem, see #26 below.]
14) See also Hayek's "pretense of knowledge" concept: that economics fancies itself as a real science like physics, but it's really a far more recursive domain, therefore much more limited in what it can do.
15) Education as "a positional good--at least when it comes to the job market" therefore it's a fallacy to assume that more university graduates means a collective better standard of living or higher productivity. Worse, since education simply doesn't signal as much as it did, say, 30 years ago (when having a college education was a more meaningful metric), and since the job market adjusts well to relative qualifications, now jobs that would have required a high school degree now require a BA, even though those jobs are no more cognitively demanding nor require this level of education. Thus the so called "democratization of college" only serves to depress the wages of those lacking a college degree while it collectively takes away the gains for those who do! The more people who do get the degree the less it signals about the achievement of the individual who has it. A "positional arms race." [In other words, if everybody has the thing how valuable is the thing? Also, kudos to the author to have the courage to say some of this stuff, it is increasingly risky to voice these truths in the post-modren era.]
16) Continuing with education as a positional good: the second-order effects of people chasing this particular positional good are really messed up: it drives people to chase only specific "brand-name" schools; it preposterously drives up tuition at these schools; it may drive much more graduate education that our labor market or our students need (but yet they need it in order to have yet another positional good that differentiates them, since the value of a bachelor's degree has been democratized away), and so on. These are all behaviors designed to distinguish the individual in comparison to an overall lowered standard.
17) The author has a genuine insight on how businesses have a built-in restraint on spending too much time on metrics: because at some point it takes up too much cost structure. Yet universities and nonprofits don't have a bottom line, thus the metric fetish just leads to more and more creep and cost structure growth. This means more and more admin staff--which suits the administrators just fine!!
18) There's also an ironic kind of Escher problem in this book: the author has to use metrics to show that metrics don't work.
19) Re the chapter on medicine: this chapter could have been much more powerful, covering many more problems in the industry, but it ends up boring. Worse, the author appears to be somewhat outside his circle of competence here as well (see also #12 above). There are many other medical sub-domains the author could have covered, here are just a few that come to my mind:
* spurious metrics used in drug studies to "prove" the efficacy of meds
* the lack of visible information on iatrogenic harm (a reverse "the light is better here" problem)
* the various ways "screening risks" are hidden or ignored
* specificity problems in medical tests (the latest and greatest example here would be unknown specificity in COVID tests, which led to all sorts of unintended problems)
* how easily metrics can be gamed or goalposts can be moved in drug trials, even while the trial is ongoing, with catastrophic ramifications for patient health, med safety, and which meds are actually indicated; and so on.
20) Re: the chapter on the military and the chapter on business: both chapters are short, superficial, and the author is out of his depth.
21) The business chapter would have benefited enormously from a rather obvious source of useful information on metrics: Ray Dalio and his well-known firm Bridgewater Associates, which pioneered many striking and highly effective ways of using metrics for employee (and management) performance. It's borderline shocking that there is absolutely no mention of Dalio or Bridgewater anywhere in this book. This is a glaring and quite honestly embarrassing omission. How can you write a book about metrics and have no idea about Bridgewater?
22) The author also doesn't know enough about the WCOM scandal to crosscheck his sources: he gives this away as he cites Bernie Ebbers as the mastermind who had inside knowledge of all the manipulated metrics, when anyone who's ever met either man would know that Scott Sullivan was the brains of that operation. I guess you don't know what you don't know.
23) He's also off the mark with the standard consensus complaint about capitalism: that because of its fixation on quarterly earnings results, capitalism suffers from "short-termism." This is a naive empiricism-type error, the kind of criticism you'd make if you didn't really understand why quarterly results are (and are sometimes not) useful, how they can (and cannot) be gamed, what investors and shareholders want and need from those results, why companies can benefit enormously from frequent, periodic guideposts against a plan, etc. Perhaps as a fieldwork exercise, the author should invest in some non-domestic companies that only file annual results, or better yet, companies that don't file financial results at all?
24) Examples of unintended but predictable negative consequences of metrics fixation:
* "Goal displacement": following the goals that are measured rather than the real goals. Thus "the metric becomes the target."
* Short-termism
* Increased costs in employee time/company time
* Diminishing utility (your metrics might reveal your "low performers" or offer you low-hanging fruit right away, but then the marginal value of keeping these metrics falls off rapidly once this is uncovered)
* "Rule cascades": a new rule is put in place in every instance where someone games, cheats or subverts metrics, thus arrives a flood of new rules that everybody has to follow in addition to the original metrics, further slowing down and distracting the institution from its actual purpose
* Rewarding luck
* Discouraging risk-taking, reducing initiative
* Discouraging innovation, people will do what the metrics measure and not do anything else
* Discouraging cooperation and a common purpose, instead merely producing metric competition inside an organization
* Degradation of work: it loses its intrinsic value and becomes extrinsic as workers are working simply to meet the metrics.
25) Finally, I can't help but share the wonderful old joke about confounding what you can easily see with what actually matters (and if the author had used a lot more stories like this he'd have written a better book):
A drunk was crawling around on his knees under a streetlight.
A passerby asks him, "Why are you on your hands and knees? Can I help you?"
The man replies, "I dropped my car keys about half a block away and I'm looking for them."
"You dropped your keys half a block away? Then why are you looking for them here?"
"The light's better here!"
To Read:
Edward N. Luttwak: The Pentagon and The Art of War
Matthew Stewart: The Management Myth: Why the Experts Keep Getting it Wrong