Thanks for joining me for another edition of the SerenityThroughSweat blog. This week I came across a concept I hadn’t heard of before. This concept fits well across my work, fitness, and fatherhood domains, and I wanted to share it with you.
Charles Goodhart is a british economist, who worked for the Bank of England and the London school of Economics.
He is most famous for a small footnote he made while attending a conference held by the reserve bank of Australia in 1975. Goodhart wrote in his footnotes “whenever a government seeks to rely on a previously observed statistical regularity for control purposes, that regularity will collapse”.
The concept was simplified to, “when a measure becomes a target, it ceases to be a good measure” This became known as Goodhart’s Law.
In other words, when we use a measurement to reward performance, we have provided an incentive to manipulate the measurement in order to receive the reward.
The result can sometimes be actions that actually damage the effectiveness of the measured system while paradoxically improving the measurement of system performance.
There are all sorts of examples that I’m sure we can think about in our own lives but there were some fun ones that I read about that I wanted to share.
British officials in colonial India wanted to reduce the cobra population and offered rewards to locals who brought them a cobra skin. This led to locals breeding cobras to trade in the skins, actually increasing the cobra population.
The current academic environment emphasizes standardized test scores which leads to teaching to the test rather than well rounded education.
Airlines are ranked on their on time performance metrics. This has resulted in greatly inflated scheduled flight times so they can say they are arriving early.
Humans, and most every living thing for that matter, respond to incentives. However, if there is a way to game the system, to receive that incentive through easier or alternate means, ingenuity will certianly find a way.
The problem in overcoming Goodhart’s law, even when we are aware of it (which I previously was not), is that we have to measure something.
I wrote a post a long time back, (February of 2020, which seems like a lifetime ago) what gets measured gets managed (read it here). And, I believe that is still fundamentally true. The question then becomes what are you measuring/managing, and how do you prevent that measurement from being manipulated or its usefulness removed.
One answer relies on differntiating between Measures of Effectiveness MOE vs Measures of Performance MOP.
MOEs are inherently what we really want to know, but tend to be much murkier and harder to quantify. MOP’s tend to be much more concrete, but much easier to manipulate or to not effectively represent overall effectiveness.
Take a training plan for a runner or triathlete for example. What we really want to know when examining the plan is, “is the athlete getting faster, more fit, better prepared for their race goal” that would be the MOE.
But, until you run the race, its hard to really tell for sure. So, we rely on using MOP’s, like avergare heart rate, pace, recovery scores after a workout, or heart rate variability HRV.
No one is intentionally trying to sabotage their own training plan, but if you think athletes aren’t going out a little bit harder on an evaluation workout mid training plan I’ve got some real estate to sell you.
A workout plan desnigned to improve your HRV for example, might well be extremely successful when measure by change in HRV, but not very helpful in improving race performance.
The easy measure, is easy to track and improve. It may however, be minimally correlated to overall desired goals, or even detrimental.
Running an airline is even more complex. There is an ever-growing list of metrics you can look to in trying to determine if you are running the operation well.
On time performance D0, A0, or A14, cost per available seat mile (CASM), or company stock price, could all be used to asses operational performance or overall health.
On-time performance certainly belongs at the top of the list. It is tracked by the Department of Transportation, and every major brand competes and advertises based on the metric.
However, as we already mentioned, that metric is highly flawed and is subject to significant manipulation.
D0 means the aircraft leaving on or before the schedule departure time. This is measured by the parking break release after all exterior doors have been closed.
There is a push to improve D0 numbers as a MOP, which often results in requests to release the parking brake prior to actually being ready to depart. Thus “meeting” D0 goals without actually improving the system or meeting the overall goal of a well run airline.
We already metioned above that scheduled times are inflated to be able to meet the A0 or A14 times. The DOT counts an arrival of even 14 minutes late as an “ontime” arrival. So airlines give themselves a large buffer through inflated schedule times and the 14 minute DOT grace window.
Airline capacity is measured in seat miles. How many seats you are flying across how many miles, and the overall cost of one of those units is a significant competitive metric. This number can be artificially manipulated or boosted in short term at the expense of long term system health by a number of factors. Lower labor cost, lower fuel cost, increased seat miles. Focusing on just CASM, may result in decisons that adversely affect long term company health, in an, attempt to boost short term CASM goals.
Company stock price might be the ultimate example of Goodhart’s law. Decisions that boost quarterly profits and stock prices can often be at the expense of long term growth and health of the company. Often times executive compensation in stock or stock options provides a direct financial incentive to make decisions that are better for the stock price than they are for the company.
That is a whole lot of words, and a whole lot of examples describing the problem, without a lot of actionable solutions.
The hard part is there isnt an easy fix. The trap of Goodhart’s law is easy to slide into. MOP’s are much easier than MOE’s.
Only by constantly reevaluating the overall goal or mission, can we make sure the MOP’s don’t become corrupted.
By constantly asking, “does this course of action or measure serve my overall goal, or help me get closer to the path there?”, can we avoid the pitfalls of collapsing statistical regularities.
It is no easy task. Constant vigilance and tactical adjustments are required. But, there is serenity to be found along the route. And, would you really want it any other way?
Thanks for joining me, stay safe and stay sweaty my friends.