From Distorted
When the Numbers Improve but the Outcome Does Not
Before accepting a better number, check what it counts, what it compares, and who it describes.
“We closed twice as many repair requests.”
Over how many days? And does “closed” mean the repair was finished? Those details change what the number tells us.
Imagine an apartment complex with leaking taps and broken heaters. Tenants want them fixed. The maintenance office counts requests marked “closed.” A closed request does not always mean a finished repair.
Put the denominator back
Fictional worked example. All numbers describe a made-up apartment maintenance team. The second report covers the days right after the first. Both use the same rule for counting a request as closed.
A denominator is the number you divide by. Here, we divide closed requests by working days. That gives us the average number of requests closed per day.
| Measure | Earlier report | Later report |
|---|---|---|
| Working days covered | 5 | 10 |
| Requests closed | 100 | 200 |
| Requests closed per working day | 100 ÷ 5 = 20 | 200 ÷ 10 = 20 |
The total doubled. The average per day stayed the same. Both are correct.
A longer period explains the bigger total. We do not know if staff did more work for each hour they worked. We would need their working hours to check that. We also do not know if closing more requests meant solving more problems.
Keep the denominator beside the number: requests per day, or complaints per 100 tenants. Dropping that part can change the meaning.
Check the outcome beside the activity
The team also counts requests opened again within 30 calendar days of being closed. It tracks every request for the full 30 days. Both reports use the same rules.
| Outcome after closure | Earlier report | Later report |
|---|---|---|
| Closed requests followed for 30 days | 100 | 200 |
| Requests reopened within 30 days | 20 | 50 |
| Reopening rate | 20 ÷ 100 = 20% | 50 ÷ 200 = 25% |
The reopening rate rose from 20 to 25 percent. Subtract 20 from 25: 5 percentage points. Divide that rise of 5 by the starting rate of 20: a 25 percent increase. Both describe the same change. Show the starting and ending rates so readers can follow it.
More requests reopened out of each 100 closed. This does not tell us why. Later requests might have been harder. Staffing might have changed. Tenants might have found it easier to report an unsolved problem. The table does not prove any of these ideas.
A request can stay closed while a tenant stays unhappy. This measure cannot tell the whole story.
Explore the numbers visually
Choose a view of the same fictional maintenance team. Every bar starts at zero. The scale and units are shown below it.
The total doubled. The later report covers twice as many working days. Choose “Per day” to compare the daily averages.
Average daily closures stayed at 20. This does not tell us how much work staff did per hour or how well problems were solved.
Every closed request was followed for the full 30 calendar days. The rate rose from 20% to 25%: 5 percentage points, or a 25% rise from the earlier rate. The record does not explain why.
Ask who the summary leaves out
A report on closed requests leaves out people still waiting. A survey tells us about those who answered, but may tell us little about those who did not. Check who was counted before treating a report as a picture of everyone.
An average can hide large differences. If the average wait falls, do the longest waits fall too? Do some groups still wait much longer? Ask for those details when enough data exists to support them. A percentage based on very few people may give a weak picture of a larger group.
We still need to know about open requests, waits, staff hours, and how hard the requests were. We can calculate daily closures without those details. We need those details to judge how well repairs are being handled.
When the counting rule changes
Suppose the team changes its rule. A request now counts as closed only after the tenant says the problem is fixed. The total might fall even if service gets better. The new rule may give a better measure.
Ask for both rules, why they changed, and when. If possible, count both periods with the same rule. If that cannot be done, explain the change and avoid a direct comparison.
Different kinds of change also need care. A basket of goods might cost $100, then $110, then $115.50. The price rises first by 10 percent, then by 5 percent. It rises more slowly, but it still rises. Both statements are true. They do not show that anyone lied.
Does the report help readers understand both?
Write the claim the record supports
Return to the maintenance team. A careful report could say:
The team closed 200 requests over ten working days, compared with 100 over the previous five. Average daily closures remained at 20. Among requests followed for a full 30 days, the reopening rate rose from 20 to 25 percent. We do not know how many staff hours the repairs took or why more requests reopened.
This keeps both the larger total and the worse reopening rate in view.
Before sharing a number, finish this sentence: “This measures ___, among ___, over ___, compared with ___.” Then name one thing the evidence cannot tell you.
Keep a Claim Record. Save the exact claim, the math, the counting rules, and one question about what is missing.
From Distorted: Chapter 4, “The Money Script”; Chapter 10, especially “Illusion of Mastery”; and Chapter 12, especially “Metrics as Mirage.” The numerical examples here were created for this guide.
Clarity Framework: Investigate Claims · Verify Facts. Explore the Framework.
Question something we published?
Send the page link, the passage at issue, and the evidence we should examine.
Save this page
You can copy this link to your notes or bookmarks.