From Distorted
When the Numbers Improve but the Outcome Does Not
Before accepting a better number, check what it counts, what it compares, and who it describes.
“We handled twice as many cases.”
Over how many days? And what does “handled” mean? We need those details to judge the result.
Customers want problems fixed well and on time. A service desk may count closed cases. A closed case does not always mean a solved problem.
Put the denominator back
Fictional worked example. All numbers describe a made-up service team. The second report covers the days right after the first. Both use the same rule for counting a case as closed.
A denominator is the number you divide by. Here, we divide closed cases by working days. That gives us the average number of cases closed per day.
| Measure | Earlier report | Later report |
|---|---|---|
| Working days covered | 5 | 10 |
| Cases closed | 100 | 200 |
| Cases closed per working day | 100 ÷ 5 = 20 | 200 ÷ 10 = 20 |
The total doubled. The average per day stayed the same. Both are correct.
A longer period explains the bigger total. We do not know if staff did more work for each hour they worked. That measure is called productivity, and we do not have their working hours. We also do not know if closing more cases meant solving more problems.
Keep the denominator beside the number: cases per day, or complaints per 100 customers. Dropping that part can change the meaning.
Check the outcome beside the activity
The team also counts cases opened again within 30 calendar days of being closed. It tracks every case for the full 30 days. Both reports use the same rules.
| Outcome after closure | Earlier report | Later report |
|---|---|---|
| Closed cases followed for 30 days | 100 | 200 |
| Cases reopened within 30 days | 20 | 50 |
| Reopening rate | 20 ÷ 100 = 20% | 50 ÷ 200 = 25% |
The reopening rate rose from 20 to 25 percent. Subtract 20 from 25: 5 percentage points. Divide that rise of 5 by the starting rate of 20: a 25 percent increase. Both describe the same change. Show the starting and ending rates so readers can follow it.
More cases reopened out of each 100 closed. This does not tell us why. Later cases might have been harder. Staffing might have changed. Customers might have found it easier to report an unsolved problem. The table does not prove any of these ideas.
A case can stay closed while a customer stays unhappy. This measure cannot tell the whole story.
Explore the numbers visually
Choose a view of the same fictional service team. Every bar starts at zero. The scale and units are shown below it.
The total doubled. The later report covers twice as many working days. Choose “Per day” to compare the daily averages.
Average daily closures stayed at 20. This does not tell us how much work staff did per hour or how well problems were solved.
Every closed case was followed for the full 30 calendar days. The rate rose from 20% to 25%: 5 percentage points, or a 25% rise from the earlier rate. The record does not explain why.
Ask who the summary leaves out
A report on closed cases leaves out people still waiting. A survey tells us about those who answered, but may tell us little about those who did not. Check who was counted before treating a report as a picture of everyone.
An average can hide large differences. If the average wait falls, do the longest waits fall too? Do some groups still wait much longer? Ask for those details when enough data exists to support them. A percentage based on very few people may give a weak picture of a larger group.
We still need to know about open cases, waits, staff hours, and how hard the cases were. We can calculate daily closures without those details. We need more to judge the service.
A changed measure needs an explanation
Suppose the team changes its rule. A case now counts as closed only after the customer says the problem is fixed. The total might fall even if service gets better. The new rule may give a better measure.
Ask for both rules, why they changed, and when. If possible, count both periods with the same rule. If that cannot be done, explain the change and avoid a direct comparison.
Different kinds of change also need care. A basket of goods might cost $100, then $110, then $115.50. The price rises first by 10 percent, then by 5 percent. It rises more slowly, but it still rises. Both statements are true. They do not show that anyone lied.
Does the report help readers understand both?
Write the claim the record supports
Return to the service team. A careful report could say:
The team closed 200 cases over ten working days, compared with 100 over the previous five. Average daily closures remained at 20. Among cases followed for a full 30 days, the reopening rate rose from 20 to 25 percent. These records do not establish changes in productivity or explain the higher reopening rate.
This keeps both the larger total and the worse reopening rate in view.
Before sharing a number, finish this sentence: “This measures ___, among ___, over ___, compared with ___.” Then name one thing the evidence cannot tell you.
Keep a Claim Record. Save the exact claim, the math, the counting rules, and one question about what is missing.
From Distorted: Chapter 4, “The Money Script”; Chapter 10, especially “Illusion of Mastery”; and Chapter 12, especially “Metrics as Mirage.” The numerical examples here were created for this guide.
Clarity Framework: Investigate Claims · Verify Facts. Explore the Framework.
Save this page
You can copy this link to your notes or bookmarks.