From Distorted
Questions Worth Following
Begin with a claim. Turn it into a question you can check.
“This policy makes people safer.” Safer in what way? Which people? Compared with what? Over how long, and at what cost? Those details help us judge the claim.
“Unwritten Chapters” in Distorted invites us to examine difficult subjects. These six short plans show where to begin.
Every example below is fictional. These plans do not report finished or ongoing studies. To use a plan, find real records and reliable sources. Each plan sets a scope: what to check and over what period. It then names possible explanations and evidence that could help judge them.
What would show that a safety policy worked?
Public power
Scope. Imagine a train operator says a new station rule reduced reported assaults. Compare twelve months before and after the change at those stations. Count passenger trips too. Feeling safer, reporting fewer assaults, and suffering less harm are different outcomes.
Competing explanations. The rule may help. Changes in passenger numbers, staff, or how people report harm may also matter. Assaults might move to nearby places.
Evidence needed. Get dates, monthly assault reports, passenger counts, and counting and reporting rules. Check effects on different passengers. Compare similar stations without the rule. Check whether they differ in other ways too.
What could weaken the claim? Perhaps assaults were already falling. Similar stations without the rule may show a larger drop. Reporting may have become harder. Each would challenge the explanation. Fewer reports alone do not show that the rule caused less harm.
Begin with: Reading the Numbers and Compare the Stories.
Did the evidence change, or did the category change?
Identity and belonging
Scope. Imagine a workplace reports more “exclusion,” or being left out, after changing its staff survey. Compare the last old survey with the first new one. Check the words, who could answer, how many did, and what counted as exclusion.
Competing explanations. More people may be left out. New questions may capture experiences missed before. Different people may answer, or feel more willing to speak. Several changes may matter at once.
Evidence needed. Keep both surveys, counting rules, explanations of the changes, and full patterns of answers. Look for a test of both versions with similar groups of people.
What could weaken the claim? If the increase disappears when the same counting rule is used, that would challenge a claim that experiences got worse. It would not prove the new rule is wrong. What to measure and how to measure it are separate questions.
Begin with: Changing Claims and Follow the Revision.
What does a recommendation system actually change?
Media and technology
Scope. A video app chooses what to suggest next. Imagine a claim that these suggestions keep teenagers watching late at night. Choose ages, a country, four weeks, account settings, and what “late” means. Suggested videos, watched videos, viewing time, and sleep are different measures.
Competing explanations. Suggestions may matter. So may interests, friends' links, school schedules, or other app features. Teens who are already awake may watch more.
Evidence needed. Find reliable studies that explain their methods, permission, and privacy rules. Check what participants saw, how groups were chosen, and whether the study can show a cause. Testing one account cannot show how children are affected.
What could weaken the claim? A good comparison may find that changing suggestions makes no meaningful difference to viewing. That would weaken this explanation. Evidence about viewing alone cannot show a change in sleep.
Begin with: Keep a Claim Record.
What changed after the ethical commitment?
Institutions and moral language
Scope. Imagine a retailer promises to throw away 30% less stock within two years. Find the starting amount and which goods and parts of the business count. Does “thrown away” include goods donated or sent elsewhere? Total waste differs from waste per item sold.
Competing explanations. Buying only what is needed or reusing goods may help. So may lower sales, counting fewer parts of the business, or sending stock elsewhere.
Evidence needed. Keep the original promise and stock records. Check how the starting amount was measured and how transfers are counted. Look for outside checks. Follow what happened to the goods.
What could weaken the claim? Counting fewer stores would challenge a claim about improvement across the same business. If goods sent elsewhere were later thrown away, that would challenge a claim that waste was prevented. A narrower claim may still hold.
Begin with: Future Promises and When a Story Becomes Evidence.
What choices did participants actually have?
Public power and institutions
Scope. Imagine an employer calls a new scheduling app “voluntary,” meaning people can choose whether to use it. Check the first three months. Read the invitation and sign-up steps. What other option could workers use? What happened if they said no? Written choice may differ from real choice.
Competing explanations. Workers may like the app. They may also have been signed up automatically, lacked clear information, or faced costs for choosing another option. A high sign-up count does not explain why people joined.
Evidence needed. Check notices from that time, settings, other options, and records of what happened. Hear from workers who joined, refused, or left. Legal questions need the relevant local rules and expert advice.
What could weaken the claim? Penalties or an unusable alternative would challenge a claim of free choice. Clear information and a workable option would count against a claim that everyone had to join. Still check individual experiences.
Begin with: The Sound of Authority and Keep a Claim Record.
Who received the gains from automation?
Work and technology
Scope. Imagine a service team says an AI assistant helps it do more work with the same effort. This is a claim about productivity. Compare the same task over eight weeks before and after. Count finished cases, quality, repeat contacts, staff hours, and time spent fixing answers.
Competing explanations. The tool may help. Cases may be easier, staffing may change, or the rule for “finished” may change. Employees or customers may do more work fixing errors.
Evidence needed. Check similar cases, counting rules, quality checks, training time, and costs. Find out how saved time was used. Faster answers may not solve problems. Gains for the business may not benefit each worker.
What could weaken the claim? The gain may disappear once case difficulty and time fixing errors are counted. Faster first answers with problems left open would support only a narrower claim about speed.
Begin with: Fluent Answers and Reading the Numbers.
Choose one question and preserve its history
Name the real claim and who made it. Choose what to check. Write the possible explanations before choosing one. What evidence could change your answer?
Use Keep a Claim Record to begin. Save important changes with their dates and reasons. If key facts are missing, say what you still do not know.
From Distorted
These plans build on “Unwritten Chapters,” Chapters 28 and 29, and the Clarity Framework. Including a question here does not mean a claim about it is true.
Save this page
You can copy this link to your notes or bookmarks.