From Distorted
Questions Worth Following
Begin with a claim. Turn it into a question you can check.
“This policy makes people safer.” Safer in what way? Which people? Compared with what? Over how long, and at what cost? Those details help us judge the claim.
“Unwritten Chapters” in Distorted invites us to examine difficult subjects. These six short plans show where to begin.
These are fictional examples of how to plan an inquiry. Each names a question, possible explanations, and records to look for. They do not describe studies we have carried out or are running.
What would show that a safety policy worked?
Public power
The question. Imagine a train operator says brighter lighting at its stations reduced reported assaults. Compare twelve months before and after the lights were installed. Count passenger trips too. Feeling safer, reporting fewer assaults, and suffering less harm are different outcomes.
Other possible reasons. Better lighting may help. Changes in passenger numbers, staff, or how people report harm may also matter. Assaults might move to nearby places.
Evidence needed. Get dates, monthly assault reports, passenger counts, and counting and reporting rules. Check effects on different passengers. Compare similar stations without the new lighting. Check whether they differ in other ways too.
What could weaken the claim? Perhaps assaults were already falling. Similar stations without the new lighting may show a larger drop. Reporting may have become harder. Each would challenge the explanation. Fewer reports alone do not show that the lighting reduced harm.
Begin with: Reading the Numbers and Compare the Stories.
Did things get worse, or did the survey change?
Identity and belonging
The question. Imagine a workplace survey reports that more staff feel left out of meetings and decisions. The survey questions changed this year. Compare the last old survey with the first new one. Check the words, who could answer, how many did, and what counted as exclusion.
Other possible reasons. More people may be left out. New questions may capture experiences missed before. Different people may answer, or feel more willing to speak. Several changes may matter at once.
Evidence needed. Keep both surveys, counting rules, explanations of the changes, and full patterns of answers. Look for a test of both versions with similar groups of people.
What could weaken the claim? If the increase disappears when the same counting rule is used, that would challenge a claim that experiences got worse. It would not prove the new rule is wrong. What to measure and how to measure it are separate questions.
Begin with: Changing Claims and Follow the Revision.
What does a recommendation system actually change?
Media and technology
The question. A video app chooses what to suggest next. Imagine a claim that these suggestions keep teenagers watching late at night. Choose ages, a country, four weeks, account settings, and what “late” means. Suggested videos, watched videos, viewing time, and sleep are different measures.
Other possible reasons. Suggestions may matter. So may interests, friends' links, school schedules, or other app features. Teens who are already awake may watch more.
Evidence needed. Find reliable studies that explain their methods, permission, and privacy rules. Check what participants saw, how groups were chosen, and whether the study can show a cause. Testing one account cannot show how children are affected.
What could weaken the claim? A good comparison may find that changing suggestions makes no meaningful difference to viewing. That would weaken this explanation. Evidence about viewing alone cannot show a change in sleep.
Begin with: Keep a Claim Record.
Did the store throw away less food?
Institutions and moral language
The question. Imagine a grocery chain promises to throw away 30% less food within two years. Find the starting weight and which stores count. How does it track food donated or sent elsewhere? Less waste in total is different from less waste for each item sold.
Other possible reasons. Ordering less surplus food or selling it before it spoils may help. So may lower sales, counting fewer stores, or sending food elsewhere.
Evidence needed. Keep the original promise and food-waste records. Check how the starting weight was measured and how transfers are counted. Look for outside checks. Follow what happened to food sent elsewhere.
What could weaken the claim? Counting fewer stores would challenge a claim about improvement across the same business. If food sent elsewhere was later thrown away, that would challenge a claim that waste was prevented. A narrower claim may still hold.
Begin with: Future Promises and When a Story Becomes Evidence.
Could workers really choose not to use the app?
Public power and institutions
The question. Imagine an employer calls a new scheduling app “voluntary,” meaning people can choose whether to use it. Check the first three months. Read the invitation and sign-up steps. What other option could workers use? What happened if they said no? Written choice may differ from real choice.
Other possible reasons. Workers may like the app. They may also have been signed up automatically, lacked clear information, or faced costs for choosing another option. A high sign-up count does not explain why people joined.
Evidence needed. Check notices from that time, settings, other options, and records of what happened. Hear from workers who joined, refused, or left. Legal questions need the relevant local rules and expert advice.
What could weaken the claim? Penalties or an unusable alternative would challenge a claim of free choice. Clear information and a workable option would count against a claim that everyone had to join. Still check individual experiences.
Begin with: The Sound of Authority and Keep a Claim Record.
Did the chatbot save work or move it to someone else?
Work and technology
The question. Imagine an online store says its chatbot saves staff time answering questions about orders. Compare similar questions over eight weeks before and after it was added. Count questions answered correctly, repeat contacts, staff hours, and time spent fixing wrong answers.
Other possible reasons. The chatbot may help. Questions may be easier, staffing may change, or the store may count a chat as “finished” before the problem is solved. Customers may spend more time trying to reach a person.
Evidence needed. Compare similar questions and check whether the answers solved the problem. Include training, costs, and time spent correcting mistakes. Find out how saved staff time was used. Ask customers what they had to do next.
What could weaken the claim? Staff may save no time once corrections and repeat calls are counted. A faster first reply may still leave customers waiting for help.
Begin with: Fluent Answers and Reading the Numbers.
Choose one question and preserve its history
Name the real claim and who made it. Choose what to check. Write the possible explanations before choosing one. What evidence could change your answer?
Use Keep a Claim Record to begin. Save important changes with their dates and reasons. If key facts are missing, say what you still do not know.
From Distorted
These plans build on “Unwritten Chapters,” Chapters 28 and 29, and the Clarity Framework. Including a question here does not mean a claim about it is true.
Question something we published?
Send the page link, the passage at issue, and the evidence we should examine.
Save this page
You can copy this link to your notes or bookmarks.