A plain English guide to the terms you’ll meet in the lab, the ways an answer can go wrong, and the habits that help you judge it fairly.
You do not need to memorise these labels. Start with what was asked, what the evidence says and whether the answer fits. An answer may have several problems—or none.
Describe the specific problem you can demonstrate. A wrong answer alone does not reveal why the model produced it.
A statement that conflicts with an established fact or the evidence you were given.
A source says 18 of 30 people preferred a design. The answer says that is 90%; it is actually 60%.
Ask yourself: Do the names, dates, quantities and calculations match the source?
Practise this skillIn AI discussions, this term describes generated content that is false or fabricated, such as a made-up quotation or reference. NIST also uses the term confabulation. It does not establish deliberate lying.
An answer names a study, author and journal, but checking the publisher’s records establishes that the cited study does not exist.
Ask yourself: Can you verify the source and does it actually support the claim? A failed quick search alone does not prove something was invented.
Practise this skillThe answer draws a stronger conclusion than its evidence justifies. That conclusion may be true, but it has not been established.
A survey of one class finds that most prefer morning lessons. The answer concludes that all students learn better in the morning.
Ask yourself: Has a limited finding become a sweeping claim? Watch for “all”, “always” and “proves”.
Practise this skillTwo things occurring together does not, by itself, show that one caused the other.
People using a study app get higher marks. They might already be more motivated; the result alone does not establish that the app caused the improvement.
Ask yourself: Could another factor explain the link? What would help establish cause and effect?
Practise this skillA conclusion does not follow from the reasons offered, or different parts of the answer conflict.
“All club members receive a pass. Maya has a pass, so she must be a club member.” Non-members might receive passes too.
Ask yourself: Does each step follow? Could the stated reasons be true while the conclusion is false?
Practise this skillAn answer can contain correct information and still fail the task: for example, by ignoring a format, word limit or source restriction.
The prompt asks for two reasons using only the supplied passage. The answer gives five reasons and introduces outside claims.
Ask yourself: Read the prompt again. What exactly was requested, and what constraints must the answer meet?
Practise this skillSelecting only favourable evidence, or leaving out a detail that changes the meaning of the answer.
A trial found a small improvement but had only ten participants and no comparison group. A summary reports the improvement and leaves out those limits.
Ask yourself: What relevant evidence or qualification has been left out? Not every omission matters—focus on details that change the judgement.
Practise this skillThe answer sounds more certain than the evidence allows. Cautious wording can also be misleading when the evidence is decisive.
Two equally credible notices give different start times. The answer confidently chooses one without explaining the conflict.
Ask yourself: Is certainty proportionate to the evidence? An honest “we cannot determine this yet” may be the most accurate answer.
Practise this skillInformation that was once correct may no longer answer a question about the present.
An answer uses last year’s timetable to describe tomorrow’s departure time.
Ask yourself: What date does the information apply to? For a current question, use a current authoritative source.
Practise this skillJudging an individual using a generalisation about their group rather than relevant evidence about that person.
An answer assumes an older applicant cannot learn new software, without information about their experience or ability.
Ask yourself: Is the conclusion based on this person’s evidence, or on a group label? Criticism supported by evidence is not automatically bias.
Practise this skillThese terms describe human judgement habits. Similar patterns may appear in an AI response, but a single answer does not establish the model’s internal beliefs or motives.
Giving more attention or weight to information that supports what you already believe, while overlooking or discounting contrary evidence.
You think a response is wrong. You search only for sources that criticise its claim and ignore a reliable source supporting it.
Ask yourself: What evidence would change my mind? Have I looked for the strongest evidence against my current view?
Giving too much weight to the first answer, number or impression you encounter when making a judgement.
The first response says a task takes 20 minutes. You treat that as your starting point even after the instructions establish a much longer minimum.
Ask yourself: If I had not seen the first estimate, what would the evidence lead me to conclude?
Accepting an automated answer or recommendation without giving it enough independent scrutiny.
You accept a calculation because an AI produced it, even though the figures supplied let you check it yourself.
Ask yourself: Would I apply the same checks if a person had written this answer?
For the evidence exercises, use the supplied material and these definitions. These are the lab’s scoring distinctions; other evaluation tasks may use different labels.
Unsupported does not mean false. A leap from a survey to a claim about everyone is unsupported. If opening hours tell you nothing about ticket prices, the price cannot be determined from that source.
The full comparison rubric adds appropriateness and user satisfaction to accuracy, relevance, instruction following and clarity. Paste real responses from any two platforms and apply the same standards to both.
Look for specific content such as targeted harassment, discriminatory abuse, exposure of private information or instructions enabling harm. Record the relevant passage and explain the concern using the applicable task or platform policy when supplied. A neutral educational discussion of a sensitive subject is not automatically inappropriate.
No issue identified: no concern established in the material reviewed. Flag inappropriate material: a specific concern you can explain. Needs further review: the policy, audience or other essential context is missing or unclear. A flag records your judgement; it does not report content to a platform.
Ask how well the response meets the stated user’s need, including usefulness, completeness and tone. Use 1 for not useful, 2 for mostly unsatisfactory, 3 for partly useful, 4 for useful with a small gap, and 5 for fully meeting the need. This is your assessment of likely satisfaction, not a survey of the user.
A pleasing answer can still be false or inappropriate. Keep these judgements separate and explain any trade-off. One comparison is evidence of how you evaluated those responses, not a ranking of entire platforms.
Compare platform responsesA fluent answer is not automatically correct. A confident answer is not automatically wrong. Judge both by the evidence and the task.
Try the guided introductionDefinitions are explained in our own words. All examples on this page are fictional.