The 3 ways to grade LLM outputs automatically (and when each one fails)

If you want to evaluate prompt outputs without reading every single one, there are basically three grader types: 1. Deterministic graders. Exact match, regex, JSON schema checks, small scripts. Best for: structured output, extraction, classification. Fails when: quality is subjective. You can't regex "is this summary good". 2. LLM-as-judge. A model grades the output against criteria you define. Best for: tone, helpfulness, correctness of free text. Fails when: you're not explicit. Always spot-check it against your own judgment first, and give it explicit criteria. A vague judge is a useless ju