Grab Bench: Evaluating AI on Grab-shaped production work

Introduction What worried us wasn’t the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests. When we analyzed the row-level failures, a clear pattern emerged: SQL generation: kept the query shape but changed the underlying metric. Tool calling: selected the right tool family but drifted on parameters. Profile updates: cited every event instead of only the evidence that supported the claim. Coding