How do you validate an evaluation dataset for classifier training?

I’ve been building a classifier that tries to detect whether an LLM/agent response is grounded in the information available to it, the idea is to open source the model so that folks can actually train on top of it. Most classifiers I have researched seem to be ill fitting for this use case. But my problem is that I am unsure of how to benchmark my classifier using my own test dataset, I don't trust it to be wholly representative. I’m trying to figure out how people would approach validating this properly. A few things I’d love help with: Are there open datasets for groundedness / faithfulness