How do we validate LLM-as-a-judge accuracy?

“We calibrate and validate every LLM judge against human-rater ground truth before it is deployed at scale, measuring inter-annotator agreement with Krippendorff's alpha and Cohen's kappa. Agreement scores and audit trails are documented for every deployment, and human validation is retained wherever the stakes are highest.”