Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Researchers propose using cross-model disagreement to identify ambiguous data points, allowing experts to focus on refining codebooks for large-scale annotation.
Developing robust codebooks for large-scale annotation is time-consuming. This study suggests using LLM disagreement to surface complex cases, enabling experts to provide targeted feedback. This approach aims to improve annotation quality and efficiency by prioritizing human effort where models struggle to reach a consensus.