SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
Proposes a cognitive diagnosis model to evaluate LLM skills beyond aggregate benchmark scores.
Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the diverse scope of skills covered by benchmark datasets. To address this limitation and better exploit benchmark data, we propose SkillEval, a new evaluation framework that performs cross-model, skill-level analysis. The framework integrates advan…