When Forgetting Looks Like Improvement: Metric Masking in Streaming Diarizer Adaptation and the Price of Rehearsal
Small-data adaptation in streaming diarizers can improve in-domain performance while masking significant degradation in speaker attribution.
Researchers found that adapting streaming diarizers on small datasets leads to inconsistent performance across evaluation corpora. While in-domain diarization improves, the model often suffers from catastrophic forgetting, where gains in one metric hide losses in others.