How do you test for catastrophic forgetting after fine-tuning?
A higher overall score can hide the loss of familiar capabilities. Use a paired retention test to decide whether a fine-tuned model is ready.
evaluation
3 posts with the tag “evaluation”
How should you split fine-tuning data when prompts repeat?
A random row split can put the same question in training and validation. Choose the holdout unit that matches the claim you want to test.
Can an agent's pass rate rise while it completes fewer tasks?
A higher agent pass rate can hide fewer completed tasks. Compare scored pass rate, scoring coverage and confirmed successes before ranking two evaluations.