When I first ran an RLHF training job, I was surprised at how easily the reward model scores increased during the training process.