Have you tried SFT to guide it early to more efficient trajectories? I know it hurts generalization and I've seen folks claim that it hurts final performance, but for my limited budget I literally will not do rubric based RL without SFT to shape the thinking to the rubric anymore.
It's so much more compute efficient for my goals, though I do deal with subjective preference which makes exploration during training much harder
It's so much more compute efficient for my goals, though I do deal with subjective preference which makes exploration during training much harder