Just spit balling: could you have the agent pre warm the cache as part of your workflow? Like at the start of working on the code have the agent run a compile in the background. That way when the agent is ready to "really" compile there is a warm cache