Sure, but it's easy to accidentally start up the agent in the wrong directory, like when you open a new terminal. I've done it before when I was distracted (albeit with Claude, not Grok).
But isn't that user error? If you want to run a program that can read your current directory's contents and possibly upload that to the cloud, would you run that in a directory with your private keys?
