I am sure majority of people don't use password for ssh keys. The good solution is to use password manager like 1password which will prompt you to approve ssh.