TL;DR Using a remote signer or a ledger does a great job at protecting secret keys. But it doesn't protect against the withdrawal of XTZ from accounts. Anyone that gains access to the baker can withdraw funds from it using a simple tezos-client command. Is this correct?
I thought I had this all figured out and had a cozy feeling of my little tezzies sleeping securely at night. But now I'm questioning everything I've done to secure them. I asked a question about security with solo baking a month or two ago and it has now come back to haunt me.
Quick summary:
I have a baker with just a minimum of XTZ to cover deposits. I have an originated KT1 address I created from a Ledger TZ1 implicit address. The majority of my XTZ are reasonably secure.
However, to protect my baker's XTZ, I decided to setup a remote signer on a different, very isolated box. For all practical purposes only my Tezos Node can talk to it via a private network.
As Luke pointed out in my earlier question linked to above, and what I didn't fully understand at the time was:
Because the signer has no authentication, anyone that can login to your baking system somehow can sign messages with the remote signer, including balance transfer of your bond/deposit.
Now I fully understand what he meant. So what's the point of having a remote-signer given this problem?
Even if I was using a ledger, wouldn't the problem still be the same? If I could get access to the baking system, I could withdrawal money from the account.
In both remote-signer and ledger scenarios, it seems we protect the keys -- they stay safe -- but the XTZ can still be transferred out and is not safe.
If we want to automate baking (remote-signer or ledger), we need to set things up this way. We can't sit around our computers all day and enter the password when prompted.
What am I missing here?