Holding a private key in production

The same problem showed up at two different jobs, years apart. Where do you keep an Ethereum private key in a live service so it doesn’t leak, and doesn’t become the thing every request waits on. Passwords leak, you rotate them, life goes on. Keys don’t work that way. The money leaves in one transaction and no rotation brings it back. So you lock the key down hard. Fine. Except every lock you add sits right on the signing path, so the more paranoid the storage, the slower every signature that has to go through it.

At DreamTeam I put the keys behind HashiCorp Vault. Raw key never left it. The service couldn’t hold it, log it, or page it into memory even by accident, all it could do was send Vault a payload and get a signature back. Slow? A little, there’s an extra hop. Nobody cared, nothing user-facing waited on that path. Unstoppable was the harder version of the same thing. Signing sat on a hot path there, and an extra network round trip per signature would have been visible to users. Locking a key down is easy if you’re allowed to be slow. I wasn’t. Keeping the key material protected while the requests on top stayed fast, that was the actual exercise, and it took real design work instead of just “put it in Vault and forget it”.

None of this is work you demo. It doesn’t show up in a product screenshot, and when it works nobody notices it exists. But a mishandled key costs you the whole system, not a bug ticket, so it earns the attention.