We need to remember that security is not a boolean "secure"/"not secure", but a spectrum. A compromised server does not have to be "game-over"; there are ways that you can still mitigate risk.
For example, a server can be compromised in multiple ways:
- arbitrary reads
- arbitrary code execution
- (root) privilege escalation
If you can bind your secrets so that they can only be used from your server, then arbitrary reads are definitely a problem, but the attacker could not use the secret without also getting arbitrary code execution.
If the endpoints you're connecting to can support signatures/MACs, then you can bind the key to the server by using some sort of Key Management System (KMS), like an HSM, TPM, cloud service, or Hashicorp Vault. The KMS should be configured only to allow access from a particular server, store the key and perform the signature/mac function, and then. In this way, arbitrary reads don't give the attacker anything, because there's nothing to read.
Even if the endpoints don't support signatures/MACs, if you can rotate the secrets regularly (using deployment automation or something like Hashicorp Vault), then the window which the attacker can use the secrets is pretty small, which can give you more time to detect and remediate the vulnerability that allowed the original attack.
Even if you can't rotate secrets, you then you can still use a KMS to grant access to the key at runtime; you may be able to use the anomalies in the KMS audit logs to help pinpoint the point of initial compromise. This doesn't prevent your data from leaking, but it can help you narrow down the vulnerability that needs to be patched.
Besides that, you can use OS-level protections to prevent secrets from leaking; e.g., if you have multiple services on the same server, then run each of them as a separate user, and give each application read-only access to a file that contains just the secrets that application needs. That way, even if the attacker gets arbitrary read and arbitrary code execution, it can still only read the secrets for the particular user it compromised, not all the users on the server (unless the attacker also gets privilege escalation).
You can go further into context-specific hardening techniques depending on your threat model: like encrypting disks if your storage disks are liable to be stolen/removed from the server, using OS-level protections, like SELinux or AppArmor on Linux to prevent reading secrets even by root processes, etc.
The moral is: there are things you can do to make attacks harder. Nothing is foolproof, but if you can make the attacker work hard enough, they might give up and move onto an easier target.
