How do I securely configure a CI server to digitally sign binaries?

Question

There are many sites that explain how to run signtool.exe on a .pfx certificate file, which boil down to:

signtool.exe sign /f mycert.pfx /p mypassword /t http://timestamp.server.com \
  /d "My description"   file1.exe file2.exe

I have a continuous integration CI process setup (using TeamCity) which like most CI processes, does everything: checks out source, compiles, signs all .exes, packages into an installer, and signs the installer .exe. There are currently 3 build agents, running identical VMs, and any of them can run this process.

Insecure implementation

To accomplish this today, I do a couple Bad Things(TM) as far as security is concerned: the .pfx file is in source control, and the password for it is in the build script (also in source control). This means that any developers with access to source code repository can take the pfx file and do whatever nefarious things they'd like with. (We're a relatively small dev shop and trust everyone with access, but clearly this still isn't good).

The ultimate secure implementation

All I can find about doing this "correctly", is that you:

• Keep the pfx and password on some secure medium (like an encrypted USB drive with finger-based unlock), and probably not together
• Designate only a couple of people to have access to sign files
• Only sign final builds on a non-connected, dedicated machine that's kept in a locked-up vault until you need to bring it out for this code-signing ceremony.

While I can see merit in the security of this.. it is a very heavy process, and expensive in terms of time (running through this process, securely keeping backups of certificates, ensuring the code-signing machine is in a working state, etc).

I'm sure some people skip steps and just manually sign files with the certificate stored on their personal system, but that's still not great.

It also isn't compatible with signing files that are then used within the installer (which is also built by the build server) -- and this is important when you have an installed .exe that has a UAC prompt to get admin access.

Middle ground?

I am far more concerned with not presenting a scary "untrusted application" UAC prompt to users than proving it is my company. At the same time, storing the private key AND password in the source code repository that every developer (plus QA and high-tier tech support) have access to is clearly not a good security practice.

What I'd like is for the CI server to still sign during the build process like it does today, but without the password (or private key portion of the certificate) to be accessible to everyone with access to the source code repository.

Is there a way to keep the password out of the build or secure somehow? Should I be telling signtool to use a certificate store (and how do I do that, with 3 build agents and the build running as a non-interactive user account)? Something else?

Rye

Which version control are you using? In centralized VC, like TFS, you can secure even a single file and restrict access to administrators only. – Lancashire

Git. That does give me an idea though -- I could perhaps store that file in a separate repository with different security, and have Teamcity check it out as well. If I don't get any better answers I'll update the post with whatever I end up doing. – Rye

Answer 1

I ended up doing a very similar approach to what @GiulioVlan suggested, but with a few changes.

MSBuild Task

I created a new MSBuild task that executes signtool.exe. This task serves a couple main purposes:

• It hides the password from ever being displayed
• It can retry against the timestamp server(s) upon failures
• It makes it easy to call

Source: https://gist.github.com/gregmac/4cfacea5aaf702365724

This specifically takes all output and runs it through a sanitizer function, replacing the password with all *'s.

I'm not aware of a way to censor regular MSBuild commands, so if you pass the password on commandline directly to signtool.exe using it will display the password -- hence the need for this task (aside from other benefits).

Password in registry

I debated about a few ways to store the password "out-of-band", and ended up settling on using the registry. It's easy to access from MSBuild, it's fairly easy to manage manually, and if users don't have RDP and remote registry access to the machine, it's actually reasonably secure (can anyone say otherwise?). Presumably there are ways to secure it using fancy GPO stuff as well, but that's beyond the length I care to go.

This can be easily read by msbuild:

$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\1 Company Dev@CodeSigningCertPassword)

And is easy to manage via regedit:

Why not elsewhere?

• In the build script: it's visible by anyone with source code
• Encrypted/obfuscated/hidden in source control: if someone gets a copy of the source, they can still figure this out
• Environment variables: In the Teamcity web UI, there is a detail page for each build agent that actually displays all environment variables and their values. Access to this page can be restricted but it means some other functionality is also restricted
• A file on the build server: Possible, but seems a bit more likely it's inadvertently made accessible via file sharing or something

Calling From MSBuild

In the tag:

<Import Project="signtool.msbuild.tasks"/>

(You could also put this in a common file with other tasks, or even embed directly)

Then, in whichever target you want to use for signing:

<SignTool  SignFiles="file1.exe;file2.exe" 
   PfxFile="cert.pfx" 
   PfxPassword="$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\1 Company Dev@CodeSigningCertPassword)"
   TimestampServer="http://timestamp.comodoca.com/authenticode;http://timestamp.verisign.com/scripts/timstamp.dll" />

So far this works well.

Rye

Good solution, you can even add ACL to a registry key and further limit access to it. – Lancashire

If you're running on windows you can import the certificate into the certificate store. Give the build agent access to the private key, specify the certificate to use by its name or thumbprint. You can even mark the key as un-exportable. Now you only need to protect the password to the user account your build process is running under (Which you should be doing anyway) – Keely

Answer 2

One common technique is to leave keys and certificates in Version Control, but protect them with a password or passphrase. The password is saved in environment variables local to the machine, which can be easily accessed from scripts (e.g. %PASSWORD_FOR_CERTIFICATES%).

One must be careful not to log these values in plain text.

Lancashire

Then when you check out an older version to build it, you would presumably have to remember to check out the trunk version of the keys, otherwise you would be signing against the obsolete key... – Irreparable

My suggestion is to keep the keys/certs in the same repo/folder structure unless you want to setup a much more complex build process. Remember that private keys are protected by the password and you can have multiple passwords around (e.g. %Y2014_PASSWD% and %Y2016_PASSWD%). – Lancashire

Yeah, I'd rather not make the build structure complex, but I'd also rather not tie my signing keys to the current version of the software since what they should be tied to is the current date. In many ways, it sucks that CI systems don't have a solution to this problem. In Jenkins, I can set up SSH credentials via private key, so I should be able to set up application signing credentials in much the same way. I just don't have the foggiest idea how to go about writing a plugin for that. – Irreparable

Just because something is common does not make it smart or secure. Storing private certificate keys in source control is insane, even if they are password protected. – Laraelaraine