Parameter Store Vs Encrypted Environment Variables for Lambda

Question

I recently was getting ready for the Security Speciality Exam, and I got the question to choose between using the Parameter Store to store a secret database connection URL which could hold passwords or to use KMS encrypted environment variables in the Lambda.

IMO Environment Variables are preferable because otherwise for Lambda Functions which are invoked many thousands or hundreds of thousands of times a day, this could start to cost a considerable amount of cost or could even result in hitting account limits.

In addition there is added latency to fetching the parameter each invocation, which may not be significant but nevertheless adds up. In general I would love to see a reference syntax implemented for Lambda environment variables to resolve to AWS SSM parameter values similar to what has now been implemented for Cloudformation for both SSM and secrets manager.

But until then why is SSM preferred over using KMS encrypted environment variables, considering the increased cost and latency? (This is what I have seen recommended in practice exams)

Danita

Answer 1

This article has some useful points:

• Hard to share configs across projects
• Hard to implement fine-grained access control
• [SSM Parameter Store] records a history of changes

So it would generally be a more flexible architecture to use SSM. That said, if these benefits really won't apply to you then you can still use environment variables and reduce the latency, as you pointed out. It is not so much that one is wrong, but that another is generally considered more of a "well-architected" approach. But specific cases may warrant other implementations.

This article mentions the better security it brings.

"While this approach [using environment variables] is simple and straightforward, it comes with considerable security drawbacks - the secrets exist in plaintext in the environment. Any other process, library, or dependency running inside the process has access to the environment which has already been exploited multiple times."

Security is a big consideration, and most things done to improve security bring latency or processing costs compared to less secure alternatives.

Some other thoughts to consider:

• https://dev.to/dvddpl/where-do-you-keep-credentials-for-your-lambda-functions-5dno
• https://www.stackery.io/blog/serverless-secrets/

Stalk

Answer 2

The biggest reason that comes to mind, is the ability to consume your tokens/secrets in a decoupled way, thus allowing other services leverage the same tokens. For example, if you have two lambda's that both need to invoke an external service with the same API token, then you only need to update a single place. If you did not do this, and lets say the token got rotated, then you would need to reconfigure each lambda to use a new token rather than making a single update to SSM.

Cotto

This is a good point, I hadn't have thought of that, most of the time I do usually just have one Lambda, but it would be more awkward if many Lambda has the same secrets. – Danita

Answer 3

The fine-grained security and single-source-of-truth advantages of SSM over regular env vars have been mentioned above as advantages.

In order to avoid the disadvantage of latency, this article provides the current best practice to avoid it when reading params from SSM:

https://aws.amazon.com/blogs/compute/sharing-secrets-with-aws-lambda-using-aws-systems-manager-parameter-store/

Basically what you need to do, is read the env vars once globally. Subsequent invocations of the same lambda already have access to them, so no extra round trips to SSM required.

Sublingual

"Finally, the function executed for 65 ms, of which 63.5 ms was the GetParametersByPath call to Parameter Store." - I sometimes have Lambdas which are under 10ms. This is not good. – Danita