How To Store Your AWS Lambda Secrets

Do it right, so you don’t have to redo it when your application load increases

https://miro.medium.com/max/1400/0*rsK6aNmJQJj8qNYt

Don’t tell anyone your secrets. Do tell people how to store secrets. (Photo by Kristina Flour on Unsplash)

Most Lambda deployments use environment variables to pass configuration to the deployed function. It’s the recommendation that AWS makes in the documentation, and it’s an attractive choice because it’s easy, encrypted at rest, and allows for flexibility in how you get the values there in the first place.

There are many articles with good recommendations about lambda configurations already. Why should you read this one?

Instead of comparing and contrasting approaches, this is a how-to guide for anyone whose primary values are minimising cost without compromising scalability or security. If you have additional or different needs, I recommend reading the links above as well.

This guide is aimed at small to medium teams working in contexts where security matters, but fine-grained permission management might not.

Just tell me the answer

If you’re here from Google and just want a recommendation, feel free to skip to the end for the summary. If you want the detailed rationale behind the recommendation, read on.

Secrets in Environment Variables

Storing configuration in environment variables is fine for details that aren’t secret (like API endpoint locations, hostnames, public keys, etc). However, it’s not so effective for configuration that is potentially sensitive like passwords or API keys.

Even though Lambda environment variables are encrypted at rest, they’re visible to anyone who has the permissions to see the Lambda in the console. This isn’t great, as it violates the principle of least privilege — there’s no need for sensitive data to be easily accessed by people or services that don’t need it.

There’s a practical problem too — having environment variables store sensitive data next to non-sensitive data makes it much more likely that values are accidentally exposed by logs from CloudFormation or general execution. It’s harder for people to remember to protect sensitive data if they’re used to it displaying in the course of their daily work, especially if it is displayed next to non-sensitive data.

It’s worth noting that even though the Serverless Secrets Plugin uses this approach, the documentation starts with the following warning:

IMPORTANT NOTE: As pointed out in the AWS documentation for storing sensible information Amazon recommends to use AWS KMS instead of environment variables like this plugin.

That is, AWS recommends not putting plaintext secrets in your environment variables, and I agree.

https://miro.medium.com/max/1400/0*-pX2A6P6lQQRcvmp

Which of these are meant to be secret? (Photo by Jason D on Unsplash)

Environment Variables Encrypted With KMS

As the quote suggests, AWS recommends encrypting environment variables with KMS beforehand . This means the encrypted version is all that would be exposed in the console. The approach is straightforward — you use KMS to encrypt the value before putting it into the environment variable, and then you decrypt it with KMS inside your Lambda code. Here’s a nice walkthrough.

If all you care about is price and security, this approach is perfect. However, since we’re looking for something that scales nicely, it would be better if we had some centralised configuration.

Centralising the Configuration

AWS offers two main approaches for centralised configuration —Secrets Manager and Systems Manager’s Parameter Store (confusingly abbreviated as SSM Parameter Store).

Secrets manager

Secrets Manager is a great fit if you need detailed control over when and where each secret can be used. However, with both storage and access costs, it is considerably more expensive than the free basic storage of Parameter Store.

Because of the increased cost, I’m not considering Secrets Manager here. However, if it’s right for your needs, this tutorial is a good starting point.

https://miro.medium.com/max/1400/0*YwHfzwTq1-TEUaIx

Not that kind of centralised (Photo by K8 on Unsplash)

Secrets in SSM’s Parameter Store

Parameter Store has a couple of nice features, including hierarchical parameters. This means you can name parameters with a path for ease of retrieval. For example, if you name your parameters with namespaces separated by slashes:

aws ssm put-parameter              \\
    --name "/your/app/some_value"  \\
    --type "SecureString"          \\
    --value "foo is a value"    aws ssm put-parameter              \\
    --name "/your/app/other_data"  \\
    --type "SecureString"          \\
    --value "another value"

Then you can give your function permissions to retrieve all parameters under that path:

- Effect: Allow
  Action:
   - ssm:GetParameters
   - ssm:GetParameter
  Resource: 'arn:aws:ssm:<REGION>:<ACCOUNT>:parameter/your/app/*'

Getting Parameters at Runtime

Another neat feature is the ability to retrieve the parameters at runtime. This is nice — it decouples deployment and configuration. If you want to do this, here’s a helper gist I wrote that smooths the process for node Lambdas.

However, Parameter Store has a drawback: the throughput is very low. Each parameter retrieved with get-parameters counts as one request and you’re only allowed 100 requests per second.

You can increase this limit, but doing so raises the price, and still only allows 1,000 requests per second.

Aside: If you go this route, I strongly recommend loading the Parameter Store parameters outside your handler, so that they will be loaded once per Lambda instance, not once per handler execution.

Reading Encrypted Values From Parameter Store at Deploy-Time

So, for nicer scaling, we want a centralised parameter store. Parameter Store’s free cost is attractive, but it won’t scale because of the requests-per-second limit.

What about reading the encrypted data (without decrypting) at deploy time? Since Parameter Store is backed by KMS, we could read Parameter Store’s secure parameters at deploy time without decrypting, and decrypt at runtime using the KMS API.

If you’re using the serverless framework, this is actually really straightforward:

${ssm:/path/to/secureparam~false}

The false says “please don’t decrypt this”.

However, I prefer not to use the Serverless framework — and using SecureString params like this isn’t supported by CloudFormation.

Aside: if you’re using the Serverless framework, and want to go this route, note that you will need to remember the parameter’s ARN to use as encryption context during decryption. More details here.

https://miro.medium.com/max/1400/0*LGP42Hf2_s082Ocr

Sometimes you can’t see the clouds for the… other clouds (Photo by Łukasz Łada on Unsplash)

Secure Parameters and CloudFormation

If you’re using CloudFormation directly, then SecureString parameters are not supported. You can’t use them as parameters:

# Broken CloudFormation:Parameters:
  SomeParam:
    Type: 'AWS::SSM::Parameter::Value<String>'
    Default: '/path/to/secureparam/'## ERROR ##An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters [/path/to/secureparam] referenced by template have types not supported by CloudFormation.

Nor can you use them outside the approved locations:

# Broken CloudFormation: Resources:
  ReceiveLambda:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          VAR_NAME: '{{resolve:ssm-secure:/path/to/secureparam:1}}'## ERROR ###Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: SSM Secure reference is not supported in: [AWS::Lambda::Function/Properties/Environment/Variables/VAR_NAME]

However, you can use String parameters, which are not encrypted by Parameter Store. So, I propose encrypting parameters with KMS first.

Aside: The excellent StackMaster tool adds better support for SSM’s Parameter Store (and many other useful features). However, it still decrypts SecureString parameters, which means they won’t be encrypted in the Lambda console.

https://miro.medium.com/max/1400/0*8uuioH_v0ijRxqnI

We’re going to need two steps down (Photo by Brett Jordan on Unsplash)

Recommendation: KMS + Parameter Store Strings

Instead of using Parameter Store’s SecureStrings to abstract away the KMS step, I recommend using KMS to encrypt strings that go into Parameter Store as plain strings. This provides:

• Low cost ($1 per month for a KMS Customer Key)
• Scalability (no rate limits)
• Security (no exposure of keys)
• Flexibility of use (can be used by CloudFormation)

This has some drawbacks:

• Not completely free
• Config only updated at deploy time (but for extra credit, you could use Parameter Store’s triggers to kick off deployment whenever the parameters change)

Manual Encryption of Parameters With KMS

First, you’ll need to create a customer managed key in KMS. Currently, this costs $1 per month, but as long as you have at least three secrets, it’s still cheaper than Secrets Manager.

This command will insert an encrypted String parameter into Parameter Store:

aws ssm put-parameter                       \\
   --type String                            \\
   --name '/YOUR/PARAM/NAME'                \\
   --value $(aws kms encrypt                \\
              --output text                 \\
              --query CiphertextBlob        \\
              --key-id <YOUR_KEY_ID>       \\
              --plaintext "PLAIN TEXT HERE")

Your user must have kms:Encrypt permission on the key you created above. Note that KMS IAM permissions need the key ARN, and won’t work with the key alias ARN.

Use Parameter Store in CloudFormation

There are a few ways to get your SSM Parameters into your CloudFormation stack. Here’s one pattern I particularly like:

Parameters:
  SomeParameter:
    Type: AWS::SSM::Parameter::Value<String>
    Default: '/your/param/name'        # This is your parameter nameResources:
  ReceiveLambda:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          SOME_PARAMETER: !Ref SomeParameter

Decrypt at Runtime Using KMS in Your Lambda

First, you will need the following permission on your Lambda’s execution role:

- Effect: Allow
  Action:
    - kms:Decrypt
  Resource:
    - <KEY_ARN>           # Note, the key ARN, not the key alias ARN

Then, you can decrypt the parameters. Here’s a code snippet that will do it in node:

That’s it!

Aside: Now all you have to do is be careful not to log or otherwise expose the decrypted secret.

Summary

To have cheap, scalable configuration without unnecessarily exposing your secrets:

• Manually encrypt your parameters using KMS before saving them to Parameter Store as a plain string (here’s a bash script to make this ultra-easy)
• Use the Parameter Store in your deployment lifecycle, ending up in an environment variable on the Lambda
• Decrypt the environment variable at runtime using KMS (here’s an example node.js module you could crib from).
• Do the decryption on function load instead of in the handler to minimise KMS calls.

I’ve written and shared a bash script to help with creating KMS-encrypted parameters. If you’re using node for your Lambdas, I’ve shared this module as a starting point for your decryption.

Finally, here’s a quote about AWS Lambda configuration from the keynote speaker at AWS:MiddleEarth (probably):

Passing Credentials to AWS Lambda Using Parameter StoreNovember 1, 2021 8:00 PM / Python

This post is part of my AWS Lambda 101 series:

1. How To Create Your First Python AWS Lambda Function
2. Setting Up a Recurring AWS Lambda Function Using AWS EventBridge
3. How To Create an Endpoint for an AWS Lambda Function Using API Gateway
4. Passing Credentials to AWS Lambda Using Parameter Store
5. Create a Twitter Bot Using Python and AWS Lambda

Learn how to use AWS Systems Manager for securely storing and retrieving secrets. Description: Need to pass credentials or secrets to your Python Lambda functions? Learn how to use AWS Systems Manager for securely storing and retrieving secrets. Status: published

Up until now, all my articles in this AWS Lambda 101 series has only used simple Python code that does not interact with any data or service outside the function. However, in many cases, you will want to pull data from somewhere or send data to a service.

For example, maybe you will pull data from a database. Or, maybe you need to send a message to Slack or Twitter.

For all of these, you will need a way to use credentials or secrets within the code. However, you should never save the secrets within your Lambda function code. Instead, you will need a way to pass these values to the Lambda function.

While there are several ways you can do this, the easiest and most affordable way is to use AWS Systems Manager’s Parameter Store. The Parameter Store is a simple key/value system that allows you to store simple strings and encrypted values.

Adding a Value to Parameter Store

To begin, log into your AWS account. Once in your AWS console, you can either click on “Services” to go to “Management & Governance”->”Systems Manager” or use their search bar to search for System Manager.

Once inside Systems Manager, you will find your dashboard as shown here:

Click on the “Parameter Store” page from within the menu to get to your values. If you do not have any values yet, you will see a landing page. Click the “Create parameter” button.

On the create parameter page, enter in a name for the value. This name will be how you retrieve the value and is case-sensitive. For example, you can name this “db_username” or “slack_token”.

In the description, enter some text to help you remember what this value is for.

For most secrets, you can stay on the “Standard” tier.

For the type, for credentials, passwords, and secrets, you will want to use “SecureString”. The SecureString type is for any sensitive data that needs to be stored and referenced in a secured way. This uses AWS Key Management Service to encrypt the value. For getting started, you can leave all the defaults here.

Lastly, enter in the value for this secret.

If you have a lot of values in Parameter Store, you can use tags to help organize them. For this value, I will skip adding tags.

Now, click the “Create parameter” button at the bottom of the page. You will see your new parameter in the “My Parameters” table.

Creating the Lambda Function

First, let’s go ahead and create a new Lambda function. If you’ve never set up one before, check out my “How To Create Your First Python AWS Lambda Function” article.

Next, let’s start writing our code for this function. The AWS Lambda functions include the boto3 package in the environment. This AWS SDK allows us to access most AWS services. We can use the client method to load in a specific client object as shown here:

**import** **boto3**

aws_client = boto3.client('ssm')

Here, we are loading in the Simple Systems Manager client which allows us to retrieve parameters from the Parameter Store by using the get_parameter method as shown in this code:

**import** **boto3**

*# Create our SSM Client.*
aws_client = boto3.client('ssm')

*# Get our parameter*
response = aws_client.get_parameter(
    Name='example_secret',
    WithDecryption=**True**)

Since we are using an encrypted value, we need to set WithDecryption to true. The name field is case-sensitive and must exactly match the name you gave the parameter. This returns an object with several values as shown here:

{
    'Parameter': {
        'Name': 'example_secret',
        'Type': 'SecureString',
        'Value': 'examplepassword',
        'Version': 1,
        'LastModifiedDate': datetime.datetime(2021, 10, 30, 17, 45, 50, 660000, tzinfo=tzlocal()),
        'ARN': 'arn:aws:ssm:us-east-1:647709874538:parameter/example_secret',
        'DataType': 'text'
    },
    'ResponseMetadata': {
        'RequestId': '48a84d0a-bad9-415a-b13f-6de2814f4330',
        'HTTPStatusCode': 200,
        'HTTPHeaders': {
            'server': 'Server',
            'date': 'Sat, 30 Oct 2021 18:19:41 GMT',
            'content-type': 'application/x-amz-json-1.1',
            'content-length': '220',
            'connection': 'keep-alive',
            'x-amzn-requestid': '48a84d0a-bad9-415a-b13f-6de2814f4330'
        },
        'RetryAttempts': 0
    }
}

Now, let’s put this within a Lambda handler:

**import** **boto3def** lambda_handler(event, context):
    aws_client = boto3.client('ssm')

    *# Get our parameter*
		response = aws_client.get_parameter(
        Name='example_secret',
        WithDecryption=**True**)

    **return** f'Our value is **{**response['Parameter']['Value']**}**'

Important Note: You should never print or log credentials or secrets in production code. I am doing it here to demonstrate how to retrieve the value and to show it’s working.

Inside the Lambda function admin area, paste the code above into the “Code” tab as shown below:

Once pasted in, click the “Deploy” button to save it.

Now, if we tried to run this code, we will get an error that our function is not authorized to perform the getParameter action. So, let’s give it that permission.

Adding the Permissions to the Lambda Function

Within AWS, there are a lot of systems and securing strategies for users and services. I will not be going into these in this article but I encourage you to read AWS’s “What is IAM?” for more information.

For this Lambda function, we are going to attach the permission needed to the Lambda’s role. To do so, go to the “Configuration” tab on the Lambda and select “Permissions” from the sidebar.

Within the “Execution role” panel, click on the role name to open up that role in IAM.

Click the “Attach policies” button. On the add permissions screen, search for the “AmazonSSMReadOnlyAccess” permission. This will allow your Lambda function to read from Systems Manager.

Check the checkbox for the “AmazonSSMReadOnlyAccess” permission and then click “Attach policiy”.

Now, go back to the configuration screen for the Lambda function (or refresh it if you kept it open). In the resource summary drop down, you should now see the “AWS Systems Manager” listed. If you select it, you should see the actions list “Allow: ssm:Get*” with other permissions.

Our Lambda is now ready to test.

Testing Our Function

Switch to the “Test” tab for the Lambda function. Then, click the “Test” button.

You will see a new “Execution result: succeeded” panel appear. If we click on the details, we will see the response as well as log output.

We see our secret that was saved in Parameter Store has made it to the Lambda function and was returned in the response!

Next Steps

Now that you have stored values in Parameter Store and retrieved them from a Lambda function, you can start setting up Lambda functions that can access databases and other services. For example, you can set up a Twitter bot and keep your access keys in Parameter Store. Or, you can set up a recurring monitor that checks your database and sends results to Slack.