How To: Manage Serverless Environment Variables Per Stage

I often find myself creating four separate stages for each ⚡ Serverless Framework project I work on: dev, staging, prod, and local. Obviously the first three are meant to be deployed to the cloud, but the last one, local, is meant to run and test interactions with local resources. It's also great to have an offline version (like when you're on a plane ✈ or have terrible wifi somewhere). Plus, development is much faster because you're not waiting for round trips to the server. 😉

A really great feature of Serverless is the ability to configure ENVIRONMENT variables in the serverless.yml file. This lets us store important global information like database names, service endpoints and more. We can even reference passwords securely using AWS's Service Manager Parameter Store and decode encrypted secrets on deployment, keeping them safe from developers and source repositories alike. 😬 Just reference the variable with ${ssm:/myapp/my-secure-value~true} in your configuration file.

Using STAGES and ENVIRONMENT variables together can create a really powerful workflow for you and your development team

I think sls invoke local -f myFunction -p /path/to/event.json is one of the most useful commands in my toolbox. Not only can you live test functions locally by simulating events, but you can completely manipulate the environment by passing in the -s flag with a stage name.

For example, if I was writing a script that interacts with a database (perhaps querying data for a report), I would most likely create a local database and point my MYSQL_HOST environment var to localhost (along with some other configs). Now running sls invoke local -f myDBFunction -p /path/to/event.json -s local would run my query against my local version. However, if I change my -s flag to dev, then I want my code to access the "dev" version of my database (which is perhaps in the cloud). This is useful for testing query and compatibility changes.

This is also great for letting you change other resources based on STAGE like SQS, S3 buckets, Dynamo DB tables, etc.

How do we configure our serverless.yml to do that?

Another great feature of the Serverless framework is your ability to "self-reference" variables within the serverless.yml file. This gives us the ability to use static (or even recursively referenced) values to set other values. I'm sure you've used this while naming functions, e.g. name: ${opt:stage}-myFunction You can also set a default value if the reference doesn't exist, e.g. stage: ${opt:stage,'dev'}, which is incredibly handy. 👍

In our case, we want to provide a list of possible options based on the STAGE provided. This can be accomplished in a number of ways. The documentation even gives you the example of including a separate file based on the STAGE name, but it is even easier than that. All you need to do is create an object under your custom: variables and provide a value for each stage:

yamlCopy code
# Custom Variables
custom:
  mysqlHost:
    local: localhost
    dev: devdb.example.com
    staging: ${ssm:/myapp/staging/mysql-host} #get from ssm
    prod: ${ssm:/myapp/prod/mysql-host} #get from ssm

Now simply self-reference the correct object key in your environment: variables section:

yamlCopy code
# Environment Variables
environment:
  MYSQL_HOST: ${self:custom.mysqlHost.${self:provider.stage}}

And that's it! Now whenever you use the -s local flag your database host will be "localhost". When you change the stage flag, so too will your host value.

Below is a more complete example:

yamlCopy code
# Serverless Config
service: myapp

# Provider
provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage,'dev'}
  ...

  # Environment Variables
  environment:
    MYSQL_HOST: ${self:custom.mysqlHost.${self:provider.stage}}
    MYSQL_USER: ${self:custom.mysqlUser.${self:provider.stage}}
    MYSQL_PASSWORD: ${self:custom.mysqlPassword.${self:provider.stage}}
    MYSQL_DATABASE: ${self:custom.mysqlDatabase.${self:provider.stage}}
    MYSQL_PORT: ${self:custom.mysqlPort.${self:provider.stage}}

# Custom Variables
custom:
  stages:
    - dev
    - staging
    - prod
  mysqlHost:
    local: localhost
    dev: devdb.example.com
    staging: ${ssm:/myapp/staging/mysql-host} #get from ssm
    prod: ${ssm:/myapp/prod/mysql-host} #get from ssm
  mysqlUser:
    local: root
    dev: myapp_devuser
    staging: myapp_stag
    prod: myapp
  mysqlPassword:
    local: root
    dev: ${ssm:/myapp/dev/mysql-password~true} #get from ssm (secure)
    staging: ${ssm:/myapp/staging/mysql-password~true} #get from ssm (secure)
    prod: ${ssm:/myapp/prod/mysql-password~true} #get from ssm (secure)
  mysqlDatabase:
    local: myapp_testdb
    dev: myapp_dev
    staging: myapp_staging
    prod: myapp_prod
  mysqlPort:
    local: '8889'
    dev: '3306'
    staging: '3306'
    prod: '3306'

# Plugins
plugins:
  - serverless-stage-manager

...

Where do we go from here?

This technique works for CI/CD systems as well. If your production environment is in a separate account, providing access to shared secrets will stay secure.

If you want to be able to access cloud services that are in a VPC, you can always create additional stages like dev_local. Then you could access remote resources through a VPN or use SSH tunnels to access resources behind a VPC. You might use port forwarding, for example, to direct MySQL traffic to localhost through to your VPC RDS instance.

If you want to save yourself from misspelling stage names, you can check out Serverless Stage Manager. This allows you to restrict the stage names used for full-stack and function deployments.

I hope you found this useful. Good luck and Go Serverless! 🤘🏻