secrets-key-manager

secrets-key-manager can be used to populate ENV (.env) files using AWS parameter store (GCP Parameter Manager and Azure App Configuration coming soon).

Setup

To setup, here is a basic workflow how you can use this tool.

As admin

1. Authenticate

   • Decide your cloud provider AWS (Parameter Store), GCP (Parameter Manager), Azure (App Configuration) etc.
   • Get the key and secret or token from the IAM console
   • You will need read, write, list permissions for this IAM user

2. Save key-values to the the cloud provider

   • Use command setKey to save one key at a time

   secrets-key-manager setKey VARIABLE_NAME=variableValue --service WEB_APP --env dev [--description Desc]

   • Use command setKeys to save multiple keys from the file to the cloud provider

   secrets-key-manager setKeys --file /tmp/.envValues --service WEB_APP --env dev

   Uses the similar format as template environment variable file, without the {{}} syntax. You can use directives to change the behaviour of the key.

   # /tmp/.envValues
   
   # Lines starting with # are comments
   NODE_ENV=development
   PORT=3010 # This commend will be saved as description in the parameter store
   # MONGO_URI=<encrypt>mongodb://localhost:... # Encrypted using Secrets Manager (TBD)
   REDIS_URL=<no-service>localhost # saves to /dev/REDIS_URL
   HEALTH_CHECK=<no-env>true # saves /ANY_ENV/WEB_APP/HEALTH_CHECK
   CERT_PROVIDER=<no-env><no-service>lets-encrypt # saves as /CERT_PROVIDER
   WEB_APP/CUSTOM_PATH=<root>VALUE_AT_PATH # Advanced use cases, fetches for the path as-is without any prefixes
   ...

3. Create template environment variable (e.g. .envTemplate) file to populate values

   • Checked-in to your repository

   git add .envTemplate
   git commit -m "Adds .envTemplate file"

   • Can be of any type bash, json, yaml, xml etc.

4. Create script to populate environment file

   • e.g. for node projects, create setup:env script in package.json

   "setup:env:dev": "secrets-key-manager populateEnv --out .env --service SERVICE_NAME --env development"

5. Add the output file to .gitignore

   echo ".env" >> .gitignore
   git add .gitignore
   git commit -m "Adds .env to .gitignore"

As the developer

1. Your teammate now need to authenticate on their systems
   • Their IAM users will need read, list permissions
2. Use command populateEnv to get keys from the cloud provider and inject them to the template env file

   npm run setup:env:dev

   • This will create a .env (or whatever name you provide) file in the project root
   • This file will be used by the application to get the values

Commands

setKey

setKey set value of key in AWS parameter store

secrets-key-manager setKey VARIABLE_NAME=variableValue --env development [--description Desc] [--encrypt]

Note, if key already exist in parameter store, you can't set it from here. Delete the key from parameter store then try this step again.

setKeys

setKeys set multiple keys in AWS parameter store from a file

secrets-key-manager setKeys --file /tmp/.envValues --env development [--encrypt]

Note, if key already exist in parameter store, you can't set it from here. Delete the key from parameter store then try this step again.

populateEnv

populateEnv pick env names from .envList and create .env file

secrets-key-manager populateEnv --env development [--file .envList] [--out .env]

getKey

getKey get a single enviroment variable key in the terminal

secrets-key-manager getKey KEY_NAME --env development

Template File Syntax

Sample env template file

for this example env is dev, and service is WEB_APP

# Lines starting with # are comments
ENV # This commend will be saved as description in the parameter store
NODE_ENV=development
PORT=3010 #hardcoded
MONGO_URI={{<encrypt>MONGO_URI}} # fetches /dev/WEB_APP/MONGO_URI parameter value
REDIS_URL={{<no-service>REDIS_URL}} # fetches for dev env, but without service prefix /dev/REDIS_URL
HEALTH_CHECK={{<no-env>HEALTH_CHECK}} # fetches for without env prefix /ANY_ENV/WEB_APP/HEALTH_CHECK
CERT_PROVIDER={{<no-env><no-service>CERT_PROVIDER}} # fetches for without env or service prefix /CERT_PROVIDER
CUSTOM_PATH={{<root>/WEB_APP/CUSTOM_PATH}} # Advanced use cases, fetches for the path as-is without any prefixes
...

Syntax

• Use {{VAR_NAME}} to define variables that will be fetched from the store.
  • e.g. MONGO_URI={{MONGO_URI}}
• By default all the variables will be fetched for env and service provided in the cli or API.
• If you want to fetch variable that is not related to any service, or is generic, use NONE as service name.
  • e.g. MONGO_URI={{<no-service>MONGO_URI}}

Directives

• You can use some special directives in <> to change variable fetching behaviour.
  • <no-service>: Ignore --service flag and fetch variable without service name
  • <no-env>: Ignore --env flag and fetch variable without env name
  • <root>: Ignore --env and --service flags and fetch variable without any prefix
  • <no-env><no-service>: Ignore both --env and --service flags and fetch variable without any prefix
  • <encrypt>: Encrypt the value before getting from or saving to the parameter store

How env and service works under the hood

• env and service are not any special environment variables, but just make-up the full name of the parameter key, and works for all the cloud providers (AWS, GCP, Azure).
• env is used to define the environment, e.g. dev, staging, prod.
• service is used to define the service name, e.g. WEB_APP, API, etc.
• env and service are used to create a hierarchy in the parameter store, but how they are stored in parameter store is different for different cloud providers.

AWS

• AWS provides Parameter Store, which supports hierarchy in the key name.

• Parameters that need encryption are encrypted using Secrets Manager.

  CLI Flags	Saved Parameter Name
  Default	/<env=ANY_ENV>[/service]/<name>
  --env and --service provided	/<env>/<service>/<key> e.g. /dev/WEB_APP/MONGO_URI
  --service is not provided	/<env>/<name> e.g. /dev/REDIS_URI
  --env is not provided	/ANY_ENV/<service>/<name> e.g. /ANY_ENV/WEB_APP/MONGO_URI
  both --env and --service are not provided	/ANY_ENV/<name> e.g. /ANY_ENV/CERT_PROVIDER
  --root flag is provided	<name> e.g. CERT_PROVIDER or /ANY/PATH/TO/KEY

GCP

• Google Cloud does not support hierarchy in parameter manager, so env and service are used as part of the key name, separated by -.

• There is a limit of 63 characters for key name, so env and service should be short.

  • e.g. use dev instead of development, WEB_APP instead of WEB_APPLICATION.
  • this service will throw an error if the key name (including env and service name) exceeds 63 characters.

• Allowed characters: A-Z, a-z, numbers (0-9), dashes (-), and underscores (_)

• Although GCP accepts dashes - but if you are using this tool to manage parameters, it will throw error to prevent you from using - as it is used as separator to create hierarchy.

  CLI Flag	Saved Parameter Name
  Default	<env=ANY_ENV>-[<service>-]<name>
  --env and --service provided	<env>-<service>-<key> e.g. dev-WEB_APP-MONGO_URI
  --service is not provided	<env>-<name> e.g. dev-REDIS_URI
  --env is not provided	<service>-<name> e.g. ANY_ENV-WEB_APP-MONGO_URI
  --env and --service are not provided	ANY_ENV-<name> e.g. ANY_ENV-MONGO_URI
  --root flag is provided	<name> e.g. CERT_PROVIDER or /ANY/PATH/TO/KEY

Azure

• TBD

Future Plans

• Make application more secure by loading variables in memory instead of .env file.