Build & deploy all AWS resources as easy as: npm run deploy-prod (AWS Sam worflow)

After manually managing my AWS resources I finally decided to use Sam templates and put everything in Git. Easy deploying to prod and test stages was also a requirement. Here is a short overview of some issues I ran into and how I resolved them. It is by no means a detailed step by step guide, but just a quick overview with some problems I ran into and solutions.

Disclaimer: This is my first experience with AWS Sam. I probably did some things sub-optimal or wrong. Idea is to share and get comments from those with more experience. It may also help those starting with AWS Sam, to speed up the process and save a week I spend investigating. Also note that current Sam version I'm using is 0.52.0 and I'm on Linux.

sam-cli installation

First requirement is sam-cli. I'm not getting into details of installing and configuring it, but you can find it here.

You will also have to add sam-cli user an permissions on AWS IAM. This can be a slow process of trial and error. Depending on your resources, and motivation to close down permissions, settings will vary. Basically you try to publish, read error, add permission, rinse and repeat. Many tutorials just tell you to use admin account but I don't like that. Here's my IAM role policy but you'll need to adapt it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DeleteUserPool",
                "cognito-idp:AddCustomAttributes",
                "iam:UntagRole",
                "iam:TagRole",
                "iam:CreateRole",
                "iam:AttachRolePolicy",
                "iam:PutRolePolicy",
                "cloudformation:CreateChangeSet",
                "apigateway:DELETE",
                "cognito-idp:DescribeUserPool",
                "iam:PassRole",
                "iam:DetachRolePolicy",
                "cloudformation:DescribeStackEvents",
                "iam:DeleteRolePolicy",
                "apigateway:PATCH",
                "cloudformation:DescribeChangeSet",
                "apigateway:GET",
                "cloudformation:ExecuteChangeSet",
                "iam:GetRole",
                "apigateway:PUT",
                "cognito-idp:UpdateUserPoolClient",
                "iam:DeleteRole",
                "cognito-idp:ListTagsForResource",
                "cloudformation:DescribeStacks",
                "cognito-idp:CreateUserPoolClient",
                "apigateway:POST",
                "cognito-idp:UpdateUserPool",
                "iam:GetRolePolicy"
            ],
            "Resource": [
                "arn:aws:cognito-idp:<region>:<AccountID>:userpool/*",
                "arn:aws:iam::<AccountID>:role/<MyApp>*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/jsbenchme*/*",
                "arn:aws:cloudformation:eu-west-1:<AccountID>:stack/aws-sam-cli-managed-default/*",
                "arn:aws:cloudformation:<region>:aws:transform/Serverless-2016-10-31",
                "arn:aws:apigateway:<region>::/restapis/*",
                "arn:aws:apigateway:<region>::/restapis"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:CreateUserPool",
                "cloudformation:GetTemplateSummary"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "events:*",
            "Resource": "arn:aws:events:iam::<AccountID>:rule/MyApp-*"
        },
        {
            "Effect": "Allow",
            "Action": "lambda:*",
            "Resource": "arn:aws:lambda:iam:<region>:<AccountID>:function:<MyLambdaFunction>-*"
        }
    ]
}

Organizing folders and files

I decided to put my AWS Serverless app templates into AWS subfolder of my project. Sam will create folder structure like this.

Run sam init and choose AWS Quick Start Templates, runtime (mine was Node) and Hello World Example. Folder structure like this will be created by Sam:

AWS
 |-- events -> where events for testing go
 |-- hello-world -> where your code goes (Lambda in my case)
 `- template.yaml -> Sam template to deploy resources through CloudFormation

First, I renamed hello-world with Lambda and put my Lambda code there, deleted some files too. Next step, which might not be as obvious at start, is splitting up template.yaml in different files. If you have many services (especially API), file becomes unmanageable very fast. so I created folder ./templates for my templates expecting to merge them easily when the time comes to build.

Issue #1 - merging AWS Sam templates

It turns out there were several issues here. First about ability of Sam to merge files, then with some references requiring resources to be in the same template and last (but not least for sure) a lot of issues I had in my API Gateway definition yaml with references to other properties. More on that later. Enough to say for now that I solved all those issues with a small tool json-refs. So, I just did npm i -D json-refs, split template in multiple files, put them in ./templates folder and now my entry point template ./templates/main.yaml looks something like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    My App

Globals:
  Function:
    Timeout: 10

Resources:
  MyAppLambdaRole:
    $ref: 'LambdaAPIhandler.yaml#/Role'
  MyAppLambdaAPIhandler:
    $ref: 'LambdaAPIhandler.yaml#/Handler'
  MyAppDynamoDB:
    $ref: 'DynamoDB.yaml'
  MyAppAPIGateway:
    $ref: 'APIGateway.yaml#/APIGateway'
  MyAppFunctionLambdaPermission:
    $ref: 'APIGateway.yaml#/LambdaInvocationPermission'
  MyAppCognitoUserPool:
    $ref: 'CognitoUserPool.yaml#/Pool'
  MyAppCognitoClient:
    $ref: 'CognitoUserPool.yaml#/Client'

This is much neater and manageable. But to make it work, we need to setup script to use json-refs and merge those files into ./template.yaml. I decided to stick with Bash...

json-refs build script

We need to point json-refs to ./templates/main.yaml and it will render all those $refs and produce merged ./template.yaml for Sam to use. I started by creating another folder ./templates/build, (also put it in .gitignore). Then created a Bash script ./build-deploy.sh which started like this:

    echo '[build] deleting build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml

It's all quite self-explanatory, except maybe the last line where I remove T00:00:00.000Z which gets appended to version-dates of the Sam template. This might be unnecessary, but I like to keep thing as they are in AWS docs and versions don't have timestamps there.

Writing AWS resource definition yaml files

Again, relatively simple. AWS Docs are your best friend. It goes slow but steady. Some issues along the way though...

Issue #2 - json-refs complains about custom tags like !Ref, !Sub...

Very easy fixable once you find out that !Ref can be replaced with Ref:, !Sub with Fn:Sub and !GetAtt with Fn::GetAtt. They are identical but your new build script will not complain. Just don't forget colon (:) after those.

Issue #3 - API Gateway definition gets huge and has repetitive entries, in some cases (inside DefinitionBody) Ref does not work

My biggest fear was to write yaml for all API Gateway paths and methods, knowing how much time I spent on it in AWS Gateway UI. Exporting Swagger and pasting it in template required some changes. Some references were broken and I failed to make them work. It might be possible, but since I already had json-refs in place, I decided to use that, it seemed easier. At the end, it turned out writing API in yaml was easier. So much stuff can be reused, just define bits and pieces within components in your yaml and reuse them with $ref. I've put individual method responses to components, lambda invocation properties, even complete repetitive response objects for many paths. Just as an illustration, it can look like this now (shortened!):

APIGateway:
  Type: AWS::Serverless::Api
  Properties:
    Name: MyAPIGateway
    DefinitionBody:
      openapi: "3.0.1"
      path:
        "/me":
          get:
            x-amazon-apigateway-integration:
              $ref: "#/components/x-amazon-apigateway-integrations/default-integration"
            responses:
              $ref: "#/components/responses/defaultResponseGroup"

# components parsed by json-refs
components:
  responses:
    defaultResponseGroup:
      "404":
        description: "404 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
      "200":
        description: "200 response"
        headers:
          $ref: "#/components/headers/default"
        content:
          $ref: "#/components/content/default"
  headers:
    default:
      Access-Control-Allow-Origin:
        schema:
          type: string
  content:
    default:
      application/json:
        schema:
          $ref: "#/components/schemas/Empty"

Of course, my definition has many paths, methods and responses, but putting some default properties and grouping them into components made it really easy to write API, easier than using API Gateway UI!

Issue #4 - circular references

Well, I let Sam create unique names for my resources so their Arns are created when deployed. But I need LambdaFunction to reference LambdaFunctionRole.Arn and LambdaFunctionRole to reference LambdaFunction.Arn. Sam cannot create both at the same time, they depend on each other. AWS docs have a good article on this and on more complex scenarios, but my solution was to name my LambdaFunction manually, so I can refer to it by name. It can be useful later in deployment process for other things too. Read on. Also check DependsOn CloudFormation attribute docs, this will help ordernig creation of resources to avoid similar issues in referrences.

Automating build & deployment to separate stages (test and prod)

My initial impression was that I'll use stageName and alias properties of some resources to build prod and test stages but it turns out best practice is to separate stages into different stacks. Actually, many people suggest using different accounts but that's not always practical. So, I decided that each stage will have it's own stack. I did use stageName too to make it even more clear which resource belongs where, although it seems unnecessary because we will make stackName also contain stage name to differentiate different stacks. General idea is to use single code base for both prod and test stage, but before the build process I need to be able to decide which stage I'm going to build and publish. So there must be a way to make template(s) dynamic, to make certain properties changeable from command line. CloudFormation parameters and Sam's --parameter-overrides argument is what we need. First, I added parameterss to my ./templates/main.yaml:

Parameters:
  parAccountId:
    Type: String
    Description: AWS Account Id
    Default: ""
  parRegion:
    Type: String
    Description: AWS Region to deploy to
    Default: ""
  parStackName:
    Type: String
    Description: The name of the stack
    Default: "myStack-test"
  parStage:
    Type: String
    Description: "The name of the stage, must me one of: test, prod"
    AllowedValues:
    - test
    - prod
    Default: "test"

... then those parameters need to be used in templates in place of hard-coded ones (i.e. ${parStackName}). Now is time to (maybe) make a step back. I'm actually editing this article after running into problems because I haven't done so. So, that step is to make naming convetion for your resources. Trust me, it will make your life easier if you just name all the resources manually, following that naming convention. Mine is: <project-name>-<resource-name>-<stack>-<stage>

Go through all templates and implement this dynamic naming. Don't foget your sam-cli IAM role that you had to create at the beginning. This naming convention is going to make it a whole lot easier to avoid issues and resorting to '*'. Remember that Lambda function? Now part of its definition will look like:

  Type: AWS::Serverless::Function
  Properties:
    FunctionName:
      Fn::Sub: "MyProject-MyAppLambdaAPIhandler-${parStackName}-${parStage}"

Now, we can use sam build and sam deploy to build and deploy our resources. We will use --parameter-overrides argument to send those parameters to CloudFormation and they will affect template there (note! local template.yaml will NOT be changed based on params! This will happen on CloudFormation):

    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"

    sam deploy --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM

Of course, we will put this in our ./build-deploy.sh script which will take arguments and build those parameters with them, from command line. You can check it in the final script source at the end of the article.

Issue #5 - Sam requires S3 bucket to deploy

Well, actually this S3 bucket is made automatically, but ONLY in "guided" deploy mode which is activated by --guided argument of sam-build. In this mode, sam asks questions about some deployment parameters and creates S3 bucket auto-magically, then stores your choices to ./samconfig.yaml for later (add this file to .gitignore!). So, in our final ./build-deploy.sh script we will check for existence of this file and if it is not there, we will start deploy in "guided" mode. Just check the final code for details.

Issue #6 - Sam ignores --capabilities argument when in "guided" mode

We need --capabilities CAPABILITY_NAMED_IAM in my implementation and we do pass it to sam build as an argument. However, if it is in "guided" mode, it ignores --capabilities argument and asks you to enter the value manually (or uses wrong default). "Workaround" is also implemented in final ./build-deploy.sh.

Outputs

You will want to know some parameters created by AWS, which are not known before deploy is done. In example your API Gateway URI. We don't want anything hard-coded so there must be a way to fetch those after build & deploy. Options to the rescue. Again in ./main.yaml I added this:

Outputs:
  apiURL:
    Description: "production stage API URL"
    Value:
      Fn::Sub: "https://${MyAppAPIGateway}.execute-api.${parRegion}.amazonaws.com/${parStage}/"

This will create apiURL output visible in AWS CloudFormation UI. But how to access it from our build script and make it available to other parts of our app (in my case frontend React app)? Again, with Sam:

aws cloudformation describe-stacks --stack-name ${stackName} \
    --query 'Stacks[0].Outputs[?OutputKey==`apiURL`].OutputValue' \
    --output text

All we need to do now is to include this in our ./build-deploy.sh and add some code to save those values into a config file readable to our app (don't forget to .gitignore it too). Also don't forget to somehow differentiate same variables for different stages! Because ApiURL for prod stage and test stage should both be saved. You can see it all in final ./build-deploy.sh code where I also added more code to offer some help and arguments management:

#!/bin/bash

###################################################################################
#
# Bash script do build & deploy to different stages
#
###################################################################################

show-help () {
    echo 
    echo "--------------------------------"
    echo " AWS deployment script "
    echo "--------------------------------"
    echo 
    echo "usage: deploy.sh [options] <action>"
    echo 
    echo "ACTION:"
    echo "    build           Builds AWS app and stores template to ./template.yaml (Default action)"
    echo "    deploy          Deploys AWS app to CloudFormation using ./template.yaml"
    echo "    build-deploy    Builds and deploys in single step"
    echo 
    echo "OPTIONS:"
    echo "  mandatory arguments:"
    echo "    --accountId=   AWS account id to deploy to CloudFormation with"
    echo "    --region=       AWS region to deploy to"
    echo "    --stackName=   stack name"
    echo "    --stage=STAGE  stack stage name to deploy to"
    echo "                   STAGE is one of: prod, test"
    echo 
}

check-arguments () {
    # action arg default
    if [ -z "$action" ]
        then action = "build"
    fi

    # check empty mandatory args
    if [ -z "$accountId" -o -z "$region" -o -z "$stackName" -o -z "$stage" ]
        then echo "Error! Mandatory arguments missing"
    else
        # check format of args
        if [ "$stage" != "prod" -a "$stage" != "test" ]
        then
            echo "Error! --stage needs to be one of: prod, test"
            exit 3
        else
            return
        fi
    fi
    # output error specs
    if [ -z $accountId ]
        then echo " --accountId is a mandatory argument"
    fi
    if [ -z $region ]
        then echo " --region is a mandatory argument"
    fi
    if [ -z $stackName ]
        then echo " --stackName is a mandatory argument"
    fi
    if [ -z $stage ]
        then echo " --stage is a mandatory argument"
    fi
    exit 2
}

aws-build () {
    echo '[build] deleteing build files'
    rm -r ./template/build/*

    echo '[build] copying templates to build folder'
    cp -r ./template/*.yaml ./template/build/

    echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
    ./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml

    echo '[build] cleaning up template.yaml'
    sed -i 's/T00:00:00.000Z//g' ./template.yaml
}

sam-build () {
    sam build --build-dir .aws-sam/build \
                --template ./template.yaml \
                --region "${region}" \
                --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"
}

sam-deploy () {
    # check if samconfig.toml exists. If not, run --guided to autocreate S3 bucket and remember it
    if [ -f "./samconfig.toml" ]
        then isGuided=""
        else
            isGuided="--guided"
            printf "This is the first deploy, Sam will be started in --guided mode.\n \
                    NOTE! \n\
                    Please accept default params except those: \n\
                        Allow SAM CLI IAM role creation [Y/n]:   Select N! \n\
                        Capabilities [['CAPABILITY_IAM']]:   Enter: CAPABILITY_NAMED_IAM \n\
                    This is to avoid a bug(?) in Sam which (only on guided run) uses default capabilities instead of ones passed as an argument. \n\
                    You can also leave defaults and (after erroring out) rerun deployment again. Secontd time it will use passed arguments and run ok. \n\
                    \n"
            read -p "Press ENTER to continue."
    fi

    sam deploy "${isGuided}" \
              --template ./template.yaml \
              --no-fail-on-empty-changeset \
              --confirm-changeset \
              --tags stack=${stackName} stage=${stage} \
              --region "${region}" \
              --stack-name "${stackName}" \
              --parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
              --capabilities CAPABILITY_NAMED_IAM
}

save-outputs () {
    echo "getting outputs and storing them in configs..."
    apiURL=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`apiURLprod`].OutputValue' \
        --output text)

    cognitoClientId=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
        --query 'Stacks[0].Outputs[?OutputKey==`cognitoClientId`].OutputValue' \
        --output text)

    if [ -z "$apiURL" -o -z "$cognitoClientId" ]
    then
        echo "Error! Error getting output params $apiURL $cognitoClientId"
        exit 5
    else
        echo '{"apiURL": "'"$apiURL"'", "cognitoClientId": "'"$cognitoClientId"'"}' > ./options-${stage}.json
    fi
}


while [ $# -gt 0 ]; do
    case "$1" in
        --accountId=*) accountId="${1#*=}"
        ;;
        --region=*) region="${1#*=}"
        ;;
        --stackName=*) stackName="${1#*=}"
        ;;
        --stage=*) stage="${1#*=}" #allowed stage names are limited in template.yaml param definition, not here
        ;;
        build) action="build";;
        deploy) action="deploy";;
        build-deploy) action="build-deploy";;
        *)  echo 
            echo "Error: invalid argument: $1"
            echo "Use --help for list of arguments"
            show-help
            exit 1
    esac
    shift
done

check-arguments
echo "Running $action command of stack: $stackName, stage: $stage to region: $region using AWS accountID: $accountId"


if [ "$action" == "build" ]
    then
        aws-build
        sam-build
fi

if [ "$action" == "deploy" ]
    then
        sam-deploy
        save-outputs
fi

if [ "$action" == "build-deploy" ]
    then
        aws-build
        sam-build
        sam-deploy
        save-outputs
fi

Now all that is left is to use our script in package.json to build&deploy (and/or other scenarios) like: ./build-deploy.sh --accountId=012345678 --region=eu-west-1 --stackName=MyApp-test --stage=test build-deploy

Learn Something New Everyday,
Connect With The Best Developers!

Sign Up Now!

& 500k+ others use Hashnode actively.

Comments (2)

Chris Nagy's photo

I always wondered about the benefits of using a bash script for deployments vs using sam deploy lets say. Can you elaborate on the pros of using a seperate bash script? Since sam build && sam deploy is so easy to use now😅

Mirko Vukušić's photo

Full(stack|dev|life)

Well, bash script actually uses sam-build and sam-deploy if you look at it. However, it does a few other things too. Mostly, I used json-refs to be able to split template into smaller (managable) files AND also optimize yaml for API Gateway's DefinitionBody. $ref there can only be used ina few places and with json-refs I can use it anywhere, making my definition body really small and easy to write/extend. Second reason is to make it easier to include AWS build/deploy into main build/deploy script that runs the same for both frontend (Webpack) and backend (AWS). And all that for different stages. It seemed much cleaner to separate the two instead of using sam-build in main build script.