Build & deploy all AWS resources as easy as: npm run deploy-prod (AWS Sam worflow)
After manually managing my AWS resources I finally decided to use Sam templates and put everything in Git. Easy deploying to prod and test stages was also a requirement. Here is a short overview of some issues I ran into and how I resolved them. It is by no means a detailed step by step guide, but just a quick overview with some problems I ran into and solutions.
Disclaimer: This is my first experience with AWS Sam. I probably did some things sub-optimal or wrong. Idea is to share and get comments from those with more experience. It may also help those starting with AWS Sam, to speed up the process and save a week I spend investigating. Also note that current Sam version I'm using is 0.52.0 and I'm on Linux.
sam-cli installation
First requirement is sam-cli. I'm not getting into details of installing and configuring it, but you can find it here.
You will also have to add sam-cli user an permissions on AWS IAM. This can be a slow process of trial and error. Depending on your resources, and motivation to close down permissions, settings will vary. Basically you try to publish, read error, add permission, rinse and repeat. Many tutorials just tell you to use admin account but I don't like that. Here's my IAM role policy but you'll need to adapt it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cognito-idp:DeleteUserPool",
"cognito-idp:AddCustomAttributes",
"iam:UntagRole",
"iam:TagRole",
"iam:CreateRole",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"cloudformation:CreateChangeSet",
"apigateway:DELETE",
"cognito-idp:DescribeUserPool",
"iam:PassRole",
"iam:DetachRolePolicy",
"cloudformation:DescribeStackEvents",
"iam:DeleteRolePolicy",
"apigateway:PATCH",
"cloudformation:DescribeChangeSet",
"apigateway:GET",
"cloudformation:ExecuteChangeSet",
"iam:GetRole",
"apigateway:PUT",
"cognito-idp:UpdateUserPoolClient",
"iam:DeleteRole",
"cognito-idp:ListTagsForResource",
"cloudformation:DescribeStacks",
"cognito-idp:CreateUserPoolClient",
"apigateway:POST",
"cognito-idp:UpdateUserPool",
"iam:GetRolePolicy"
],
"Resource": [
"arn:aws:cognito-idp:<region>:<AccountID>:userpool/*",
"arn:aws:iam::<AccountID>:role/<MyApp>*",
"arn:aws:cloudformation:eu-west-1:<AccountID>:stack/jsbenchme*/*",
"arn:aws:cloudformation:eu-west-1:<AccountID>:stack/aws-sam-cli-managed-default/*",
"arn:aws:cloudformation:<region>:aws:transform/Serverless-2016-10-31",
"arn:aws:apigateway:<region>::/restapis/*",
"arn:aws:apigateway:<region>::/restapis"
]
},
{
"Effect": "Allow",
"Action": [
"cognito-idp:CreateUserPool",
"cloudformation:GetTemplateSummary"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "events:*",
"Resource": "arn:aws:events:iam::<AccountID>:rule/MyApp-*"
},
{
"Effect": "Allow",
"Action": "lambda:*",
"Resource": "arn:aws:lambda:iam:<region>:<AccountID>:function:<MyLambdaFunction>-*"
}
]
}
Organizing folders and files
I decided to put my AWS Serverless app templates into AWS subfolder of my project. Sam will create folder structure like this.
Run sam init and choose AWS Quick Start Templates, runtime (mine was Node) and Hello World Example. Folder structure like this will be created by Sam:
AWS
|-- events -> where events for testing go
|-- hello-world -> where your code goes (Lambda in my case)
`- template.yaml -> Sam template to deploy resources through CloudFormation
First, I renamed hello-world with Lambda and put my Lambda code there, deleted some files too.
Next step, which might not be as obvious at start, is splitting up template.yaml in different files. If you have many services (especially API), file becomes unmanageable very fast. so I created folder ./templates for my templates expecting to merge them easily when the time comes to build.
Issue #1 - merging AWS Sam templates
It turns out there were several issues here. First about ability of Sam to merge files, then with some references requiring resources to be in the same template and last (but not least for sure) a lot of issues I had in my API Gateway definition yaml with references to other properties. More on that later. Enough to say for now that I solved all those issues with a small tool json-refs. So, I just did npm i -D json-refs, split template in multiple files, put them in ./templates folder and now my entry point template ./templates/main.yaml looks something like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
My App
Globals:
Function:
Timeout: 10
Resources:
MyAppLambdaRole:
$ref: 'LambdaAPIhandler.yaml#/Role'
MyAppLambdaAPIhandler:
$ref: 'LambdaAPIhandler.yaml#/Handler'
MyAppDynamoDB:
$ref: 'DynamoDB.yaml'
MyAppAPIGateway:
$ref: 'APIGateway.yaml#/APIGateway'
MyAppFunctionLambdaPermission:
$ref: 'APIGateway.yaml#/LambdaInvocationPermission'
MyAppCognitoUserPool:
$ref: 'CognitoUserPool.yaml#/Pool'
MyAppCognitoClient:
$ref: 'CognitoUserPool.yaml#/Client'
This is much neater and manageable. But to make it work, we need to setup script to use json-refs and merge those files into ./template.yaml. I decided to stick with Bash...
json-refs build script
We need to point json-refs to ./templates/main.yaml and it will render all those $refs and produce merged ./template.yaml for Sam to use.
I started by creating another folder ./templates/build, (also put it in .gitignore). Then created a Bash script ./build-deploy.sh which started like this:
echo '[build] deleting build files'
rm -r ./template/build/*
echo '[build] copying templates to build folder'
cp -r ./template/*.yaml ./template/build/
echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml
echo '[build] cleaning up template.yaml'
sed -i 's/T00:00:00.000Z//g' ./template.yaml
It's all quite self-explanatory, except maybe the last line where I remove T00:00:00.000Z which gets appended to version-dates of the Sam template. This might be unnecessary, but I like to keep thing as they are in AWS docs and versions don't have timestamps there.
Writing AWS resource definition yaml files
Again, relatively simple. AWS Docs are your best friend. It goes slow but steady. Some issues along the way though...
Issue #2 - json-refs complains about custom tags like !Ref, !Sub...
Very easy fixable once you find out that !Ref can be replaced with Ref:, !Sub with Fn:Sub and !GetAtt with Fn::GetAtt. They are identical but your new build script will not complain. Just don't forget colon (:) after those.
Issue #3 - API Gateway definition gets huge and has repetitive entries, in some cases (inside DefinitionBody) Ref does not work
My biggest fear was to write yaml for all API Gateway paths and methods, knowing how much time I spent on it in AWS Gateway UI. Exporting Swagger and pasting it in template required some changes. Some references were broken and I failed to make them work. It might be possible, but since I already had json-refs in place, I decided to use that, it seemed easier. At the end, it turned out writing API in yaml was easier. So much stuff can be reused, just define bits and pieces within components in your yaml and reuse them with $ref. I've put individual method responses to components, lambda invocation properties, even complete repetitive response objects for many paths. Just as an illustration, it can look like this now (shortened!):
APIGateway:
Type: AWS::Serverless::Api
Properties:
Name: MyAPIGateway
DefinitionBody:
openapi: "3.0.1"
path:
"/me":
get:
x-amazon-apigateway-integration:
$ref: "#/components/x-amazon-apigateway-integrations/default-integration"
responses:
$ref: "#/components/responses/defaultResponseGroup"
# components parsed by json-refs
components:
responses:
defaultResponseGroup:
"404":
description: "404 response"
headers:
$ref: "#/components/headers/default"
content:
$ref: "#/components/content/default"
"200":
description: "200 response"
headers:
$ref: "#/components/headers/default"
content:
$ref: "#/components/content/default"
headers:
default:
Access-Control-Allow-Origin:
schema:
type: string
content:
default:
application/json:
schema:
$ref: "#/components/schemas/Empty"
Of course, my definition has many paths, methods and responses, but putting some default properties and grouping them into components made it really easy to write API, easier than using API Gateway UI!
Issue #4 - circular references
Well, I let Sam create unique names for my resources so their Arns are created when deployed. But I need LambdaFunction to reference LambdaFunctionRole.Arn and LambdaFunctionRole to reference LambdaFunction.Arn. Sam cannot create both at the same time, they depend on each other. AWS docs have a good article on this and on more complex scenarios, but my solution was to name my LambdaFunction manually, so I can refer to it by name. It can be useful later in deployment process for other things too. Read on. Also check DependsOn CloudFormation attribute docs, this will help ordernig creation of resources to avoid similar issues in referrences.
Automating build & deployment to separate stages (test and prod)
My initial impression was that I'll use stageName and alias properties of some resources to build prod and test stages but it turns out best practice is to separate stages into different stacks. Actually, many people suggest using different accounts but that's not always practical. So, I decided that each stage will have it's own stack. I did use stageName too to make it even more clear which resource belongs where, although it seems unnecessary because we will make stackName also contain stage name to differentiate different stacks.
General idea is to use single code base for both prod and test stage, but before the build process I need to be able to decide which stage I'm going to build and publish. So there must be a way to make template(s) dynamic, to make certain properties changeable from command line. CloudFormation parameters and Sam's --parameter-overrides argument is what we need. First, I added parameterss to my ./templates/main.yaml:
Parameters:
parAccountId:
Type: String
Description: AWS Account Id
Default: ""
parRegion:
Type: String
Description: AWS Region to deploy to
Default: ""
parStackName:
Type: String
Description: The name of the stack
Default: "myStack-test"
parStage:
Type: String
Description: "The name of the stage, must me one of: test, prod"
AllowedValues:
- test
- prod
Default: "test"
... then those parameters need to be used in templates in place of hard-coded ones (i.e. ${parStackName}).
Now is time to (maybe) make a step back. I'm actually editing this article after running into problems because I haven't done so. So, that step is to make naming convetion for your resources. Trust me, it will make your life easier if you just name all the resources manually, following that naming convention. Mine is:
<project-name>-<resource-name>-<stack>-<stage>
Go through all templates and implement this dynamic naming. Don't foget your sam-cli IAM role that you had to create at the beginning. This naming convention is going to make it a whole lot easier to avoid issues and resorting to '*'. Remember that Lambda function? Now part of its definition will look like:
Type: AWS::Serverless::Function
Properties:
FunctionName:
Fn::Sub: "MyProject-MyAppLambdaAPIhandler-${parStackName}-${parStage}"
Now, we can use sam build and sam deploy to build and deploy our resources. We will use --parameter-overrides argument to send those parameters to CloudFormation and they will affect template there (note! local template.yaml will NOT be changed based on params! This will happen on CloudFormation):
sam build --build-dir .aws-sam/build \
--template ./template.yaml \
--region "${region}" \
--parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"
sam deploy --template ./template.yaml \
--no-fail-on-empty-changeset \
--confirm-changeset \
--tags stack=${stackName} stage=${stage} \
--region "${region}" \
--stack-name "${stackName}" \
--parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
--capabilities CAPABILITY_NAMED_IAM
Of course, we will put this in our ./build-deploy.sh script which will take arguments and build those parameters with them, from command line. You can check it in the final script source at the end of the article.
Issue #5 - Sam requires S3 bucket to deploy
Well, actually this S3 bucket is made automatically, but ONLY in "guided" deploy mode which is activated by --guided argument of sam-build. In this mode, sam asks questions about some deployment parameters and creates S3 bucket auto-magically, then stores your choices to ./samconfig.yaml for later (add this file to .gitignore!). So, in our final ./build-deploy.sh script we will check for existence of this file and if it is not there, we will start deploy in "guided" mode. Just check the final code for details.
Issue #6 - Sam ignores --capabilities argument when in "guided" mode
We need --capabilities CAPABILITY_NAMED_IAM in my implementation and we do pass it to sam build as an argument. However, if it is in "guided" mode, it ignores --capabilities argument and asks you to enter the value manually (or uses wrong default). "Workaround" is also implemented in final ./build-deploy.sh.
Outputs
You will want to know some parameters created by AWS, which are not known before deploy is done. In example your API Gateway URI. We don't want anything hard-coded so there must be a way to fetch those after build & deploy. Options to the rescue. Again in ./main.yaml I added this:
Outputs:
apiURL:
Description: "production stage API URL"
Value:
Fn::Sub: "https://${MyAppAPIGateway}.execute-api.${parRegion}.amazonaws.com/${parStage}/"
This will create apiURL output visible in AWS CloudFormation UI. But how to access it from our build script and make it available to other parts of our app (in my case frontend React app)? Again, with Sam:
aws cloudformation describe-stacks --stack-name ${stackName} \
--query 'Stacks[0].Outputs[?OutputKey==`apiURL`].OutputValue' \
--output text
All we need to do now is to include this in our ./build-deploy.sh and add some code to save those values into a config file readable to our app (don't forget to .gitignore it too). Also don't forget to somehow differentiate same variables for different stages! Because ApiURL for prod stage and test stage should both be saved. You can see it all in final ./build-deploy.sh code where I also added more code to offer some help and arguments management:
#!/bin/bash
###################################################################################
#
# Bash script do build & deploy to different stages
#
###################################################################################
show-help () {
echo
echo "--------------------------------"
echo " AWS deployment script "
echo "--------------------------------"
echo
echo "usage: deploy.sh [options] <action>"
echo
echo "ACTION:"
echo " build Builds AWS app and stores template to ./template.yaml (Default action)"
echo " deploy Deploys AWS app to CloudFormation using ./template.yaml"
echo " build-deploy Builds and deploys in single step"
echo
echo "OPTIONS:"
echo " mandatory arguments:"
echo " --accountId= AWS account id to deploy to CloudFormation with"
echo " --region= AWS region to deploy to"
echo " --stackName= stack name"
echo " --stage=STAGE stack stage name to deploy to"
echo " STAGE is one of: prod, test"
echo
}
check-arguments () {
# action arg default
if [ -z "$action" ]
then action = "build"
fi
# check empty mandatory args
if [ -z "$accountId" -o -z "$region" -o -z "$stackName" -o -z "$stage" ]
then echo "Error! Mandatory arguments missing"
else
# check format of args
if [ "$stage" != "prod" -a "$stage" != "test" ]
then
echo "Error! --stage needs to be one of: prod, test"
exit 3
else
return
fi
fi
# output error specs
if [ -z $accountId ]
then echo " --accountId is a mandatory argument"
fi
if [ -z $region ]
then echo " --region is a mandatory argument"
fi
if [ -z $stackName ]
then echo " --stackName is a mandatory argument"
fi
if [ -z $stage ]
then echo " --stage is a mandatory argument"
fi
exit 2
}
aws-build () {
echo '[build] deleteing build files'
rm -r ./template/build/*
echo '[build] copying templates to build folder'
cp -r ./template/*.yaml ./template/build/
echo '[build] runing resolver on files in build folder and saving template to ./template.yaml'
./node_modules/.bin/json-refs resolve ./template/build/main.yaml > ./template.yaml
echo '[build] cleaning up template.yaml'
sed -i 's/T00:00:00.000Z//g' ./template.yaml
}
sam-build () {
sam build --build-dir .aws-sam/build \
--template ./template.yaml \
--region "${region}" \
--parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}"
}
sam-deploy () {
# check if samconfig.toml exists. If not, run --guided to autocreate S3 bucket and remember it
if [ -f "./samconfig.toml" ]
then isGuided=""
else
isGuided="--guided"
printf "This is the first deploy, Sam will be started in --guided mode.\n \
NOTE! \n\
Please accept default params except those: \n\
Allow SAM CLI IAM role creation [Y/n]: Select N! \n\
Capabilities [['CAPABILITY_IAM']]: Enter: CAPABILITY_NAMED_IAM \n\
This is to avoid a bug(?) in Sam which (only on guided run) uses default capabilities instead of ones passed as an argument. \n\
You can also leave defaults and (after erroring out) rerun deployment again. Secontd time it will use passed arguments and run ok. \n\
\n"
read -p "Press ENTER to continue."
fi
sam deploy "${isGuided}" \
--template ./template.yaml \
--no-fail-on-empty-changeset \
--confirm-changeset \
--tags stack=${stackName} stage=${stage} \
--region "${region}" \
--stack-name "${stackName}" \
--parameter-overrides parAccountId="${accountId}" parRegion="${region}" parStackName="${stackName}" parStage="${stage}" \
--capabilities CAPABILITY_NAMED_IAM
}
save-outputs () {
echo "getting outputs and storing them in configs..."
apiURL=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
--query 'Stacks[0].Outputs[?OutputKey==`apiURLprod`].OutputValue' \
--output text)
cognitoClientId=$(aws cloudformation describe-stacks --stack-name "${stackName}" \
--query 'Stacks[0].Outputs[?OutputKey==`cognitoClientId`].OutputValue' \
--output text)
if [ -z "$apiURL" -o -z "$cognitoClientId" ]
then
echo "Error! Error getting output params $apiURL $cognitoClientId"
exit 5
else
echo '{"apiURL": "'"$apiURL"'", "cognitoClientId": "'"$cognitoClientId"'"}' > ./options-${stage}.json
fi
}
while [ $# -gt 0 ]; do
case "$1" in
--accountId=*) accountId="${1#*=}"
;;
--region=*) region="${1#*=}"
;;
--stackName=*) stackName="${1#*=}"
;;
--stage=*) stage="${1#*=}" #allowed stage names are limited in template.yaml param definition, not here
;;
build) action="build";;
deploy) action="deploy";;
build-deploy) action="build-deploy";;
*) echo
echo "Error: invalid argument: $1"
echo "Use --help for list of arguments"
show-help
exit 1
esac
shift
done
check-arguments
echo "Running $action command of stack: $stackName, stage: $stage to region: $region using AWS accountID: $accountId"
if [ "$action" == "build" ]
then
aws-build
sam-build
fi
if [ "$action" == "deploy" ]
then
sam-deploy
save-outputs
fi
if [ "$action" == "build-deploy" ]
then
aws-build
sam-build
sam-deploy
save-outputs
fi
Now all that is left is to use our script in package.json to build&deploy (and/or other scenarios) like:
./build-deploy.sh --accountId=012345678 --region=eu-west-1 --stackName=MyApp-test --stage=test build-deploy