Infrastructures in the public cloud can become very complex. You probably also want to run multiple environments (DTAP) of your infrastructure. This is where Infrastructure as Code (IaC) comes in handy. This means that you describe your infrastructure as code and with this code you can easily provision and update your resources.
Many of you will use Terraform for this - it's the de facto standard for multi-cloud IaC. Yes, you can use the services from cloud providers itself (AWS CloudFormation, GCP Cloud Deployment Manager or Azure Resource Manager), but these will only work on the corresponding cloud providers. I prefer to learn one IaC solution and use that for all cloud providers.
Terraform supports all major cloud providers very well. Modules are maintained and quite up to date with new cloud services and configuration options. HashiCorp - the company behind Terraform - decided to build their own DSL for describing your infrastructure in Terraform: HCL. It's a more human-friendly version of JSON (which was by the way never meant for humans to write and read, so that makes sense):
resource "google_storage_bucket" "bucket-for-videos" {
name = "bucket-for-videos"
location = "EU"
website {
main_page_suffix = "index.html"
not_found_page = "404.html"
}
}
It all looked very good to me so I decided to use Terraform for my new project. I was able to start fairly quickly, however during the process the HCL started to annoy me. Simple things that can be easily done in every programming language were suddenly difficult, or I needed to look it up. Things like:
- for loops
- variable substitution
- if/else logic
- imports
- custom logic
- auto completion, syntax checking (in IDE)
- dependencies
- testing
were suddenly a bit hard - or at least not intuitive - to accomplish. Yes, (most) of these things is possible in Terraform, but implementing it really destroys the clean syntax and makes your configuration way more complex than it needs to be.
So I started looking for an alternative and I found Pulumi , which combines the best of both worlds for me: allowing me to describe my infrastructure in Python code, while using the complete and well-maintained Terraform modules in the background (I don't want yet another company to write these modules). Creating a storage bucket now looks a bit more familiar:
import pulumi
from pulumi_gcp import storage
for purpose in ['photos', 'videos', 'docs']:
storage.Bucket(f'bucket-for-{purpose}')
We can do whatever Python-magic we want here.
Pulumi is open source and free to use, but then you have to take care of the storage of the state file yourself. Pulumi supports local files (can be version controlled, but only use this if you're working on your own to avoid conflicts) and object storage like AWS S3 and Google Cloud Storage.
I've been running Pulumi in production for a couple of months now on a medium sized project (~150 cloud resources per environment, 4 environments) and I'm quite happy so far. Funny side-note is the fact that because Pulumi's documentation isn't that comprehensive, I keep using the original Terraform module documentation. Pulumi supports all configuration parameters 1:1; they even link to the original Terraform module from their documentation.
➜ infra git:(master) ✗ pulumi up
Previewing update (tst):
Type Name Plan Info
pulumi:pulumi:Stack tst
~ └─ gcp:appengine:FirewallRule allow-nginx-proxy-external update [diff: ~sourceRange]
Resources:
~ 1 to update
117 unchanged
Do you want to perform this update? details
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:tst::infra::pulumi:pulumi:Stack::tst]
~ gcp:appengine/firewallRule:FirewallRule: (update)
[id=myproject/900]
[urn=urn:pulumi:tst::infra::gcp:appengine/firewallRule:FirewallRule::allow-nginx-proxy-external]
[provider=urn:pulumi:tst::infra::pulumi:providers:gcp]
~ sourceRange: "10.0.0.0/8" => "0.0.0.0/0"
Do you want to perform this update?
yes
> no
details
I'm curious to hear what your experiences with Pulumi (and Terraform) are, so please leave a comment.