Infrastructure as Code in TypeScript, Python, Go, C#, Java, or YAML with Pulumi

We cannot talk about Infrastructure as Code (IaC) without mentioning Terraform. It is one of the most popular Infrastructure as Code (IaC) technologies in the industry today. Pulumi is another IaC technology that is gaining in popularity (but our spell checker hates it). With Pulumi, you can code infrastructure in your language of choice whether it’s TypeScript, Python, Go, C#, Java, or YAML. Unlike in Terraform, with Pulumi you can use functions, classes, and packages. You can even use the same integrated development environment (IDE) and test the code with the same framework as your non-infrastructure code. Pulumi also comes with an excellent dashboard that is available by default and without any additional configuration.

We have created an open-source repository that deploys a collection of resources we commonly use here at Formidable. This repository has a Lambda function that has application code in an S3 bucket with logging to CloudWatch. Communicating with the Lambda is done through API Gateway, and caching with CloudFront. The templates for these resources are written in Pulumi and Terraform so that a direct comparison between the two can be made.

Let’s take a deeper look at some of the differences between the two technologies.

Language Support

The biggest difference between the two technologies is the languages that are used to define your infrastructure.

Terraform uses a domain-specific language (DSL) that is unique to Terraform: the HashiCorp Configuration Language (HCL). Pulumi does not have a DSL but uses a multitude of imperative languages including TypeScript, Python, Go, C#, Java, and YAML. These languages are imperative, but the resulting infrastructure is declarative, similar to Terraform.

Code Examples

// Creating a Lambda function in Terraform

resource "aws_lambda_function" "lambda_function" {
  function_name = "${local.lambda_name}"

  s3_bucket = aws_s3_bucket.code_bucket.id
  s3_key = aws_s3_object.lambda_zip.key
  source_code_hash = data.archive_file.zip_lambda_code.output_base64sha256

  handler = "index.handler"
  runtime = "nodejs16.x"
  role = aws_iam_role.lambda_role.arn

  depends_on = [
    aws_cloudwatch_log_group.lambda_log_group,
    aws_iam_role_policy_attachment.lambda_logging
  ]
}

Creating a Lambda function in Terraform. Full code here

// Creating a Lambda function in Pulumi
import * as aws from "@pulumi/aws";

lambdaFunction = new aws.lambda.Function(`${lambdaName}`, {
    name: lambdaName,
    role: lambdaRole.arn,
    handler: "index.handler",
    runtime: "nodejs16.x",

    s3Bucket: codeBucket.id,
    s3Key: lambdaCodeObject.key,
    sourceCodeHash: lambdaZipHash,
  }, {
    dependsOn: [
      lambdaRole,
      cloudWatch,
    ],
  });

Creating a Lambda function with Pulumi. Modified example from here

In both examples, we use a variable to store the name of the Lambda function. With Terraform, it’s local.lambda_name (Terraform has local and non-local variables) and we use a similar variable lambdaName with Pulumi. With both Terraform and Pulumi, you can also see references to other objects. In Terraform, we’re getting an S3 bucket’s ID with aws_s3_bucket.code_bucket.id. With Pulumi, we have the S3 bucket set to a variable (codeBucket) so we can get the ID with just codeBucket.id.

With Pulumi, we are also able to set resources to variables rather than having to remember the resource type and name as required with Terraform. Pulumi also makes debugging easier as we can insert a console.log for fast debugging and not have to figure out how to get troublesome sections to output with Terraform, or enable TF_LOG for debugging.

Once you strip away the instantiation, variable references, and references to other resources, you’ll notice the syntax for declaring a resource to be pretty similar. You have to change snake_case to camelCase, but it is extremely easy to switch over from Terraform to Pulumi once you can navigate the surrounding syntax.

Language Comparisons

With Terraform, the HCL language is much more strict compared to the languages available with Pulumi. With Pulumi, you are more likely to be using standard non-HCL programming languages. This allows you to create your own classes, control flow, and code blocks.

With this power comes great responsibility as it also allows you to introduce bad programming practices. There is not much room for “creativity” with the HCL language as there are limited ways to accomplish specific tasks. This makes the HCL language consistent and easy to read (if you are familiar with the language) regardless of who wrote it. With Pulumi, there is no limit to the creativity (and therefore complexity) of the resulting code.

There are times with Terraform where the ability to write a loop or condition statement would be preferred over figuring out Terraform’s built-in functions. An example of this:

aws_accounts = {
  "account-a" = {
    email      = "account-a@formidable.com"
    first_name = "Account"
    last_name  = "A"
    managed_policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
    users = [
      "user-c@formidable.com"
    ]
  }

  "account-b" = {
    email      = "account-b@formidable.com"
    first_name = "Account"
    last_name  = "B"
    users = [
      "user-d@formidable.com"
    ]
  }
}

locals {
  aws_account_user_associations = flatten([
    for account_key, account in local.aws_accounts : flatten([
      for user in lookup(account, "users", []) : {
        account = lower(account_key)
        user    = user
      }
    ])
  ])

	aws_account_users = {
    for user in distinct(local.aws_account_user_associations[*].user) : user => [
      for association in local.aws_account_user_associations : association.account if association.user == user
    ]
  }
}

HCL Insanity (not from the Terraform/Pulumi example repository)

This is an extreme example that would be significantly simplified (and much easier to read) if we were able to use a language like TypeScript. In this example, we are creating two local variables (aws_account_user_associations and aws_account_users) based on the aws_accounts variable due to Terraform’s limitations with iterating.

On the other hand, when working with Pulumi, there are times when Terraform’s built-in functions would have been preferred.

data "archive_file" "zip_lambda_code" {
  type = "zip"
  source_file = "${var.working_dir}/../lambda/index.js"
  output_path = "${var.working_dir}/lambda.zip"
}

resource "aws_lambda_function" "lambda_function" {
	...
  source_code_hash = data.archive_file.zip_lambda_code.output_base64sha256
	...
}

Creating a Zip and Getting an Encoded Hash in Terraform. Full code here

const lambdaZip = new AdmZip();
lambdaZip.addLocalFolder("../lambda");
lambdaZip.writeZip(lambdaZipName);

const crypto = require('crypto');
const fs = require('fs');
const fileBuffer = fs.readFileSync(lambdaZipName);
const lambdaZipHash = crypto.createHash('sha256').update(fileBuffer).digest('base64');

For users that are unfamiliar with JavaScript, this may require internet searching and experimentation to wrangle the code into a working state.

With Terraform, it’s easy to split larger projects into reusable modules and to separate resources into separate files. This is possible to do with Pulumi, but I found it much easier with Terraform. There is also this workaround that works with Pulumi, but it does not follow best programming practices.

If you want to test your infrastructure code, with Terraform you’ll have to use a third-party tool such as Terratest or Kitchen-Terraform as it’s not possible to write unit or integration tests with Terraform alone, other than variable validation. Testing Pulumi code is well documented and straightforward if you have written unit tests for that specific language previously.

Where Can You Use Terraform and Pulumi

With both of these tools, you can run with (some of these I had never heard of before):

• Amazon Web Services (AWS)
• Azure
• Google Cloud Platform (GCP)
• Kubernetes
• Openstack
• Alibaba Cloud
• Oracle Cloud Infrastructure
• Hashicorp Cloud Platform
• phoenixNAP Bare Metal (with Terraform it’s not an official module, but it is verified)

Terraform has the ability to span multiple cloud providers in a single platform, which you cannot do with Pulumi.

Cost

Pulumi is free for a single user. If you have multiple users, you have to purchase plan. With Terraform, if you use Terraform Cloud it is free for up to five users. State management is one of the most common reasons for using Terraform Cloud (more on this shortly). If you manage the state on your own infrastructure, you can remove the most common need for Terraform Cloud. Although there is some cost related to this (typically an S3 bucket and a DynamoDB table), the price is close to negligible due to the small amount of data being stored with these resources and the limited traffic associated with them.

With Pulumi, the team plan is good for up to 15 members, but if you have more than 15, you have to upgrade to their enterprise plan, requiring a custom quote. With Terraform, you can get around these costs by just self-managing the state on your own resources.

State Management

When you generate a Terraform plan (the changes that Terraform intends to make) it compares three sources:

• Your Terraform templates
• The current state of your deployed resources
• The state of your resources after the last time you ran Terraform

Based on these three “sources of truth” it comes up with a plan to get your deployed resources to match your Terraform templates. The state of your resources after the last time you ran Terraform, is generally considered to be the “state”.

By default, Terraform stores your state locally which is not ideal if you’re working with other engineers. Terraform recommends their Terraform Cloud/Enterprise service which allows Hashicorp to store your state with them (similar to how Pulumi does it).

Most companies working with Terraform and AWS store their state in an S3 bucket (cloud storage) and then store state locks in a DynamoDB (database) table. The problem with this solution is that you need infrastructure in place to manage your infrastructure (some potential solutions are to have a separate Terraform stack for just these resources with that state stored locally, a CloudFormation stack, or to use a third-party wrapper like terragrunt). Here is an example of storing the state with terragrunt (if you’re just using Terraform, it’s pretty similar):

remote_state {
  backend = "s3"

  config = {
    region = local.config.environment.aws_region

    bucket  = "${local.config.environment.project_name}-${local.config.environment.unique_identifier}-remote-state"
    key     = "terraform.tfstate"
    encrypt = true

    dynamodb_table = "${local.config.environment.project_name}-${local.config.environment.unique_identifier}-remote-locks"
  }
}

Storing state with terragrunt. Full code here

With Pulumi, the state is always stored with Pulumi—you don’t have any choice in the matter. This makes starting a Pulumi stack easier but does require that you create an account with Pulumi and generate a token (side note: if you create an account with an email, there’s no multi-factor authentication (MFA) option, so you may want to use the GitHub, GitLab, Atlassian, or SAML single sign-on (SSO) option for enhanced security). If you want to share state with multiple users, you have to purchase a plan. State locking (prevents multiple users from trying to update infrastructure at the same time, and potentially with conflicting templates) is unavailable without a plan.

Pulumi also has an excellent dashboard that allows you to see your stacks, resources (in a list or graph view), as well as an activity log.

Example of Pulumi Dashboard

Terraform and Pulumi both have multiple commands for updating and managing the state. With both tools, you can manually remove or import resources into the state, as well as refresh (update the state to match what’s currently “reality”) the state.

The need to refresh the state with Terraform is less common as it is done automatically when Terraform commands are run. With Pulumi, the need is more frequent, especially when manual changes are made to the infrastructure. When you delete a resource, Pulumi still assumes that the resource exists and won’t recreate the resource until the state is refreshed (this can turn into a debacle if you try to remove the resource from the template file as Pulumi will attempt to delete the non-existing resource).

Terraform also can taint and untaint. This is generally used infrequently, but it allows resources to be manually marked as needing to be replaced. Pulumi does not have this functionality, however.

Documentation and Support

Terraform is significantly more established and popular than Pulumi, contributing to more robust and complete documentation. Terraform’s official documentation is superb and has been used as an example of what documentation should look like. With Terraform’s popularity comes more support and community advice and activity. It’s much easier to find search results for Terraform issues compared to Pulumi when problems arise.

Pulumi’s documentation is great if you are using common resources on a more popular cloud platform. Most of the time, it has everything needed.

An interesting example of Pulumi’s documentation is Lambda functions. In Pulumi’s documentation for Lambdas, under sourceCodeHash, there is a reference to a function called filebase64sha256. Searching for this function just returns Terraform references. If you include Pulumi in the search query, the only reference that comes up is a GitHub action run result. Looking up the #391 reference, you can find this pull request. It appears it is possible to use this function with Pulumi’s Terraform bridge, but we ended up implementing this functionality using native TypeScript (example from earlier).

So Which Should You Use?

Whether to use Terraform or Pulumi depends on your circumstances. Below, you’ll find situations which may favor using one of the technologies over the other:

• Situations Favoring Terraform:

  • The team is already familiar and established with Terraform
  • Need to share templates/resources across applications and be able to update them in a single place (modules)
  • Using less common or non-standard resources and/or cloud platforms
  • A desire for templates to be accessible to all engineers or developers, regardless of their experience level
  • Working in a multi-cloud environment
  • The team does not have a defined coding style or pattern guideline for their programming language of choice
  • Cost is a concern (it’s possible to run Terraform in a large team environment with limited costs)

• Situations Favoring Pulumi:

  • The team consists of experts in TypeScript, Python, Go, C#, Java or YAML

  • The team has a defined coding style or pattern guideline for their programming language of choice

  • Want to allow for flexibility in templates allowing for multiple paths to the same solution

  • Better unit and integration testing support out of the box

  • Better integration with IDEs

  • Want to have a dashboard to easily see deployed resources and configurations (without having to install third-party tools)

    Here is a side-by-side comparison of the two tools, showing the differences.

  	Terraform	Pulumi
  Languages Supported	HCL	TypeScript, Python, Go, C#, Java, or YAML
  Cost	Terraform Cloud is free for up to 5 users, but Terraform Cloud is not necessary if you manage the state on your own platform (S3 bucket and DynamoDB as an example)	A Pulumi plan is required if you are using more than one user
  Biggest Advantages	Clean and standardized templates, more robust documentation and better module/code reuse	Code is written in a native language allowing for more flexibility and has a great dashboard GUI
  State Management	Local, through Terraform Cloud or self-managed remotely	Through Pulumi
  Flexibility	Limited due to use of HCL	The sky is the limit…or the limit does not exist (for all you Mean Girls fans out there)
  Readability	Code inflexibility helps ensure code is clean and readable	Depends on how the code is written and familiarity with the language
  Multi-Cloud Support	Multiple cloud providers can be used in the same environment	Only one cloud provider can be used per environment
  Documentation and Support	Extensive and comprehensive	Not as complete and doesn’t have as extensive community support as Terraform
  Code Reuse	Extensive functionality for reusing code and creating modules	Limited ability to reuse code with higher level extensions
  Testing Support	Third party tools are available for testing	Can use any framework supported by the programming language
  Dashboard / Visualization	Available through third parties	Excellent GUI dashboard is available without any configuration