Jetpack: blazingly fast Serverless packaging and deploys

May 14, 2019

We love the Serverless Framework at Formidable. We get to focus on application development while it takes care of details like packaging and deployment—all with a rich plugin ecosystem and a great technical community!

However, an oft-recurring pain point for Serverless projects is that packaging and deploying can be really, really slow.

We're thrilled to introduce the serverless-jetpack plugin, a drop-in replacement for built-in serverless CLI packaging. For many large, real-world Serverless JavaScript applications, faster packaging may be only a few lines of configuration away!

What slows down Serverless packaging?

A few months ago, a co-worker noticed that Serverless Framework deploys for a client project were taking more than 10 minutes in just the packaging stage. This made the developers miserable, and even worse, significantly hindered our production deployment speed.

We dug in to how the serverless CLI packages files and discovered that a typical / default packaging arrangement leads serverless to:

  1. Read nearly all of node_modules (and other sources) from disk into a preliminary file list.
  2. Then exclude files from the list detected as devDependencies. (And, perform additional disk I/O in node_modules to infer what are devDependencies to exclude.)

This approach hit us hard, as our client project had a git monorepo with many individually packaged Lambda functions, meaning lots of large node_modules directories driven primarily by devDependencies.

A simple, faster idea

So boiling the problem down: the vast bulk of files in node_modules read during packaging end up being excluded later because they're devDependencies. Then we thought, what if we didn't have to read all of those devDependencies? What if we somehow had the serverless CLI read just the production ones?

Well, we would need a tool to quickly get us production dependencies... But, as it turns out we have not one, but two! Modern npm and yarn, the ubiquitous package managers your project already uses, install production dependencies very quickly when used with a lockfile.

We postulated a theory: the cost of an additional, temporary yarn|npm production install would be far less than the current time the serverless CLI uses in excluding extraneous node_modules files post-read.

How it works

With this hunch in hand, we tried a simple packaging experiment:

  1. Do a fresh production yarn|npm install into a temporary directory.
  2. Do normal exclude/includes on project sources except node_modules, instead matching against the production one in our temporary directory.

The results were amazing. Across various packaging [scenarios][jp_repo_scenarios], we were able to see consistent, impressive packaging speedups. And the speedups increased as development dependencies increased.

Some timings for relative comparison from our project benchmark:

ScenarioModeTypeTimevs Base
simpleyarnjetpack5687-42.16 %
simpleyarnbaseline9832
simplenpmjetpack6163-31.37 %
simplenpmbaseline8980
individuallyyarnjetpack8259-50.44 %
individuallyyarnbaseline16665
individuallynpmjetpack9787-50.10 %
individuallynpmbaseline19613
hugeyarnjetpack13473-71.85 %
hugeyarnbaseline47864
hugenpmjetpack9451-71.45 %
hugenpmbaseline33105

(The Mode column has clickable links to scenario projects on GitHub. For Type, baseline is built-in Serverless Framework packaging, jetpack is our new plugin. Time is in milliseconds.)

Try it out!

... and thus the serverless-jetpack plugin was born!

Installation should be a breeze for most projects. Jetpack works seamlessly with built-in Serverless packaging configuration, including service and individual function-level configurations with exclude, include, etc.

yarn and npm users should add this to serverless.yml:

plugins: - serverless-jetpack

and npm users will also need:

custom: serverless-jetpack: mode: npm # (default `yarn`)

And that's mostly it!

We take correctness seriously at Formidable. In addition to the usual tests and style checks, we validate that Jetpack's output in each scenario identically matches that of built-in Serverless packaging.

As a few parting thoughts, you should be using yarn with a yarn.lock file or npm@5.7.0+ with a package-lock.json to get a full speedup. And, there are a few esoteric complexities and additional configuration options that you may want to review before integration.

All in all, if Serverless Framework packaging/deployment speed is cramping your style, hook up a jetpack and see if your deploys take off! 🚀

Related Posts

Investing in Your Technology and Team But Still Not Seeing the Desired Results?

September 9, 2021
Where do you and your company fall on the digital maturity scale? And why does it matter?
Sam Reffitt

trace-pkg: Package Node.js apps for AWS Lambda and beyond

December 15, 2020
Packaging Node.js applications for the cloud can be slow and tedious. We introduce trace-pkg, a general purpose tool to quickly and efficiently package up your application code and dependencies for deployment in AWS Lambda and beyond!

Say Hello to Charlie

October 30, 2020
Charlie is a full-stack engineer with a broad range of experience leading large and small teams. Today he explains what he does at Formidable and why he stays.