Jetpack revisited: Even faster Serverless packaging and deploys

May 28, 2019

Two weeks ago we introduced the serverless-jetpack plugin, a drop-in replacement for Serverless Framework packaging. We thought that we had sped up serverless CLI packaging and deployments as much as possible.

It turns out we were wrong.

We discovered we could make it even faster...

Reexamining the problem

As we discussed previously, serverless CLI packaging slows to a crawl when reading large node_modules directories on disk, only to later exclude most files from application packaging as devDependencies.

Our solution with serverless-jetpack was to avoid any introspection of node_modules by doing a yarn or npm production installation in a temporary directory. This takes a bit of time, but often far less than globbing on disk with built-in serverless packaging.

In the course of our initial release, we received great feedback from the community, including some undiscovered issues and new ideas as to how to tackle the speed problem.

Back to the drawing board

The existing design of Jetpack offers an easy means of replacing built-in serverless packaging with a new heuristic of our choosing. Jetpack also has a full benchmark and integration test suite to ensure that the plugin is faster than serverless while still correctly producing identical packaging output.

...so maybe we can take a different approach?

After a seemingly promising false start, we reexamined the problem. We need to not read devDependencies. How else could we do it?

Turns out that we have already tackled this problem in depth with our inspectpack project, which does detailed bundle and dependency analysis of Webpack-built JavaScript applications.

Using techniques pioneered in inspectpack, we created a small, focused library named inspectdep, which efficiently reads node_modules and discovers the on-disk locations of production modules. By hooking inspectdep into Jetpack we were able to:

  1. Quickly identify production dependencies before globbing.
  2. Use that information to skip disk I/O for devDependencies by reading only production dependencies in a project's node_modules when globbing.

In effect, Jetpack is able to do pretty much the same thing that serverless does, just much, much faster.

A faster, more powerful Jetpack

With this refactor, the severless-jetpack plugin is much simpler.

There are no yarn or npm installs anymore. All of the dependency inference is performed in pure JavaScript code. This also makes the plugin more compatible with a wider range of Serverless Framework applications.

There are no more configuration options! The plugin "just works" with existing include|exclude Serverless packaging configurations. If it works with serverless, it should work with serverless-jetpack.

So, how much faster is it? Let's check out some new benchmarks!

This time around, we have Windows VM timings from our new AppVeyor CI. The speedup for Windows seems to be even more dramatic—for the huge scenario, serverless-jetpack is over 15 times faster than built-in serverless packaging in our CI setup.

OSScenarioTypeTimevs Base
Macsimplejetpack4338-46.37 %
Macsimplebaseline8089
Macindividuallyjetpack2964-76.77 %
Macindividuallybaseline12760
Machugejetpack4524-84.03 %
Machugebaseline28321
Windowssimplejetpack2890-73.88 %
Windowssimplebaseline11063
Windowsindividuallyjetpack3110-83.90 %
Windowsindividuallybaseline19312
Windowshugejetpack3500-93.34 %
Windowshugebaseline52548

Fire it up!

We've been very pleased with the reception of serverless-jetpack since its introduction. Our internal refactoring should now make it better and easier to integrate in Serverless Framework applications. We hope the newer, shinier Jetpack speeds up your serverless packaging even more! 🚀

Related Posts

Investing in Your Technology and Team But Still Not Seeing the Desired Results?

September 9, 2021
Where do you and your company fall on the digital maturity scale? And why does it matter?
Sam Reffitt

trace-pkg: Package Node.js apps for AWS Lambda and beyond

December 15, 2020
Packaging Node.js applications for the cloud can be slow and tedious. We introduce trace-pkg, a general purpose tool to quickly and efficiently package up your application code and dependencies for deployment in AWS Lambda and beyond!

Say Hello to Charlie

October 30, 2020
Charlie is a full-stack engineer with a broad range of experience leading large and small teams. Today he explains what he does at Formidable and why he stays.