Jetpack revisited: Even faster Serverless packaging and deploys

Two weeks ago we introduced the serverless-jetpack plugin, a drop-in replacement for Serverless Framework packaging. We thought that we had sped up serverless CLI packaging and deployments as much as possible.

It turns out we were wrong.

We discovered we could make it even faster...

Reexamining the problem

As we discussed previously, serverless CLI packaging slows to a crawl when reading large node_modules directories on disk, only to later exclude most files from application packaging as devDependencies.

Our solution with serverless-jetpack was to avoid any introspection of node_modules by doing a yarn or npm production installation in a temporary directory. This takes a bit of time, but often far less than globbing on disk with built-in serverless packaging.

In the course of our initial release, we received great feedback from the community, including some undiscovered issues and new ideas as to how to tackle the speed problem.

Back to the drawing board

The existing design of Jetpack offers an easy means of replacing built-in serverless packaging with a new heuristic of our choosing. Jetpack also has a full benchmark and integration test suite to ensure that the plugin is faster than serverless while still correctly producing identical packaging output.

...so maybe we can take a different approach?

After a seemingly promising false start, we reexamined the problem. We need to not read devDependencies. How else could we do it?

Turns out that we have already tackled this problem in depth with our inspectpack project, which does detailed bundle and dependency analysis of Webpack-built JavaScript applications.

Using techniques pioneered in inspectpack, we created a small, focused library named inspectdep, which efficiently reads node_modules and discovers the on-disk locations of production modules. By hooking inspectdep into Jetpack we were able to:

1. Quickly identify production dependencies before globbing.
2. Use that information to skip disk I/O for devDependencies by reading only production dependencies in a project's node_modules when globbing.

In effect, Jetpack is able to do pretty much the same thing that serverless does, just much, much faster.

A faster, more powerful Jetpack

With this refactor, the severless-jetpack plugin is much simpler.

There are no yarn or npm installs anymore. All of the dependency inference is performed in pure JavaScript code. This also makes the plugin more compatible with a wider range of Serverless Framework applications.

There are no more configuration options! The plugin "just works" with existing include|exclude Serverless packaging configurations. If it works with serverless, it should work with serverless-jetpack.

So, how much faster is it? Let's check out some new benchmarks!

This time around, we have Windows VM timings from our new AppVeyor CI. The speedup for Windows seems to be even more dramatic—for the huge scenario, serverless-jetpack is over 15 times faster than built-in serverless packaging in our CI setup.

Fire it up!

We've been very pleased with the reception of serverless-jetpack since its introduction. Our internal refactoring should now make it better and easier to integrate in Serverless Framework applications. We hope the newer, shinier Jetpack speeds up your serverless packaging even more! 🚀