We love the Serverless Framework at Formidable. We get to focus on application development while it takes care of details like packaging and deployment—all with a rich plugin ecosystem and a great technical community!
However, an oft-recurring pain point for Serverless projects is that packaging and deploying can be really, really slow.
We're thrilled to introduce the serverless-jetpack plugin, a drop-in replacement for built-in serverless CLI packaging. For many large, real-world Serverless JavaScript applications, faster packaging may be only a few lines of configuration away!
What slows down Serverless packaging?
A few months ago, a co-worker noticed that Serverless Framework deploys for a client project were taking more than 10 minutes in just the packaging stage. This made the developers miserable, and even worse, significantly hindered our production deployment speed.
We dug in to how the serverless CLI packages files and discovered that a typical / default packaging arrangement leads serverless to:
- Read nearly all of
node_modules(and other sources) from disk into a preliminary file list. - Then exclude files from the list detected as
devDependencies. (And, perform additional disk I/O innode_modulesto infer what aredevDependenciesto exclude.)
This approach hit us hard, as our client project had a git monorepo with many individually packaged Lambda functions, meaning lots of large node_modules directories driven primarily by devDependencies.
A simple, faster idea
So boiling the problem down: the vast bulk of files in node_modules read during packaging end up being excluded later because they're devDependencies. Then we thought, what if we didn't have to read all of those devDependencies? What if we somehow had the serverless CLI read just the production ones?
Well, we would need a tool to quickly get us production dependencies... But, as it turns out we have not one, but two! Modern npm and yarn, the ubiquitous package managers your project already uses, install production dependencies very quickly when used with a lockfile.
We postulated a theory: the cost of an additional, temporary yarn|npm production install would be far less than the current time the serverless CLI uses in excluding extraneous node_modules files post-read.
How it works
With this hunch in hand, we tried a simple packaging experiment:
- Do a fresh production
yarn|npminstall into a temporary directory. - Do normal exclude/includes on project sources except
node_modules, instead matching against the production one in our temporary directory.
The results were amazing. Across various packaging [scenarios][jp_repo_scenarios], we were able to see consistent, impressive packaging speedups. And the speedups increased as development dependencies increased.
Some timings for relative comparison from our project benchmark:
| Scenario | Mode | Type | Time | vs Base |
|---|---|---|---|---|
| simple | yarn | jetpack | 5687 | -42.16 % |
| simple | yarn | baseline | 9832 | |
| simple | npm | jetpack | 6163 | -31.37 % |
| simple | npm | baseline | 8980 | |
| individually | yarn | jetpack | 8259 | -50.44 % |
| individually | yarn | baseline | 16665 | |
| individually | npm | jetpack | 9787 | -50.10 % |
| individually | npm | baseline | 19613 | |
| huge | yarn | jetpack | 13473 | -71.85 % |
| huge | yarn | baseline | 47864 | |
| huge | npm | jetpack | 9451 | -71.45 % |
| huge | npm | baseline | 33105 |
(The Mode column has clickable links to scenario projects on GitHub. For Type, baseline is built-in Serverless Framework packaging, jetpack is our new plugin. Time is in milliseconds.)
Try it out!
... and thus the serverless-jetpack plugin was born!
Installation should be a breeze for most projects. Jetpack works seamlessly with built-in Serverless packaging configuration, including service and individual function-level configurations with exclude, include, etc.
yarn and npm users should add this to serverless.yml:
plugins: - serverless-jetpack
and npm users will also need:
custom: serverless-jetpack: mode: npm # (default `yarn`)
And that's mostly it!
We take correctness seriously at Formidable. In addition to the usual tests and style checks, we validate that Jetpack's output in each scenario identically matches that of built-in Serverless packaging.
As a few parting thoughts, you should be using yarn with a yarn.lock file or npm@5.7.0+ with a package-lock.json to get a full speedup. And, there are a few esoteric complexities and additional configuration options that you may want to review before integration.
All in all, if Serverless Framework packaging/deployment speed is cramping your style, hook up a jetpack and see if your deploys take off! 🚀
Related Posts
Investing in Your Technology and Team But Still Not Seeing the Desired Results?
Sam Reffitt