The awkward valley to ESM: Node.js, Victory, and D3

Last year, we detailed the coming challenges of Node.js’ support of ECMAScript Modules ("ESM"). Fast forwarding to this year, we got to experience some of the forecasted challenges firsthand! In this post, we will dive into some ESM integration adventures with the Victory charting and data visualization library and reflect on what our exemplary challenges mean for the Node.js ecosystem.

What is ESM again?

As a quick refresher (covered in more detail in our previous ESM blog post), there are presently two main ways to consume code from one file to another in Node.js.

CommonJS - the old way

CommonJS is the longstanding, legacy scheme for importing/exporting code synchronously in Node.js, and looks like:

// Import
const colors = require("colors");

// Export
module.exports = { red: "red", blue: "blue" };

ESM - the new way

ECMAScript Modules ("ESM") offer a new syntax and two means of importing/exporting code:

// Import
import * as colors from "colors";      // synchronous, static
const colors = await import("colors"); // async, dynamic

// Export
export { red: "red", blue: "blue" };

The CommonJS and ESM approaches to consuming code files don’t seem too different on the surface — switch a require() with an import ... from and you should be good to go! Unfortunately, there are some significant practical concerns that we’ll examine in more detail in the sections below, most notably:

1. While ESM code can easily consume both ESM and CommonJS libraries…
2. CommonJS code cannot easily consume ESM libraries. The CommonJS code must be refactored to asynchronously import() upstream ESM libraries.

The awkward valley of ESM on Node.js

CommonJS, and Node.js’ adoption of it, predates ESM by a significant amount of time. Learning from the challenges of CommonJS and other approaches, ESM has risen in popularity with the community and standards committees gaining wide adoption with bundlers introducing support in 2015 and browsers starting in 2017.

Bundlers and browsers had a much easier introduction situation than Node.js. Bundlers already transform source code to an ultimately different final bundle form, so they just needed support for the ingestion and transformation of the new ESM format. Browsers had no native import process (other than global scope <script> tags), so there was no existing import system to compete with or replace.

By contrast, Node.js has long depended on consuming external code via CommonJS. Thus, it was not surprising that full ESM support reached Node.js much later in May 2020, in Node.js v12.17.0. Since then, there has been a push to get more and more Node.js developers porting over and starting new projects in ESM mode. But, there is also an enormous amount of running, existing projects, and applications happily using CommonJS.

And this leads us to what we’ll term an “awkward valley” of ESM in the Node.js ecosystem. On one side, we have bundlers, browsers, and Node.js in ESM mode that can easily use the full world of CommonJS and ESM code. On the other side, we now have Node.js in CommonJS mode that faces significant hurdles when depending on code that is only provided as ESM.

The future is most certainly with everyone eventually moving over to the ESM-only side, but the practical reality right now is a large part of the Node.js community is still on the CommonJS side.

Let’s discuss this side of the awkward valley a bit more.

Unhappy ESM surprises

Our Victory library includes many packages that use various D3 libraries under the hood to provide utilities for composing and manipulating charts and visualizations. For example, we depend on libraries like d3-scale to scale data sets for visual representations.

A regular part of open source project maintenance is tracking and updating dependencies to catch improvements and bug/security fixes. In March 2022, we received a seemingly normal upgrade request to bump d3-color to version 3+ to address security issues. We upgraded and released packages a week later. And then we received bug reports about the following error for a D3 transitive dependency of the upgraded d3-color:

[ERR_REQUIRE_ESM]: Must use import to load ES Module

We dug in and investigated further and discovered that as of June 2021, the D3 project had updated most of the D3 libraries to support only ESM. Specifically, most D3 projects changed package.json to "type": "module" and only shipped ESM code files, dropping support for CommonJS. We ultimately found that all but one of the D3 libraries we rely on in Victory had been updated to support only ESM.

In terms of how we used D3, all of our dependencies came from static imports like:

import * as d3Array from "d3-array";
import * as d3Scale from "d3-scale";

And nearly all the imported D3 functions were used in synchronous functions in Victory.

With this starting point in hand, we reviewed our options along the lines that we discussed in our previous blog post:

1. Victory becomes ESM-only: We could convert all our Victory packages similarly to how D3 did and support only ESM. Victory would be usable in the browser, with all modern front-end bundlers, and on the server for Node.js applications using ESM. However, we would have to drop support for Node.js applications using CommonJS.
2. Convert D3-consuming APIs to asynchronous: While CommonJS cannot statically import ESM code, it may asynchronously await import() libraries. We could refactor every function in our API that uses D3 code to become asynchronous. We actually tried a proof-of-concept branch for this approach before concluding that the change was undesirable because: (1) we had to change a cascading and significant number of APIs in a semantic version major (breaking) manner, (2) our previously fast, CPU-bound API functions now had the overhead of asynchronous loading, which was sure to have at least a modest performance impact, and (3) the API was changing in a manner that didn’t feel right — the asynchronous parts of a library typically are reserved for actual waiting on something like I/O, not just importing a library.
3. Transpile and vendor in the D3 libraries we use: Victory already uses Babel to perform all manner of transpilations for our source code, including providing versions for both CommonJS and ESM. We could just use these same tools to transpile D3 source and provide the now missing CommonJS versions so that Victory’s APIs didn’t have to change and we continued support for Node.js applications using CommonJS.

The more we surveyed Node.js CommonJS usage in the wild, and in our own client projects at Formidable, the more we found that there is still a lot of CommonJS usage for upstream applications in the Node.js ecosystem. That meant that we really couldn’t choose option #1 to make Victory only work in ESM integrations, and thus applications.

We had already discounted option #2 as discussed above. Victory’s existing API made (fairly reasonable) assumptions everywhere that the D3 functions we used would be synchronous in nature. The changes to support making Victory asynchronous anywhere a method touched a D3 function ended up being in our estimation too much change for the wrong reasons.

And that left us with option #3, making a CommonJS version of the necessary D3 libraries.

DIY time — manually transforming ESM-only code for universal support

The big question after deciding that we’d vendor in Victory’s D3 dependencies confronted us — how should we technologically accomplish this?

One option would be just copying and pasting in the code from the various D3 libraries into real, git-tracked source code. However, this presented a maintenance drawback in that updates to these libraries tracking the real D3 upstream libraries would be a bit of pain and error-prone, to continue manually copying and pasting.

Thus, we decided to use a build script to transpile the D3 libraries from node_modules source into our package library paths, which would be published to npm, but not tracked in git. As we implemented this latter solution, we then noticed that we didn’t really need to transpile D3 source code to support ESM, just to support CommonJS. We leveraged this observation to come up with a bespoke solution as follows:

• Our vendor package would take real package.json dependencies on the various D3 libraries.
• If a runtime import comes from ESM, we just forward it on to the real D3 library in node_modules/d3-<name>
• If a runtime require comes from CommonJS, we point it to our transpiled custom CommonJS code.

We put this approach together in a new package, victory-vendor.

Vendor package file layout

A top level peek at our victory-vendor package looks like this:

es/             # Entry points for ESM
lib/            # Entry points for CommonJS
lib-vendor/     # Transpiled CommonJS code for D3 code
node_modules/   # Our actual dependencies
package.json

Breaking this layout down a little bit more:

• es/: Provides entry points for ESM imports that point to the real dependencies in node_modules.
• lib/: Provides entry points for CommonJS requires that point to our custom transpiled code in lib-vendor/ that provides D3 libraries the otherwise unavailable CommonJS format.
• lib-vendor: Is created in a build step that takes node_modules/d3-* libraries and converts ESM imports to CommonJS requires.
• node_modules: Contains real, normal npm dependencies.

We use package.json configurations to control the runtime import mode switching in Node.js via the exports field and declare our real D3 dependencies in the dependencies field:

// victory-vendor/package.json
"exports": {
  // ...
  "./d3-*": {
    "import": "./es/d3-*.js",
    "default": "./lib/d3-*.js"
  }
},
"dependencies": {
  "d3-array": "^3.1.6",
  "d3-ease": "^3.0.1",
  "d3-interpolate": "^3.0.1",
  "d3-scale": "^4.0.2",
  "d3-shape": "^3.1.0",
  "d3-time": "^3.0.0",
  "d3-timer": "^3.0.1"
}

With this scheme, we’re able to actually use all the “real” dependencies in the ESM scenario, and only rely on our transpiled D3 libraries for CommonJS.

Example: d3-scale

Let’s see how this works in more detail with one exemplary library, d3-scale.

ESM import

Let’s start with an import in Node.js ESM mode:

import * as d3Scale from "victory-vendor/d3-scale";

This hits our package.json:exports field which translates victory-vendor/d3-scale to victory-vendor/es/d3-scale.js, which looks like:

// Our ESM package uses the underlying installed dependencies of `node_modules/d3-scale`
export * from "d3-scale";

This proxies on the real dependency of victory-vendor/node_modules/d3-scale. So, all in all, Victory’s upstream Node.js ESM consumers only hit an abstraction layer on their way to the real D3 dependency.

CommonJS import

Node.js in CommonJS mode works a bit differently. Let’s say we start with:

const d3Scale = require("victory-vendor/d3-scale");

This hits our package.json:exports field which translates victory-vendor/d3-scale to victory-vendor/lib/d3-scale.js, which looks like:

// Our CommonJS package relies on transpiled vendor files in `lib-vendor/d3-scale`
module.exports = require("../lib-vendor/d3-scale/src/index.js");

This then redirects to our victory-vendor/lib-vendor transpiled files starting at victory-vendor/lib-vendor/d3-scale/src/index.js.

Belt and suspenders

We provide both of the official Node.js runtime import specifiers with package.json:exports. However, not all tools in the ecosystem use those officially. We immediately noticed that our Jest tests failed and discovered that older versions of Jest (and presumably other tools) don’t use exports at all! So, to provide an extra out, we actually provide real files in the package to mirror our entry points with CommonJS re-exports, e.g. victory-vendor/d3-scale.js:

// This file only exists for tooling that doesn't work yet with package.json:exports
// by proxying through the CommonJS version.
module.exports = require("./lib/d3-scale");

From there, everything works exactly like the previous CommonJS scenario.

Wow, that was a lot of work!

The full Victory pull request shows the work in more detail. To make our solution more maintainable and consumable, we used build scripts to automate matching D3 package dependencies with our transpiled outputs and entry points. When we need to upgrade a given D3 package, it is as simple as updating a package.json:dependencies.d3-<name> version and building and publishing a new version of victory-vendor.

Our solution is a bit complex, with different code execution paths depending on import mode. But, we feel it is the best overall compromise with the following benefits:

• Allows ESM consumption to use the real underlying ESM packages from node_modules.
• Allows CommonJS consumption to work at all, using our transpiled code.
• Allows us, as package maintainers, to track updates upstream with D3 with straightforward package.json:dependencies updates. Having the real D3 library versions listed in dependencies is important so that package analysis tools like dependabot, etc. can inspect, flag security issues, and help with dependency maintenance and upgrades.

At the same time, it is worth noting that there were some aspects of our D3 dependencies that made our vendoring approach more workable. For example, the D3 libraries in question didn’t have transitive dependencies on non-D3 libraries, allowing us a limited world of things we needed to vendor. Victory’s most important dependencies are on D3, so now having a strategy for these collections of libraries, we likely won’t have similar hurdles for our other dependencies (which we can remove, replace, etc.).

Takeaways

ESM is widely supported in the JavaScript ecosystem and the agreed-on path forward for dependencies and imports across most modern runtimes and environments. However, the pervasiveness of CommonJS-based applications in Node.js in use worldwide presents a unique challenge as some Node.js library authors (like the author of D3) convert their code to ESM-only and immediately break nearly all existing upstream CommonJS consumption.

Time will tell, but it may potentially be years before ESM Node.js applications / consumption becomes prevalent and overwhelming. In the meantime, we’re left with an awkward valley between Node.js using CommonJS and most other runtime scenarios.

Our advice from our previous ESM post still mostly holds up now as we encounter more practical ESM situations:

• If you are an application author, go ahead and write your Node.js apps using ESM! It’s the future, and your ESM application can fairly easily consume legacy CommonJS code and libraries.
• If you are a library author, please consider producing both ESM and CommonJS code in your packages and using package.json:exports to support both runtimes. If you’re already using a transpilation step like Babel or TypeScript, this should just prove to be a minor tweak to your build and publish scripts. If you’re only going to publish one form of code in your libraries, at this point in time, we advise that it be CommonJS — so that your library can continue to support both CommonJS and ESM upstream libraries and applications.

In the hopefully not-too-distant future, we will all be using ESM in our Node.js applications and libraries. But, until that time, we hope that library authors continue to give the developer community perhaps a bit more runway with CommonJS support so that the average developer doesn’t have to go through the challenges and effort we encountered in our D3-supporting adventure with Victory.

Custom illustrations by Sophie Calhoun