Earlier in the year we released Runpkg, an online tool to help developers explore npm packages and their dependencies. Our initial prototype was created as a build step-free single-page web app, and we learned a ton building it. We were pleased with the results, but, as with any prototype, there was room for improvement. Over the past few weeks, we were able to iterate on our prototype to improve the package viewing experience, the performance and speed of the site, and the overall design and user experience. With these changes completed, we are pleased to announce the release of Runpkg v1! This work was divided among three collaborators, each with a different focus. Each of us will now explain in more detail the changes we made and what we learned.
Luke redesigned and rearchitected the app
- Designed a two-column layout with a tabbed sidebar to reduce visual clutter and make code the focus of the app
- Simplified the site's React architecture to use a single immutable state and a single reducer
The Runpkg beta was designed and built around a three-column layout with package search featuring in a full screen overlay. This design led to complex relationships between state and views, resulting in unnecessary re-renders and re-fetching of data. We decided to simplify the site both visually and architecturally by moving to a two-column layout.
We decided that the code should be the focus of the application and that everything else should be contained in a single aside element with some tabs that allow the user to focus on either searching the registry, the current package, or file.
Redesigning a UI often helps create a better visual of what state is needed and how it flows through the application; this was no exception and the change prompted us to re-think the way we managed state and updates. There were a lot of moving parts at various levels of the app and we wanted to simplify things. So we decided to try something radical:
> A single immutable state, with a single reducer, provided via an app-wide context.
The implementation was inspired by and copied from this article by Luke Hall, which claims it is possible to rig up such a configuration in 10 lines of code (which it is). What this essentially gives us is a single hook that allows you to get and set a global state, in a very similar style to redux.
Our experiment paid off in two significant ways:
- We could easily inspect the entire state of our app with one console.log, which made serializing and debugging bad state combinations much easier.
- It significantly reduced the number of props we needed to pass between components, greatly reducing the complexity of our views.
Alex improved the package viewing experience
- Fully functional, expandable file-tree built with accessibility in mind
- Feature-rich code pane with better syntax highlighting and linkable line numbers and imports
Our redesign made the code pane the focus of the app, and we wanted to improve it. We used prism-react-renderer to add line numbers and expand syntax highlighting to many more languages. The flexibility and customizability of prism-react-renderer also allowed us to introduce new features such as linking import statements and linking directly to line numbers. These improvements made the code pane easier to explore.
We also wanted to make it easier to explore the overall structure of a package, so we created a fully functional, expandable file tree. The rendered tree makes full use of button and anchor elements to ensure it is fully keyboard-accessible. Conditional rendering of expanded/collapsed subtrees ensures that only visible files are announced by screen-readers.
Kara improved performance
- Deferred recursive dependency fetching until requested by a user
- Used Web Workers to move dependency fetching to background threads
- Used Service Workers to intercept fetch requests to unpkg, and return cached results when appropriate.
Performance was the area most in need of improvement in the Runpkg prototype. Viewing larger packages on the Runpkg prototype resulted in hundreds of requests to Unpkg and multi-second load times. The culprit was a function called recursiveDependencyFetch which fetched and processed all of the given file’s dependencies, as well as the dependencies of those dependencies, and so on. With our new redesign in place, we would no longer need all of this data until a user clicked the file tab. Deferring recursive dependency fetching sped up the initial page load, but inspecting files in larger packages still resulted in unacceptable sluggishness. To combat the problem, we used Web Workers to move the expensive calculations required by our recursiveDependencyFetch function into a background thread and batched dispatches back to the main thread.
Processing large sets of dependency data in a background thread improved the performance of the UI, but we still had to worry about fetching all that data. Runpkg requests and fetches all the data it needs from unpkg, resulting in hundreds of network requests for larger packages. Fortunately, a side-effect of semver and immutable npm deploys is that identical requests to unpkg always produce identical responses. This allowed us to minimize network requests with a very simple cache. We used a Service Worker to intercept fetch requests to unpkg, and check whether the request was already cached in the CacheStorage, and either return the cached network response or fetch it. No cache invalidation required. Relying on cached data meant that on repeat visits we would have almost instantaneous resolution of our fetched imports (usually <10ms) which greatly improved network performance, and dramatically reduced data usage!
Overall, the work we have done has massively increased the reliability and performance of the app, which in turn, has made for a more enjoyable user experience. We hope that the interface will prove more useful to more people and encourage people to build tools in a similar vein.
If you have any good ideas for future runpkg features, or if you find a bug, then we encourage you to go to the GitHub repository and create an issue or a pull request.
