Bundling your Webpack based build processes & performance improvements

[side note] This article is about adjusting the neo.mjs framework:

https://github.com/neomjs/neo

The code base is MIT licensed, so even if you don’t use it, you can apply the concepts to your own projects. It is hopefully inspirational in any case.

[/side note]

You probably know the feeling: a project grows and you add more granular build tasks one by one.

At some point you take a look at your package.json script and it looks like this:

This is precisely what happened to the neo.mjs framework. A big file size for the package.json can be bad for calling “npm publish” resulting in registry timeouts, but this also can be very confusing for new developers who start using the framework.

So, it was time for a major refactoring. This is the new package.json:

Of course we still need all previous tasks and it is pretty likely that the amount of tasks will grow further.

The new entry point is the buildAll program. I actually added this one twice to the package.json, since you can either use it with command line options or using the visual inquirer interface (kind of a personality trait thing which mode you do prefer).

> node ./buildScripts/buildAll.js — help

> npm run build-all-questions

You can take a look at the program source code here:

https://github.com/neomjs/neo/blob/dev/buildScripts/buildAll.js

This program is passing the env & noquestions options to the child programs for obvious reasons. In case you just cloned the repository, you will most likely just want to call

> npm run build-all

and all sub-programs will run without prompting questions. build-all is just an entry point to combine other programs, so it will not do any builds on its own.

Let us take a look at the buildThreads program next:

https://github.com/neomjs/neo/blob/dev/buildScripts/webpack/buildThreads.js

It follows the same design pattern like buildAll (the other programs do so as well).

By default, neo.mjs is using 4 threads:

1. main (The default top level process)
2. app (webworker)
3. data (webworker)
4. vdom (webworker)

This concept is very different compared to other frameworks out there, since most parts of the framework as well as the apps you are building with it run inside the app thread.

Every app is using the same builds for main, data & vdom, so you will need to build these threads pretty rarely.

The main thread got extremely modular => you can pick which addons you want to use with framework configs. Each addon does get lazy loaded (dynamic imports).

It does make a lot of sense to keep the Webpack based entry points for each thread separate, since it is impossible to share chunks between different realms.

For more details on main thread addons, please take a look at my previous article:

https://codeburst.io/using-js-libraries-inside-a-multithreading-environment-835cd8cbc30b

It is important to know that every app will only include the main chunk inside its index.html file and main will import the apps which you want to use as needed.

Since the app thread is a combination of the app worker and your app code, it is not included inside the buildThreads program.

https://github.com/neomjs/neo/blob/dev/buildScripts/webpack/buildMyApps.js

buildMyApps is using dynamic program options for apps => it will show all available apps inside your repository (you can create new ones using the create-app script).

A good example is probably the Covid Dashboard app with around 2500 lines of code (comments excluded).

On my machine, building it for the development environment (non minified, using source maps), it takes 0.98s.

The build for the production environment (minified, no source maps) is pretty similar: 1.05s.

Smaller apps like the neo.mjs RealWorld implementation are a bit faster:

Let us take a look at buildThemes next:

https://github.com/neomjs/neo/blob/dev/buildScripts/webpack/buildThemes.js

At this point, there are 2 themes available out of the box: dark & light.

I chose to implement them using SCSS, since the wrapper allows us to generate an output with optionally including CSS variables or not (named the option CSS4, although this is not 100% correct).

The build processes are using PostCSS, so you don’t need to write different rules for different browsers using prefixes.

This is not set in stone, you are welcome to add PRs for generating CSS via JS (skill sceptical if this can work with PostCSS though).

The nice thing when using CSS vars is, that you can easily use multiple themes for one app and apply them to different parts. All themes share a common CSS source and provide a CSS var file for each mode (assuming you do want to use CSS vars). As the result, using multiple themes has a very small impact on the overall CSS file sizes.

https://github.com/neomjs/neo/blob/dev/buildScripts/webpack/buildDocsExamples.js

As seen in the screenshot, the docs app is an neo.mjs app as well. Since we want to be able to (lazy) load example apps as needed into it, the app thread builds for the docs app & example apps are combined.

While the main thread already does support Webpack based chunk splitting, this part is still missing for the app realm. Pretty complex, but on the todo list.

For creating the docs output of the framework and your own apps out of the box, neo.mjs is using jsdoc-x.

The overall build time went up to 140s+ on my machine, which was no longer reasonable at all.

I dived a bit into the jsdoc-x source code and created an small override which reduced this time to only 5s for the neo.mjs context:

https://github.com/onury/jsdoc-x/issues/14

As the result, the total build time for literally everything including npm install went down to 35.5s.

Since you will need buildAll very rarely, this feels perfect to me.

When I wanted to deploy this new version to the Github pages (online examples):

https://neomjs.github.io/pages/

Webpack created entirely different outputs for split chunks. 6 of 8 main thread addons ended up in the correct spot, while 2 others got a vendors prefix inside the path, breaking the output. It feels definitely related to having the framework inside a node_modules folder here. A small hack fixed it for now, more details here:

https://github.com/webpack/webpack/issues/10949

After spending a lot of time on the new build programs, I feel the need for mentioning the most important part about neo.mjs:

node & Webpack are supposed to be Tools to build and bundle your apps for the dist modes.

When developing apps, it is extremely convenient to run the real ES8 based code directly inside the browser with 0 builds at all.

You don’t need source maps, you don’t need hot module replacements, you can just work with the real code.

This did save me a lot of time already.

To be fair, for the neo.mjs context this only works in Chrome 80+, since Firefox & Safari are not capable yet to support JS modules inside workers.

However, in case you are creating apps on your own with just sticking to the main thread, you can do it right away with excellent browser support.

What is coming next for neo.mjs?

As mentioned above, Webpack based chunk splitting for the app realm is still open. Supporting touch events (similar to hammer js) is a big topic and what I am really looking forward to is JS module support for shared workers.

At this point we can create neo.mjs based UIs which run in multiple browser windows => multi display support.

I hope you enjoyed the article & am wishing you happy coding through the pandemic.

Best regards

Tobias