Webpack 5 feature. Sharing code between separate bundles

This is a proposal to merge my existing work into the Webpack core

Feature request

@sokra as requested, I've opened a new issue for us to discuss planning and implementation.

This is the evolution of my original issue: #8524

I want to share resources between separate bundles at runtime. Similar to DLL plugin but without needing to provide build context, and I want to do this at runtime in the browser. Not at build time. Thus enabling tech like micro-frontends to be efficient, easy to manage, and most importantly - erase page reloads between separate apps

For context, I've already implemented this feature request with great success. I feel this would have a major impact on frontend applications. I am proposing to refactor/rewrite my project and introduce it into the Webpack core. https://github.com/ScriptedAlchemy/webpack-external-import

• I expect to be able to require modules or chunks from other Webpack bundles hosted elsewhere.
• I don't want to manage externals and worry about synchronizing them across systems
• I don't want a single point of failure, like a commons chunk
• I want to load code from another build, like dynamic imports and code-splitting
• I want multiple builds to look and feel like a monolith in the client
• I am able to deploy frontend apps independently and expect Webpack to orchestrate at runtime
• I don't want to use gimmicks like SingleSPA, browser events, service workers - I want to use Webpack as the host container for foreign chunks and modules, not load other app entry points, but rather load other apps chunks
• If desired, an entire company should be able to federate code across every UI, from user-facing to backend. Effectively turning a multi build, multi team, multi mono-repo company into one SPA in the browser. Removing the need for page reloads or downloading additional code and bundles containing mostly the same node modules

What is motivation or use case for adding/changing the behavior?

This would offer a major shift in frontend architecture.

Apps are getting larger and code-splitting on its own is not as effective with things such as micro-frontend technology.

All current solutions on the market are substandard - working around the problem instead of addressing it. Application interleaving at runtime enables many new avenues for engineering.

Heres some:

• Applications can be self-healing, in the event there is a network failure - one could query other interleaved apps and load their copy of the missing dependency. This would solve deploy issues where pages break for a moment as the new code replaced old
• Code can be distributed and managed by the team maintaining it, with options for evergreen code. Navigation, Pages, Routers - could be managed and deployed by the team who owns it - others would not have to redeploy their applications to get updates
• Smaller bundles - interleaved apps wouldn't download chunks if Webpack already has the modules required by the interleaved app or fragment.
• Easy and feasible micro-frontends - the tech right now involves some serious management and time to build supporting infrastructure
• Shareable configs - interleave FE configurations without the need to rebundle or expose multiple configs.
• Avoid long & slow builds. Companies like Atlassian are running out of memory due to the size of the monorepo. This would enable smaller builds and faster deploys with less coordination across teams
• Introduction of "micro-service functions" modeling something similar to what backend engineers get to enjoy.
• Better AB testing and marketing tooling. Right now AB tests are managed via a tag manager, plastering vanilla JS onto the dom which is usually not optimized, pollifilled, or compatible with how much javascript apps are written today. Markering teams and AB teams would write native components without having to get in the way of team delivery.
• Better analytics - Tag managers and analytics engineers could write system native modules that could be woven into codebases. Most depend on some plugin or just reading the DOM. Interleaving would open a new world of slick integration.
• third party modules - vendors could take advantage of this tech and offer managed modules instead of inefficient scripts - they'd still be secure but would have a new way of implementation. Iframes could be avoided in some cases
• no global pollution of the window, globals can be accessed via the Webpack runtime
• as more apps are interleaved, they too can supply their modules to another interleaved app, resulting in less and less code needing to be downloaded as more and more modules are added to webpack

The use cases are limitless, I've only begun to imagine possibilities.

Webpack is more than capable of doing this without needing any big changes to the core. Both 4 and 5 already have what I need.

How should this be implemented in your opinion?

Seeing as I've already done it. I have a clear picture of how to implement it. However, it can be improved with some guidance. I hold little opinion over the solution, as long as it offers a similar outcome.

1. Add an additional require extension which can support interleaving. I'm currently using a slightly modified version of requireEnsure which already works very well with Webpacks chunk and module loading / caching. when requiring an interleaved module - users would specify a pattern like /<module.id> - Because my current solution is based on requireEnsure, it's pretty robust and not hacky. To handle CSS I've also taken the code template from mini-css to support side-effect loading.
2. Output an unhashed JS file that can be embedded onto other apps. This file would contain a small mapping of chunk.id to the cache busted file name.
3. Hashed module ids based on contents + package.json version of dependency + usedExports to support tree-shaken code. Hashing needs to be effective at avoiding collisions as well as to support tree-shaken code. Hashing enables effective code sharing and prevents overwriting existing modules. Ideally, if there's an option to update module id after optimization - it would likely improve the reliability of hashing. However, I have not encountered any problems. Open to alternative solutions. This hashing mechanism I am using is used by amazon who built a CLI to orchestrate code sharing across the company.
4. A slight addition to the chunk template to include some registration data or another function similar to webpackJsonp.push which registers chunks and provides some metadata to Webpack
   Heres what it would include:

chunk.id

5. an interface similar to externals, where a developer can specify what files/modules they intend to make available for interleaving. During the build, Webpack would not hash these module.ids, but instead, make the id be whatever the key is. Allowing humans to call the module by a predictable name. All dependencies of a module and all modules, in general, would be hashed - ensuring that nested dependencies can be tree shaken and will only use existing modules in Webpack if they are an exact match.
6. Some automatic chunk splitting which would ensure interleaved files are placed into their own cache group. I also recommend chunking vendor bundles and any loader JS code out of the main chunk in order to avoid interleaving the entry point because of something like css-loader getting grouped into the main chunk. In order to prevent downloading a massive 200kb chunk, I am suggesting that there be some default limit for maxSize set on cache groups. Something along the lines of enabling aggressive code-splitting by default if interleaving is enabled. The goal is a happy medium between not downloading dozens of chunks and not downloading a massive chunk for one or two modules the host build cannot supply.

"interleave": {
    "TitleComponent": "src/components/Title/index.js",
    "SomeExternalModule": "src/components/hello-world/index.js"
  }

plugins: [
    new WebpackExternalImport({
      manifestName: "website-one"
    })
  ];

Which would be used in consumer apps like this:

require.interleaved('website-one/TitleComponeent')

For better understanding, I suggest reading the links below. Please note that I am aware of many of the less elegant parts of the codebase and eager to improve the implementation to meet internal standards of Webpack (perf, bundle size, implementation strategy, better hooks to use)

• https://github.com/ScriptedAlchemy/webpack-external-import
• https://github.com/ScriptedAlchemy/webpack-external-import/blob/master/src/webpack/interleaveFn.js
• https://github.com/ScriptedAlchemy/webpack-external-import/blob/master/src/webpack/chunkSplitting.js
• https://github.com/ScriptedAlchemy/webpack-external-import/blob/master/src/webpack/optimizeChunk.js

Are you willing to work on this yourself?

yes - I have been contributing to Webpack since v1 and have a deep understanding of its API.

I'd like to be the one to bring this feature to the wider JS community as well as to Webpack. I've built multiple large scale MFEs, some consisting of over 100 MFE's at one company. I am very familiar with the challenges and alternative methods to chunk federation/interleaving - all of them are terrible.

While this solution might not be for everyone, it will have major value to massive corporations consisting of hundreds of teams of FE stacks. I am also able to test implementation updates against my own userbase - providing additional assurance to Webpack. I used the same approach when merging extract-css-chunks with mini-css (bringing HMR to mini-css)

Current results

I've been using this system for AB tests, Next.js apps, and config sharing at scale.

In a simple comparison. Interleaving three separate React apps produced only 320kb of code transferred (1.2mb uncompressed) compared to 4mb using traditional solutions available on the market.

Extra considerations

• Whitelisting instructions that would enable developers to tell Webpack to use the "host build's" version of a dependency. Cases like React where I can't have two on the page - I want to offer the option to have a chunk use the version already installed, even if its a patch version different.
• Hashing bloats builds: I'm looking for an alternative solution to hashing to give the community options, however, the savings of code sharing drastically outweigh any overhead in my opinion. 4mb to 1.2mb speaks for itself.
• Chunk manifests are inside the chunk - this could be in one single file to produce a single manifest, however, I like the idea of code splitting chunk needs into the chunk themselves. This would make the actual manifest file very small as all it looks up is chunk names to URL paths. Seeing as the manifest is either cache busted with Date.now or 304 cache control - I wanted to keep it lean in the event developers don't want to deal with cache headers and just cache bust the 2kb file on each page load. This would introduce one extra loading wave, as once the chunk is downloaded, it would specify what it needs, triggering Webpack to then download any additional files in one call. Download chunk->download any chunks containing modules Webpack host build cannot provide. Nested chunks would not create more loading waves as the chunk metadata contains a thorough map of what it needs in totality

SSR

This wouldn't work with SSR - however, I am server-side rendering interleaved code. Either with ESI or flushing out the various interleaved parts of an app - then performing an ASYNC request to the interleaved fragments origin server which could expose a rendering API. In the world of micro-frontends, they all are capable of SSR so adding a rendering endpoint is trivial. Utilities could be made for flushing out interleaved chunks that need to be SSR'd by their origin server. Other tactics I have used "eventual consistency", developers npm install the interleaved code and use it to render a somewhat up to date version, keeping the client-side code evergreen. This tactic requires more management, but as an alternative to rendering APIs