Webpack: Common chunks for code shared between Webworker and Web code?
Asked Answered
B

4

5

I have lots of code shared between web and web worker parts of my browser app.

How can I tell webpack to split my code up into common chunks so that the result is garanteed to work 100%?

The webworker code breaks (fails at runtime) after I tell webpack to generate the common chunks (which it does). Even after I fix the trivial "window not defined" error the worker just does nothing.

I believe this has to do with the webpack "target" option, which per default is set to "web". But I need "web" target because I don't have purely webworker code.

I also cannot do multiple webpack configs because I cannot do the common chunks thing with multiple configs...

What should I do?

If anybody is interested: I am trying build a minimal sized build for my app which includes the monaco editor (which provides the workers):

https://github.com/Microsoft/monaco-editor/blob/master/docs/integrate-esm.md

You can see here (at the bottom of the page) that the entry points consist of 1 main entry file + the workers.

Currently at least 6 MB is wasted because of duplicate code I am using and currently can not be split up because of this problem. That is a lot of wasted traffic.

Any ideas? :)

my webpack 4.1.1 config is basically:

module.exports = (env, options) => {
    const mode = options.mode;
    const isProduction = mode === 'production';
    const outDir = isProduction ? 'build/release' : 'build/debug';

    return {

        entry: {
            "app": "./src/main.tsx",
            "editor.worker": 'monaco-editor/esm/vs/editor/editor.worker.js',
            "ts.worker": 'monaco-editor/esm/vs/language/typescript/ts.worker.js'
        },
        output: {
            filename: "[name].bundle.js",
            path: `${__dirname}/${outDir}`,
            libraryTarget: 'umd',
            globalObject: 'this',
            library: 'app',
            umdNamedDefine: true
        },
        node: {
            fs: 'empty' 
        },
        devtool: isProduction ? undefined : "source-map",
        resolve: {
            extensions: [".ts", ".tsx", ".js", ".json"],
            alias: {
                "@components": path.resolve(__dirname, "src/components"),
                "@lib": path.resolve(__dirname, "src/lib"),
                "@common": path.resolve(__dirname, "src/common"),
                "@redux": path.resolve(__dirname, "src/redux"),
                "@services": path.resolve(__dirname, "src/services"),
                "@translations": path.resolve(__dirname, "src/translations"),
                "@serverApi": path.resolve(__dirname, "src/server-api")
            }
        },
        optimization: isProduction ? undefined : {
            splitChunks: {
                minSize: 30000,
                minChunks: 1,
                name: true,
                maxAsyncRequests: 100,
                maxInitialRequests: 100,
                cacheGroups: {
                    default: {
                        chunks: "all",
                        priority: -100,
                        test: (module) => {
                            const req = module.userRequest;
                            if (!req) return false;
                            return (!/node_modules[\\/]/.test(req));
                        },
                    },
                    vendor: {
                        chunks: "all",
                        test: (module) => {
                            const req = module.userRequest;
                            if (!req) return false;
                            if (!/[\\/]node_modules[\\/]/.test(req)) return false;
                            return true;
                        },
                        priority: 100,
                    }
                }
            },
        },
        module: {
            rules: [...(isProduction ? [] : [
                {
                    enforce: "pre", test: /\.js$/, loader: "source-map-loader",
                    exclude: [
                        /node_modules[\\/]monaco-editor/ 
                    ]
                }
            ]),
            {
                test: require.resolve('jquery.hotkeys'),
                use: 'imports-loader?jQuery=jquery'
            },
            {
                test: /\.tsx?$/,
                loader: "awesome-typescript-loader",
                options: {
                    configFileName: 'src/tsconfig.json',
                    getCustomTransformers: () => {
                        return {
                            before: [p => keysTransformer(p)]
                        };
                    }
                }
            },
            {
                test: /\.(css|sass|scss)$/,
                use: extractSass.extract({
                    use: [
                        {
                            loader: 'css-loader',
                            options: {
                                minimize: isProduction
                            }
                        },
                        {
                            loader: "postcss-loader",
                            options: {
                                plugins: () => [autoprefixer({
                                    browsers: [
                                        'last 3 version',
                                        'ie >= 10'
                                    ]
                                })]
                            }
                        },
                        { loader: "sass-loader" }
                    ],
                    fallback: "style-loader"
                })
            },
            {
                test: /node_modules[\/\\]font-awesome/,
                loader: 'file-loader',
                options: {
                    emitFile: false
                }
            },
            {
                test: { not: [{ test: /node_modules[\/\\]font-awesome/ }] },
                rules: [
                    {
                        test: { or: [/icomoon\.svg$/, /fonts[\/\\]seti\.svg$/] },
                        rules: [
                            { loader: 'file-loader?mimetype=image/svg+xml' },
                        ]
                    }, {
                        test: { not: [/icomoon\.svg$/, /fonts[\/\\]seti\.svg$/] },
                        rules: [
                            {
                                test: /\.svg(\?v=\d+\.\d+\.\d+)?$/,
                                use: {
                                    loader: 'svg-url-loader',
                                    options: {}
                                }
                            },
                        ]
                    },
                    {
                        test: /\.(png|jpg|gif)$/,
                        loader: 'url-loader'
                    },
                    { test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/font-woff" },
                    { test: /\.woff2(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/font-woff" },
                    { test: /\.ttf(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/octet-stream" },
                    { test: /\.eot(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader" },
                ]
            },

            ]
        },
        plugins: [
            new HardSourceWebpackPlugin({
                cacheDirectory: '../node_modules/.cache/hard-source/[confighash]', configHash: function (webpackConfig) {
                    return require('node-object-hash')({ sort: false }).hash(Object.assign({}, webpackConfig, { devServer: false }));
                },
                environmentHash: {
                    root: process.cwd(),
                    directories: [],
                    files: ['../package-lock.json'],
                }
            }),
            new webpack.ProvidePlugin({
                "window.$": "jquery"
            }),
            new CleanWebpackPlugin(outDir),
            extractSass,
            new HtmlWebpackPlugin({
                title: 'my title',
                filename: 'index.html',
                minify: isProduction ? {
                    collapseWhitespace: true,
                    collapseInlineTagWhitespace: true,
                    removeComments: true,
                    removeRedundantAttributes: true
                } : false,
                template: 'index_template.html',
                excludeChunks: ['ts.worker', "editor.worker"]
            }),
            new webpack.IgnorePlugin(/^((fs)|(path)|(os)|(crypto)|(source-map-support))$/, /vs[\\\/]language[\\\/]typescript[\\\/]lib/)
        ].concat(isProduction ? [new webpack.optimize.LimitChunkCountPlugin({
            maxChunks: 1
        })] : [])
    }
};
Bush answered 19/3, 2018 at 0:20 Comment(6)
Looks like this bug github.com/webpack/webpack/issues/6642Roumell
unfortunately the fix in that thread (remove HotModuleReplacementPlugin) does not apply to me, don't have that plugin enabled. I think this applies more to me: github.com/webpack/webpack/issues/6525 . So missing feature in webpack and no bug?Bush
I think this will solve your problem: https://mcmap.net/q/475363/-webpack-4-universal-library-targetToxicant
I don't know a lot about workers, but this seems related as well? github.com/webpack/webpack/issues/6472Roumell
manually changing global object (window=self) did not help. worker just does nothing then (runs, but does not do its work, no errors). i really suspect it's because their is no "web+webworker" combined target. targets do more than just vary global objects, i supposeBush
Don't change it manually. Do change it like it is described in the answer (target: 'umd', globalObject: 'this')Toxicant
D
3

EDIT: Alright I wrote a webpack plugin based on everyone's knowledge just put together.

https://www.npmjs.com/package/worker-injector-generator-plugin

You can ignore the content below, and use the plugin or if you want to understand how the plugin came to be and do it by hand yourself (so you don't have to depend on my code) you can keep reading.

=====================================================

Alright after so much researching I figured out this solution, you need to create an injection file, for a simple case you need the https://github.com/webpack-contrib/copy-webpack-plugin as it works pretty well... so let's say your setup is:

entry: {
    "worker": ["./src/worker.ts"],
    "app": ["./src/index.tsx"],
  },

And you have setup your common plugins already let's say this example.

optimization: {
    splitChunks: {
      cacheGroups: {
        commons: {
          name: 'commons',
          chunks: 'initial',
          minChunks: 2
        },
      }
    }
  },

You need to now create an injection "Vanilla JS" which might look like this:

var base = location.protocol + "//" + location.host;
window = self;

self.importScripts(base + "/resources/commons.js", base + "/resources/worker.js");

Then you can add that alongside your worker, say in src/worker-injector.js

And using the copy plugin

new CopyPlugin([
      {
        from: "./src/worker-injector.js",
        to: path.resolve(__dirname, 'dist/[name].js'),
      },
    ]),

Make sure your output is set to umd.

output: {
    filename: '[name].js',
    path: path.resolve(__dirname, 'dist'),
    libraryTarget: "umd",
    globalObject: "this",
  }

This is nothing but a hack, but allows you to use everything as it is without having to do something as overblown.

If you need hashing (so that copy plugin doesn't work) functionality you would have to generate this file (rather than copying it), refer to this.

How to inject Webpack build hash to application code

For that you would have to create your own plugin which would generate the vanilla js file and consider the hash within itself, you would pass the urls that you want to load together, and it would attach the hash to them, this is more tricky but if you need hashes it should be straightforward to implement with your custom plugin.

Sadly so far there doesn't seem to be other way.

I could probably write the plugin myself that would do the workaround and create the injectors, but I do think this is more of a hack and shouldn't be the accepted solution.

I might later go around and write the injector plugin, it could be something as:

something like new WorkerInjectorGeneratorPlugin({name: "worker.[hash].injector.js", importScripts: ["urlToLoad.[hash].js", secondURLToLoad.[hash].js"])

refer to this issues for reference, and why it should be fixed within webpack and something as a WorkerInjectorGeneratorPlugin would be pretty much a hack plugin.

https://github.com/webpack/webpack/issues/6472

Demerol answered 19/1, 2020 at 0:16 Comment(3)
OMG, hours of hair-pulling with Monaco Editor and Webpack and Web Workers... to finally find your answer & npm package... and everything now works! <3Nerveracking
@OlivierLance :) Glad it helped, I pulled my hair for days; it doesn't seem to be a common issue developers hit nevertheless since it still seems unresolved; webpack should be able to have some importScripts and async functionality for webworker targets so that the code gets embed; currently the way I made it it makes you depend on an extra file (the injector); which basically does the same thing a normal website would do by fetching sources; this logic needs to be implemented directly into the file itself webpack generates. But we'll wait :)Demerol
Yep, I actually want to try another approach with a plugin that would act kinda like the MonacoEditorWebpackPlugin by making use of the internal WebWorkerTemplatePlugin, which seems to be injected at compilation time. I'm not very fluent in Webpack internals so that might not be feasible, but I'll try :)Nerveracking
C
2

This is really bad answer, but i've managed to share chunks between workers and main thread.

The clue is that

  1. globalObject has to be defined as above to (self || this):
output: {
    globalObject: "(self || this)"
}
  1. Webpack loads chunks with document.createElement('script') and document.head.appendChild() sequence, which is not available in worker context, but we have self.importScript. So it's just a matter of "polyfiling" it. Here is working "polyfill" (straight from the hell):
console.log("#faking document.createElement()");
(self as any).document = {
    createElement(elementName: string): any {
        console.log("#fake document.createElement", elementName);
        return {};
    },
    head: {
        appendChild(element: any) {
            console.log("#fake document.head.appendChild", element);
            try {
                console.log("loading", element.src);
                importScripts(element.src);
                element.onload({
                    target: element,
                    type: 'load'
                })
            } catch(error) {
                element.onerror({
                    target: element,
                    type: 'error'
                })
            }
        }
    }
};
  1. Ensure, that your real code is resolved after polyfill is installed, by using dynamic import, which will. Assuming, that normal "worker main" is in "./RealWorkerMain", that would be "main worker script":
// so, typescript recognizes this as module
export let dummy = 2;

// insert "polyfill from hell" from here

import("./RealWorkerMain").then(({ init }) => {
    init();
});
  1. You may need to configure dynamic import in webpack, as documented here is not easy too, this answer was very helpful.
Corinthian answered 14/1, 2020 at 23:54 Comment(2)
This is the only answer that makes sense but oh god it's ugly :(Demerol
@Zbigniew Zagórski Thanks a ton you saved my day ! About 4. I didn't need the plugin @babel/plugin-syntax-dynamic-import but this one @babel/plugin-transform-runtime to avoid ReferenceError: regeneratorRuntime is not defined error. (using babel 7.11.x)Forras
T
1

You're looking for universal library target, aka umd.

This exposes your library under all the module definitions, allowing it to work with CommonJS, AMD and as global variable.

To make your Webpack bundle compile to umd you should configure output property like this:

output: {
    filename: '[name].bundle.js',
    libraryTarget: 'umd',
    library: 'yourName',
    umdNamedDefine: true,
},

There is an issue with Webpack 4, but if you still want to use it, you can workaround the issue by adding globalObject: 'this' to the configuration:

output: {
    filename: '[name].bundle.js',
    libraryTarget: 'umd',
    library: 'yourName',
    umdNamedDefine: true,
    globalObject: 'this'
},
Toxicant answered 19/3, 2018 at 11:36 Comment(3)
will not work :( no console errors so i don't have to change global object manually. but worker still does not work (monaco editor does not underline errors anymore which it does if i don't split chunks). added my webpack config in starting postBush
It's hard to tell why it won't work, espessially if you don't get any errors. You need to debug it: make sure it brings all the chunks in the network tab then put some breakpoints and see where it fails.Toxicant
It simply does not execute, that's why it does not work. The code hangs forever, waiting for the other data that will never come because it never requested nor executed it.Demerol
F
0

Native Worker support is introduced in webpack 5. With this feature, you can share chunks between app code and webwokers with simple splitChunk options like

{
    optimization: {
        splitChunks: {
            chunks: 'all',
            minChunks: 2,
        },
    },
}

When combining new URL for assets with new Worker/new SharedWorker/navigator.serviceWorker.register webpack will automatically create a new entrypoint for a web worker.

new Worker(new URL("./worker.js", import.meta.url))

The syntax was chosen to allow running code without bundler too. This syntax is also available in native ECMAScript modules in the browser.

https://webpack.js.org/blog/2020-10-10-webpack-5-release/#native-worker-support

Firewood answered 6/1, 2021 at 7:52 Comment(1)
I don't see how this is working. Webpack 5 does detects webworker and produces it. But Webpack 5 doesn't share modules between main app and the worker instead in bundled code you have copy of N modules in main app and copy of N modules in worker. Here are clarifications github.com/webpack/webpack/issues/6472#issuecomment-797078303Fructificative

© 2022 - 2024 — McMap. All rights reserved.