RuntimeError: memory access out of bounds - gatsby development extremely slow

Question

Developing on Gatsby has been over time significantly slowing down - at times taking up 20-30 seconds for the hot loader to refresh the page. After around 10-20 updates to a page - it eventually always runs into the following error below:

error UNHANDLED REJECTION


  RuntimeError: memory access out of bounds



  - source-map-consumer.js:335 BasicSourceMapConsumer._parseMappings
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/source-map-consumer.js:335:44

  - source-map-consumer.js:315 BasicSourceMapConsumer._getMappingsPtr
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/source-map-consumer.js:315:12

  - source-map-consumer.js:387 _wasm.withMappingCallback
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/source-map-consumer.js:387:57

  - wasm.js:95 Object.withMappingCallback
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/wasm.js:95:11

  - source-map-consumer.js:371 BasicSourceMapConsumer.eachMapping
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/source-map-consumer.js:371:16

  - source-node.js:87 Function.fromStringWithSourceMap
    [frontend]/[gatsby]/[react-hot-loader]/[source-map]/lib/source-node.js:87:24

  - webpack.development.js:150 onSourceMapReady
    [frontend]/[gatsby]/[react-hot-loader]/dist/webpack.development.js:150:61

  - next_tick.js:68 process._tickCallback
    internal/process/next_tick.js:68:7

Our site is has roughly 250+ static pages in /pages, and is programmatically building around 30 pages from *.yml files generated by netlify-cms.

I suspect that it has something to do with sourcemaps becoming exceedingly large - a wild guess is that it may have something to do with what we're doing in gatsby-node.js or cms.js for netlify-cms' admin page - although I'm not sure where to start to begin debugging an issue like this. Any pointers would be greatly appreciated.

A temporary solution we have implemented is changing Gatsby's default source maps from cheap-module-eval-source-map to eval which has brought hot-module recompile times down from 4-6 to 1-2 seconds. However we would obviously prefer to have proper sourcemaps with sane performance.

    // For local development, we disable proper sourcemaps to speed up
    // performance.
    if (stage === "develop") {
        config = {
            ...config,
            devtool: "eval",
        }
    }

We have also recently rebased our forked gatsby-netlify-cms-plugin on v4.1.6 which contained the following notable improvements: https://github.com/gatsbyjs/gatsby/pull/15191 https://github.com/gatsbyjs/gatsby/pull/15591

This update has helped quite a bit. However, we'd ideally prefer to get our HMR compile times into the sub 500ms range WITH proper sourcemaps.

gatsby-node.js

const path = require("path");
const fs = require("fs");
const remark = require("remark");
const remarkHTML = require("remark-html");
const fetch = require("node-fetch");
const _ = require("lodash");

const {
    assertResourcesAreValid,
    sortResourcesByDate,
    getFeaturedResources,
} = require("./src/utils/resources-data");

const parseForm = require("./src/utils/parseEloquaForm");
const {
    ELOQUA_COMPANY,
    ELOQUA_USERNAME,
    ELOQUA_PASSWORD,
} = require("./src/config-secure");
let elqBaseUrl;

const MARKDOWN_FIELDS = [
    "overviewContentLeft",
    "overviewContentRight",
    "assetContent",
    "content",
];

function fetchJson(url, options) {
    function checkResponse(res) {
        return new Promise(resolve => {
            if (res.ok) {
                resolve(res);
            } else {
                throw Error(`Request rejected with status ${res.status}`);
            }
        });
    }

    function getJson(res) {
        return new Promise(resolve => {
            resolve(res.json());
        });
    }

    return fetch(url, options)
        .then(res => checkResponse(res))
        .then(res => getJson(res));
}

function fetchEloqua(url) {
    const base64 = new Buffer(
        `${ELOQUA_COMPANY}\\${ELOQUA_USERNAME}:${ELOQUA_PASSWORD}`
    ).toString("base64");
    const auth = `Basic ${base64}`;

    return fetchJson(url, { headers: { Authorization: auth } });
}

function getElqBaseUrl() {
    return new Promise(resolve => {
        if (elqBaseUrl) {
            resolve(elqBaseUrl);
        } else {
            return fetchEloqua("https://login.eloqua.com/id").then(
                ({ urls }) => {
                    const baseUrl = urls.base;
                    elqBaseUrl = baseUrl;
                    resolve(baseUrl);
                }
            );
        }
    });
}

function getElqForm(baseUrl, elqFormId) {
    return fetchEloqua(`${baseUrl}/api/rest/2.0/assets/form/${elqFormId}`);
}

function existsPromise(path) {
    return new Promise((resolve, reject) => {
        return fs.exists(path, (exists, err) =>
            err ? reject(err) : resolve(exists)
        );
    });
}

function mkdirPromise(path) {
    return new Promise((resolve, reject) => {
        return fs.mkdir(path, err => (err ? reject(err) : resolve()));
    });
}

function writeFilePromise(path, data) {
    return new Promise((resolve, reject) => {
        return fs.writeFile(path, data, err => (err ? reject(err) : resolve()));
    });
}

exports.onPreBootstrap = () => {
    // Cache the eloqua base URL used to make Eloqua requests
    return new Promise(resolve => {
        getElqBaseUrl().then(() => {
            resolve();
        });
    });
};

exports.onCreateWebpackConfig = ({ stage, actions, plugins, loaders }) => {
    let config = {
        resolve: {
            modules: [path.resolve(__dirname, "src"), "node_modules"],
        },
        plugins: [
            plugins.provide({
                // exposes jquery as global for the Swiftype vendor library
                jQuery: "jquery",
                // ideally we should eventually remove these and instead use
                // explicit imports within files to take advantage of
                // treeshaking-friendly lodash imports
                $: "jquery",
                _: "lodash",
            }),
            plugins.define({
                CMS_PREVIEW: false,
            }),
        ],
    };

    if (stage === "build-html") {
        config = {
            ...config,
            module: {
                rules: [
                    {
                        // ignore these modules which rely on the window global on build phase
                        test: /jquery|js-cookie|query-string|tabbable/,
                        use: loaders.null(),
                    },
                ],
            },
        };
    }

    actions.setWebpackConfig(config);
};

exports.onCreateNode = ({ node, actions }) => {
    const { createNode, createNodeField } = actions;
    const { elqFormId, happyHour, resourcesData } = node;
    const forms = [];

    if (resourcesData) {
        assertResourcesAreValid(resourcesData);

        const sortedResources = sortResourcesByDate(resourcesData);
        const featuredResources = getFeaturedResources(resourcesData);

        createNodeField({
            name: "sortedResourcesByDate",
            node,
            value: sortedResources,
        });

        createNodeField({
            name: "featuredResources",
            node,
            value: featuredResources,
        });
    }

    // Convert markdown-formatted fields to HTML
    MARKDOWN_FIELDS.forEach(field => {
        const fieldValue = node[field];

        if (fieldValue) {
            const html = remark()
                .use(remarkHTML)
                .processSync(fieldValue)
                .toString();

            createNodeField({
                node,
                name: field,
                value: html,
            });
        }
    });

    function createFormFieldsNode({ elqFormId, nodeFieldName }) {
        return getElqBaseUrl()
            .then(baseUrl => getElqForm(baseUrl, elqFormId))
            .then(form => {
                createNodeField({
                    node,
                    name: nodeFieldName,
                    value: parseForm(form),
                });
                return Promise.resolve();
            })
            .catch(err => {
                throw `Eloqua Form ID ${elqFormId} - ${err}`;
            });
    }

    // Fetch main Eloqua form and attach to node referencing elqFormId
    if (elqFormId) {
        const mainForm = createFormFieldsNode({
            elqFormId,
            nodeFieldName: "formFields",
        });

        forms.push(mainForm);
    }

    // The main event landing page has two forms, the main form and a happy hour
    // form. This gets the happy hour form.
    if (happyHour && happyHour.elqFormId) {
        const happyHourForm = createFormFieldsNode({
            elqFormId: happyHour.elqFormId,
            nodeFieldName: "happyHourFormFields",
        });

        forms.push(happyHourForm);
    }

    return Promise.all(forms);
};

exports.onCreatePage = ({ page, actions }) => {
    const { createPage, deletePage } = actions;

    // Pass the page path to context so it's available in page queries as
    // GraphQL variables
    return new Promise(resolve => {
        const oldPage = Object.assign({}, page);
        deletePage(oldPage);
        createPage({
            ...oldPage,
            context: {
                slug: oldPage.path,
            },
        });
        resolve();
    });
};

exports.createPages = ({ graphql, actions }) => {
    const dir = path.resolve("static/compiled");
    const file = path.join(dir, "blocked-email-domains.json");
    const lpStandardTemplate = path.resolve("src/templates/lp-standard.js");
    const lpEbookTemplate = path.resolve("src/templates/lp-ebook.js");
    const lpWebinarSeriesTemplate = path.resolve(
        "src/templates/lp-webinar-series.js"
    );
    const lpThankYouTemplate = path.resolve("src/templates/lp-thank-you.js");
    const lpEventMainTemplate = path.resolve("src/templates/lp-event-main.js");
    const lpEventHappyHourTemplate = path.resolve(
        "src/templates/lp-event-happy-hour.js"
    );
    const lpEventActivityTemplate = path.resolve(
        "src/templates/lp-event-activity.js"
    );
    const lpEventRoadshowTemplate = path.resolve(
        "src/templates/lp-event-roadshow.js"
    );
    const { createPage } = actions;

    return graphql(`
        {
            // a bunch of graphQL queries
       }
    `).then(result => {
        if (result.errors) {
            throw result.errors;
        }

        // Create pages from the data files generated by the CMS
        result.data.allLpStandardYaml.edges.forEach(({ node }) => {
            createPage({
                path: `resources/${node.slug}`,
                component: lpStandardTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpThankYouYaml.edges.forEach(({ node }) => {
            createPage({
                path: `resources/${node.slug}`,
                component: lpThankYouTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpEbookYaml.edges.forEach(({ node }) => {
            createPage({
                path: `resources/${node.slug}`,
                component: lpEbookTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpWebinarSeriesYaml.edges.forEach(({ node }) => {
            createPage({
                path: `resources/${node.slug}`,
                component: lpWebinarSeriesTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpEventMainYaml.edges.forEach(({ node }) => {
            createPage({
                path: `events/${node.slug}`,
                component: lpEventMainTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpEventHappyHourYaml.edges.forEach(({ node }) => {
            createPage({
                path: `events/${node.slug}`,
                component: lpEventHappyHourTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpEventActivityYaml.edges.forEach(({ node }) => {
            createPage({
                path: `events/${node.slug}`,
                component: lpEventActivityTemplate,
                context: {
                    ...node,
                },
            });
        });

        result.data.allLpEventRoadshowYaml.edges.forEach(({ node }) => {
            createPage({
                path: `events/${node.slug}`,
                component: lpEventRoadshowTemplate,
                context: {
                    ...node,
                },
            });
        });

        // Build copy of blocked-email-domains.yml as JSON in /static
        // This is referenced by the Eloqua-hosted forms on go.memsql.com
        const { blockedEmailDomains } = result.data.miscYaml;
        const domainsArray = blockedEmailDomains
            .trim()
            .split("\n")
            .map(rule => rule.toLowerCase());

        return existsPromise(dir)
            .then(exists => (exists ? Promise.resolve() : mkdirPromise(dir)))
            .then(() => writeFilePromise(file, JSON.stringify(domainsArray)));
    });
};

package.json

{
   ...,
    "browserslist": [
        ">0.25%",
        "not dead"
    ],
    "devDependencies": {
        "@babel/core": "7.1.5",
        "@babel/plugin-syntax-dynamic-import": "7.2.0",
        "@storybook/addon-knobs": "5.0.11",
        "@storybook/addon-storysource": "5.0.11",
        "babel-eslint": "8.2.2",
        "babel-loader": "8.0.4",
        "eslint": "4.12.1",
        "eslint-plugin-babel": "4.1.2",
        "eslint-plugin-flowtype": "2.39.1",
        "eslint-plugin-import": "2.8.0",
        "eslint-plugin-prettier": "2.3.1",
        "eslint-plugin-react": "7.5.1",
        "file-loader": "2.0.0",
        "gatsby": "2.8.8",
        "gatsby-link": "2.1.1",
        "gatsby-plugin-intercom-spa": "0.1.0",
        "gatsby-plugin-netlify-cms": "vai0/gatsby-plugin-netlify-cms#918821c",
        "gatsby-plugin-polyfill-io": "1.1.0",
        "gatsby-plugin-react-helmet": "3.0.12",
        "gatsby-plugin-root-import": "2.0.5",
        "gatsby-plugin-sass": "2.1.0",
        "gatsby-plugin-sentry": "1.0.1",
        "gatsby-plugin-sitemap": "2.1.0",
        "gatsby-source-apiserver": "2.1.2",
        "gatsby-source-filesystem": "2.0.39",
        "gatsby-transformer-json": "2.1.11",
        "gatsby-transformer-yaml": "2.1.12",
        "html-webpack-plugin": "3.2.0",
        "node-fetch": "2.3.0",
        "node-sass": "4.12.0",
        "react-lorem-component": "0.13.0",
        "remark": "10.0.1",
        "remark-html": "9.0.0",
        "uglify-js": "3.3.28",
        "uglifyjs-folder": "1.5.1",
        "yup": "0.24.1"
    },
    "dependencies": {
        "@fortawesome/fontawesome-pro": "5.6.1",
        "@fortawesome/fontawesome-svg-core": "1.2.6",
        "@fortawesome/free-brands-svg-icons": "5.5.0",
        "@fortawesome/pro-regular-svg-icons": "5.4.1",
        "@fortawesome/pro-solid-svg-icons": "5.4.1",
        "@fortawesome/react-fontawesome": "0.1.3",
        "@storybook/addon-a11y": "5.0.11",
        "@storybook/addon-viewport": "5.0.11",
        "@storybook/addons": "5.0.11",
        "@storybook/cli": "5.0.11",
        "@storybook/react": "5.0.11",
        "anchorate": "1.2.3",
        "autoprefixer": "8.3.0",
        "autosuggest-highlight": "3.1.1",
        "balance-text": "3.3.0",
        "classnames": "2.2.5",
        "flubber": "0.4.2",
        "focus-trap-react": "4.0.0",
        "formik": "vai0/formik#d524e4c",
        "google-map-react": "1.1.2",
        "jquery": "3.3.1",
        "js-cookie": "2.2.0",
        "lodash": "4.17.11",
        "minisearch": "2.0.0",
        "moment": "2.22.0",
        "moment-timezone": "0.5.23",
        "netlify-cms": "2.9.0",
        "prop-types": "15.6.2",
        "query-string": "5.1.1",
        "react": "16.6.1",
        "react-add-to-calendar-hoc": "1.0.8",
        "react-anchor-link-smooth-scroll": "1.0.12",
        "react-autosuggest": "9.4.3",
        "react-dom": "16.6.1",
        "react-helmet": "5.2.0",
        "react-player": "1.7.0",
        "react-redux": "6.0.0",
        "react-remove-scroll-bar": "1.2.0",
        "react-select": "1.2.1",
        "react-slick": "0.24.0",
        "react-spring": "6.1.8",
        "react-truncate": "2.4.0",
        "react-typekit": "1.1.3",
        "react-waypoint": "8.0.3",
        "react-youtube": "7.6.0",
        "redux": "4.0.1",
        "slick-carousel": "1.8.1",
        "typeface-inconsolata": "0.0.54",
        "typeface-lato": "0.0.54",
        "whatwg-fetch": "2.0.4",
        "xr": "0.3.0"
    }
}

cms.js

import CMS from "netlify-cms";

import "typeface-lato";
import "typeface-inconsolata";
import "scss/global.scss";

import StandardLandingPagePreview from "cms/preview-templates/StandardLandingPagePreview";
import StandardThankYouPagePreview from "cms/preview-templates/StandardThankYouPagePreview";
import EbookLandingPagePreview from "cms/preview-templates/EbookLandingPagePreview";
import WebinarSeriesLandingPagePreview from "cms/preview-templates/WebinarSeriesLandingPagePreview";
import EventMainLandingPagePreview from "cms/preview-templates/EventMainLandingPagePreview";
import EventHappyHourLandingPagePreview from "cms/preview-templates/EventHappyHourLandingPagePreview";
import EventActivityLandingPagePreview from "cms/preview-templates/EventActivityLandingPagePreview";
import EventRoadshowLandingPagePreview from "cms/preview-templates/EventRoadshowLandingPagePreview";

// The following window and global config settings below were taken from here.
// https://github.com/gatsbyjs/gatsby/blob/master/docs/docs/visual-testing-with-storybook.md
// They're required because the netlify-cms runs on a separate webpack config,
// and outside of Gatsby. This ensures any Gatsby components imported into the
// CMS works without errors

// highlight-start
// Gatsby's Link overrides:
// Gatsby defines a global called ___loader to prevent its method calls from creating console errors you override it here
global.___loader = {
    enqueue: () => {},
    hovering: () => {},
};

// Gatsby internal mocking to prevent unnecessary errors
global.__PATH_PREFIX__ = "";

// This is to utilized to override the window.___navigate method Gatsby defines and uses to report what path a Link would be taking us to
window.___navigate = pathname => {
    alert(`This would navigate to: https://www.memsql.com${pathname}`);
};

CMS.registerPreviewTemplate("lp-standard", StandardLandingPagePreview);
CMS.registerPreviewTemplate("lp-ebook", EbookLandingPagePreview);
CMS.registerPreviewTemplate(
    "lp-webinar-series",
    WebinarSeriesLandingPagePreview
);
CMS.registerPreviewTemplate("content", StandardLandingPagePreview);
CMS.registerPreviewTemplate("content-syndication", StandardLandingPagePreview);
CMS.registerPreviewTemplate("programmatic", StandardLandingPagePreview);
CMS.registerPreviewTemplate("sponsored-webcast", StandardLandingPagePreview);
CMS.registerPreviewTemplate("webcast", StandardLandingPagePreview);
CMS.registerPreviewTemplate("web-forms", StandardLandingPagePreview);
CMS.registerPreviewTemplate("other", StandardLandingPagePreview);
CMS.registerPreviewTemplate("lp-thank-you", StandardThankYouPagePreview);
CMS.registerPreviewTemplate("lp-event-main", EventMainLandingPagePreview);
CMS.registerPreviewTemplate(
    "lp-event-happy-hour",
    EventHappyHourLandingPagePreview
);
CMS.registerPreviewTemplate(
    "lp-event-activity",
    EventActivityLandingPagePreview
);
CMS.registerPreviewTemplate(
    "lp-event-roadshow",
    EventRoadshowLandingPagePreview
);

Environment

  System:
    OS: Linux 4.19 Debian GNU/Linux 10 (buster) 10 (buster)
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.17.3 - /usr/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 75.0.3770.142
    Firefox: 60.8.0
  npmPackages:
    gatsby: 2.10.4 => 2.10.4
    gatsby-cli: 2.7.2 => 2.7.2
    gatsby-link: 2.2.0 => 2.2.0
    gatsby-plugin-intercom-spa: 0.1.0 => 0.1.0
    gatsby-plugin-manifest: 2.1.1 => 2.1.1
    gatsby-plugin-netlify-cms: vai0/gatsby-plugin-netlify-cms#e92ec70 => 4.1.6
    gatsby-plugin-polyfill-io: 1.1.0 => 1.1.0
    gatsby-plugin-react-helmet: 3.1.0 => 3.1.0
    gatsby-plugin-root-import: 2.0.5 => 2.0.5
    gatsby-plugin-sass: 2.1.0 => 2.1.0
    gatsby-plugin-sentry: 1.0.1 => 1.0.1
    gatsby-plugin-sitemap: 2.2.0 => 2.2.0
    gatsby-source-apiserver: 2.1.3 => 2.1.3
    gatsby-source-filesystem: 2.1.0 => 2.1.0
    gatsby-transformer-json: 2.2.0 => 2.2.0
    gatsby-transformer-yaml: 2.2.0 => 2.2.0
  npmGlobalPackages:
    gatsby-dev-cli: 2.5.0

Update:

Here is a profile of a HMR - making a change, saving, undoing that change, then saving again.

The compile time for this was around 1300ms for each change. Note this is with us setting sourcemaps in development to eval - the cheaper option.

Anyone see anything that we can do to improve this time? It appears graphql is recompiling our queries even though our change is literally updating text.

Here's the performance profile result using Chrome Devtools.

Scever

do you have enough memory on your device? – Cymophane

16GB and this issue happens even with ~8 GB available. – Pinkster

I think the problem will be with the number of promise requests in your gatsby-node file. The node file is run during development to create the nodes. Hence, it will take time for all those to be resolved. # Memory out of bound This error is thrown during memory allocation when you are allocating memory in a wrong way. – Cymophane

@Doc-Han It doesn't seem like the promises are being run on HMR, and is only run during the initial build phase. Also unfortunately making these requests is also unavoidable - we cannot reduce the number of calls and changing them to be synchronous is obviously out of the question. – Scever

Answer 1

As shown, the documentation of gatsby-source-filesystem Gatsby adds a limitation to the concurrent downloads to prevent the overload of processRemoteNode. To fix and modify any custom configuration, they expose a GATSBY_CONCURRENT_DOWNLOAD environment variable:

To prevent concurrent requests overload of processRemoteNode, you can adjust the 200 default concurrent downloads, with GATSBY_CONCURRENT_DOWNLOAD environment variable.

In your running command, set the value that fixes your issue. In my case, it was 5:

"develop": "GATSBY_CONCURRENT_DOWNLOAD=5 gatsby develop"

Interstadial