Letting web app users run multi-module JavaScript code

[2019-10-11] dev, javascript

In a web app of mine, I wanted to let end users run multi-module JavaScript that they enter via text fields. It turns out that this simple idea is relatively difficult to implement. In this blog post, I’ll explain how to do it. It is less polished than usual – I mainly wanted to get the knowledge out there.

Overview

Use Rollup to combine the user’s multiple modules into a single string:
- Input: multiple modules, passed to Rollup as strings.
- Output: the bundle in a string and a source map.
eval() the text string.
If that produces an exception then the line and column numbers in the exception’s stack trace refer to the bundle, not the input modules.
- Therefore: Use the source map and the library source-map to go from bundle source code locations to input module names and source code locations.
- Unfortunately, Chrome, Safari, and Firefox all have different stack trace formats, which is why this approach only works in Chrome and other V8-based browsers.

The web app itself is bundled via webpack.

Bundling via Rollup

Before we can bundle, we need a way to pass the modules to Rollup as strings (vs. files in a file system). That can be achieved via a plugin:

function createPlugin(moduleSpecifierToSourceCode: Map<string,string>): Plugin {
  return {
    name: 'BundleInputPlugin', // this name will show up in warnings and errors
    resolveId(id) {
      if (moduleSpecifierToSourceCode.has(id)) {
        // Tell Rollup to not ask other plugins or check the file system to find this ID
        return id;
      }
      return null; // handle other IDs as usually
    },
    /** Override source code by returning strings */
    load(id) {
      const source = moduleSpecifierToSourceCode.get(id);
      if (source) {
        return source;
      }
      return null; // other ids should be handled as usually
    },
  };
}

Now we can bundle the code:

const mainModuleName = '__theMainModule__';
const regexPrefixSuffix = /^var __theMainModule__ = ([^]*);\n$/;

export interface BundleDesc {
  code: string;
  sourceMap: SourceMap;
}

export async function createBundle(
  moduleSpecifierToSourceCode: Map<string,string>,
  entrySpecifier: string): Promise<BundleDesc> {
  try {
    const bundle = await rollup({
      input: entrySpecifier,
      plugins: [createPlugin(moduleSpecifierToSourceCode)],
    });  
    const {output} = await bundle.generate({
      format: 'iife',
      name: mainModuleName,
      sourcemap: true,
    });
    const chunkOrAssetArr = [...output];
    if (chunkOrAssetArr.length !== 1 || chunkOrAssetArr[0].type !== 'chunk') {
      throw new Error();
    }
    
    let {code, map} = chunkOrAssetArr[0];
  
    const expressionMatch = regexPrefixSuffix.exec(code);
    if (!expressionMatch) {
      throw new InternalError('Unexpected bundling result: ' + JSON.stringify(code));
    }
    code = expressionMatch[1];
  
    if (!map) {
      throw new InternalError();
    }
    return {
      code,
      sourceMap: map,
    };
  } catch (err) {
    if (err.code === 'PARSE_ERROR') {
      // Handle syntax errors
    } else {
      throw err;
    }
  }
}

Where do the strings for the modules come from?

Even for “global” modules such as 'assert', we need to provide strings. webpack’s raw-loader can help here:

//@ts-ignore
import srcAssert from 'raw-loader!../modules/assert.js';

That allows us to develop modules for the user’s code with the same tools as the web app (even though the web app exists at a meta-level, relative to the user’s code).

Bundle formats

Rollup supports several bundle formats (AMD, CommonJS, ES module, etc.). The Rollup REPL helps with exploring the different bundle formats. The best format for our purposes is “iife”. The following code is an example of this format:

var myBundle = (function (exports) {
	'use strict';

	const foo = 1;

	exports.foo = foo;

	return exports;

}({}));

If we remove 'var myBundle = ' at the beginning and the semicolon at the end, we can eval() this code.

Syntax errors

If Rollup finds syntax errors, it throws instances of Error that include the following properties:

{
  code: 'PARSE_ERROR',
  loc: {
    file: './test.mjs',
    column: 8,
    line: 1,
  },
  frame: '1: iimport a from 'a';\n           ^\n2: import * as b from 'b';',
}

.frame is a string that shows the lines surrounding the syntax error and points to its location.

Importing Rollup

I had to import Rollup as follows in order to not lose the static type information in my IDE:

import { Plugin, rollup, SourceMap } from 'rollup';

Alas, that imports from the Node.js version of Rollup, not from the browser version. I fixed that by adding the following entry to webpack.config.js:

resolve: {
  alias: {
    'rollup': 'rollup/dist/rollup.browser.es.js',
  }
},

Executing the bundle

To execute the bundle, a simple eval.call(undefined, str) suffices. Why eval.call() and not eval()? The former is a safer version of the latter (details).

Handling exceptions

If the evaluated code throws an exception, we get stack traces that look as follows:

Error: Problem!
    at Object.[foo] (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:7:13)
    at Object.bar (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:10:19)
    at Object.get baz [as baz] (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:13:12)
    at f (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:17:9)
    at g (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:20:5)
    at eval (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:22:3)
    at eval (eval at <anonymous> (first-steps-bundle.js:33), <anonymous>:28:2)
    at eval (<anonymous>)
    at first-steps-bundle.js:33
Error: Actual: 6, expected: "blank"
    at equal (eval at executeCode (first-steps.js:NaN), <anonymous>:28:19)
    at Object.eval [as code] (eval at executeCode (first-steps.js:NaN), <anonymous>:40:7)
    at runTests (eval at executeCode (first-steps.js:NaN), <anonymous>:12:26)
    at eval (eval at executeCode (first-steps.js:NaN), <anonymous>:44:16)
    at eval (eval at executeCode (first-steps.js:NaN), <anonymous>:48:2)
    at executeCode (first-steps.js:77)

The stack trace includes line numbers for the web app and line numbers for the end user’s code. I could not find any stack trace parsing library that parsed such stack traces correctly. Hence, I did some quick and dirty parsing via a regular expression.

The <anonymous> line numbers refer to the bundled code. We can use the library source-map to get the original module names and locations:

import { SourceMapConsumer } from 'source-map';

const result = SourceMapConsumer.with(sourceMap, null,
  async (consumer) => { // (A)
    const result = [];
    for (const l of parsedStackTraceLines) {
      const {source, line, column} = consumer.originalPositionFor({
        line: l.lineNumber,
        column: l.columnNumber,
      });
      result.push(···);
    }
    return result;
  });

By wrapping the code contained in the callback (line A), SourceMapConsumer.with() ensures that the resources it creates are disposed of, after they are not needed, anymore.

Setting up the source-map library

Internally, source-map uses Wasm code. I had to use version 0.8.0-beta.0 in order to load that code from the web app bundle.

The JavaScript code looks like this:

//@ts-ignore
import arrayBuffer from 'source-map/lib/mappings.wasm';
(SourceMapConsumer as any).initialize({
  'lib/mappings.wasm': arrayBuffer,
});

Thanks to webpack and the arraybuffer-loader, we can import the Wasm code into an ArrayBuffer. We need the following entry in webpack.config.js:

module: {
  rules: [
    {
      test: /\.wasm$/,
      type: 'javascript/auto',
      loader: 'arraybuffer-loader',
    },    
  ],
},

Letting web app users run multi-module JavaScript code

Overview #

Bundling via Rollup #

Where do the strings for the modules come from? #

Bundle formats #

Syntax errors #

Importing Rollup #

Executing the bundle #

Handling exceptions #

Setting up the source-map library #