Configure Karma to load pegjs with requirejs

Asked 24/11, 2015 at 13:46 Answered 2/12, 2015 at 7:32

Solved javascript node.js requirejs karma-runner pegjs

Trying to test a project using PegJS and requirejs. I have a couple of source files, implemented as AMD module (define) which loads through the require API. Below the directory structure:

js/
   somefile.js
   main.js
   parser.js
test/
   parser.spec.js

I've written a parser.js module to load a PegJS grammar file and use PegJS to create a peg parser:

define(function() {
  'use strict';

  var PEG = require('pegjs');
  var grammarFile = 'grammar.peg'

return {
  parse: function(fs, content, debug) {
    var grammar = fs.readFileSync(grammarFile, 'utf8').toString();
    // Build parser from grammar
    var parser = PEG.buildParser(grammar, { trace: debug });
    [...]

This works fine with a main.js executed on the command line with node. Now I want to test my project using karma, jasmine and PhantomJS. I have a karma.conf.js like this:

frameworks: ['jasmine', 'requirejs'],
files: [
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js',
],

Also have a require bootstrap file called test-main.js which is configured this way:

'use strict';

var allTestFiles = [];
var TEST_REGEXP = /(spec|test)\.js$/i;

// Get a list of all the test files to include
Object.keys(window.__karma__.files).forEach(function(file) {
  console.log(file);
  if (TEST_REGEXP.test(file)) {
    // Normalize paths to RequireJS module names.
    // If you require sub-dependencies of test files to be loaded as-is (requiring file extension)
    // then do not normalize the paths
    var normalizedTestModule = file.replace(/^\/base\/|\.js$/g, '');
    allTestFiles.push(file);
  }
});

require.config({
  // Karma serves files under /base, which is the basePath from your config file
  baseUrl: '/base/js',
  // dynamically load all test files
  deps: allTestFiles,
  // we have to kickoff jasmine, as it is asynchronous
  callback: window.__karma__.start
});

Now, when I launch my test (grunt karma), I got this error:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: Error{message: 'Module name "pegjs" has not been loaded yet for context: _. Use require([])

So I try to include pegjs in the files loaded by Karma this way karma.conf.js:

files: [
  { pattern: 'node_modules/pegjs/lib/**/*.js', included: true  },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

When I do this, I still get an error:

Error: Module name "utils/arrays" has not been loaded yet for context: _. Use require([])

Looking inside pegjs module, there is indeed an arrays.js file:

compiler/
compiler.js
grammar-error.js
parser.js
peg.js
utils/
  arrays.js
  classes.js
  objects.js

So trying to include arrays too:

files: [
  { pattern: 'node_modules/pegjs/lib/utils/arrays.js', included: true },
  { pattern: 'node_modules/pegjs/lib/**/*.js', included: true  },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

I get:

ReferenceError: Can't find variable: module
at /blabla/node_modules/pegjs/lib/utils/arrays.js:108

Because of:

108 module.exports = arrays;

So, intead of loading the npm module, I tried to load the bower module this way:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

And here you go again:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: Error{message: 'Module name "pegjs" has not been loaded yet for context: _. Use require([])

Also tried not to include pegjs in the karma generated web page:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

But it fails with:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: 'There is no timestamp for /base/bower_components/pegjs/peg-0.9.0!'

Tried to put the bower_component folder inside the js folder but no luck.

So I don't know were to go from here... Couldn't find anything relevant on Google or here. It seems to be a specific problem to requirejs/pegjs with karma... Any idea is welcome.

UPDATE following dan's answer:

So I switch from a synchronous require to a asynchronous require in parser.js:

define(['../bower_components/pegjs/peg-0.9.0'], function(PEG) {
  'use strict';

  var grammarFile = 'grammar.peg'

return {
  parse: function(fs, content, debug) {
    var grammar = fs.readFileSync(grammarFile, 'utf8').toString();
    // Build parser from grammar
    var parser = PEG.buildParser(grammar, { trace: debug });
    [...]

Tried to include the pegjs bower component in karma.conf.js:

{ pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },

or not include it:

{ pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },

But always get the same error:

Error: Script error for "/blabla/bower_components/pegjs/peg-0.9.0", needed by: /blabla/js/parser.js
http://requirejs.org/docs/errors.html#scripterror
at /blabla/node_modules/requirejs/require.js:140

Yes the file exists:

$ file /home/aa024149/share/logofrjs/bower_components/pegjs/peg-0.9.0.js 
/blabla/bower_components/pegjs/peg-0.9.0.js: ASCII text, with very long lines

UPDATE2: Finally understood and found an acceptable solution.

Balsamic answered 24/11, 2015 at 13:46 Comment(0)

So with the help of the various answers and comments from dan and pieceOpiland I finally came to a way to do what I want.

So first, pegjs, like many javascript libraries comes in two formats: npm modules and bower modules.

Npm modules are used for script made for node and called from the command line. Bower modules are used for script loaded in a browser.

First misunderstanding from my part was that the 'require' would work in node and in the browser indistinctly. This is wrong. It seems the only way to require a module so that it works in the browser is through a asynchronous call like:

require(['module'], function(module) {
  ...
});

Another misunderstanding was that I could load npm modules in the browser. That some kind of magic would operate for the various npm files to be loaded with my page. That might be possible but only using some special tool like browserify. Without special transformation, only the bower version can be loaded in the browser. Additionally, pegjs bower module is made in a way so that global variables are defined like so:

var PEG = {
 ...
}

module.exports = PEG;

Basically, the bower module plugs a global variable (actually several global variables) to the top level scope.

So instead of having my client code (the one running in the browser AND in node) loading the module, I actually load the module in either:

main.js through a synchronous require to the npm module like so: var PEG = require('pegjs');
main-browser.js through the global variable which becomes available when you load the bower pegjs script (through a <script> tag)

Both those 'mains' are then injecting the PEG variable to my parser function.

For karma to work, I just need to include the pegjs bower module in the generated page (karma.conf.js extract):

files: [
 { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },
 { pattern: './test/**/*.spec.js', included: false },
 { pattern: './js/**/*.js', included: false},
 './test/test-main.js',
],

Balsamic answered 2/12, 2015 at 7:32 Comment(1)

Forget about all the require/bower non-sense, do yourself a favor, move to webpack! – Balsamic 15/11, 2016 at 13:21

It sounds like you're loading pegjs via requirejs. If this is the case, pegjs should not be a file that is included. In your karma.conf.js, have you tried the following:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

The value for included indicates whether or not the webpage that the karma server generates should have a script tag for that file or not (See http://karma-runner.github.io/0.13/config/files.html). So your karma.config of:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

will cause karma to generate a webpage with a head tag similar to:

<head>
  <script src="/base/bower_components/pegjs/peg-0.9.0.js"></script>
  <script src="/base/require.js"></script>
  <script src="/base/test/test-main.js"></script>
</head>

In my experience, I've seen a lot of behavior similar to this that was caused by marking my files as included: true. If there is a file which you are trying to load with requirejs, make sure that it is marked as included: false.

I believe that this marks it for a pre-processing job, but I'm not entirely sure why this makes such a difference.

Capsule answered 29/11, 2015 at 22:0 Comment(3)

Yes pieceOpiland, that is one of the many combination I tried but it still fails. I'll update the question. – Balsamic 30/11, 2015 at 7:42

Hmm, the .js extension typically added by requirejs is missing in the most recent error you posted. Would you mind including your requirejs config as well? – Capsule 1/12, 2015 at 1:40

Ok I have completed the require config I use but I doubt you'll have more useful information. Note that my project works fine in the command line with node. The problem is with the karma testing. – Balsamic 1/12, 2015 at 7:22

As far as I know, Karma is a testing framework which will run your tests in the browser.

This isn't suitable for testing many node modules.

The browser doesn't have the facility to do this synchronously: var PEG = require('pegjs'). This is why it is asking you to use require([]) which you pass a callback to be executed when pegjs finishes loading.

Using the bower version of pegjs and ensuring it is loaded before require('pegjs') is called may help here. This would ensure that pegjs is already loaded for context _ (the default requirejs context, I presume).

It also can't load files from the file system with fs.readFileSync(grammarFile, 'utf8') so you will have to do it another way. You can ask Karma to host your peg grammar by placing it in the files array and then loading it using the requirejs text plugin.

If the module you are testing is aimed at running on node.js and not in the browser then it may be more suited to using a test framework that does not run the code in the browser, but runs it in node so you have all the node modules available to you. If you are aiming this at the browser, I would rewrite it to more specifically target the browser.

Vardon answered 1/12, 2015 at 10:17 Comment(6)

Tried that too (see updated question), but it does not seem to be able to load the bower version either. And anyway, it seems the bower version is not compatible with the npm version because when I try to run my program with the command line, I got an error saying one of pegjs function (buildParser) is not available. So both modules (bower/npm) are not interchangeable which is a problem. I would like to have my parser to be used in a Browser AND in command line. I initially though it was one of the premise of nodejs to be able to do that. – Balsamic 1/12, 2015 at 13:3

As per your point on the grammar file, I will just refactor my project so that the main.js load the file and the parser takes a chunk of text. I'm trying to stay away from requirejs mumbo jumbo as much as possible... – Balsamic 1/12, 2015 at 13:18

Is it a necessity that the parser is generated each time the script is run? Could it be generated as a build artefact which would be deployed with the app? Does opening the developer console tell you anything when running the bower version in the browser? Does the browser successfully get the bower version of pegjs (check the network tab doesn't have a 404)? Does the script error pause in the debugger and give anything meaningful as it is usually caused by a bad script? – Vardon 1/12, 2015 at 16:37

Additionally, you could use the paths property of requirejs config to set the path to pegjs. require.config({baseUrl: '/base/js', paths: {'pegjs': '../bower_components/pegjs/peg-0.9.0'} }). Then you can use define(['pegjs'], function(PEG) {}). Although, I don't expect this to help directly. – Vardon 1/12, 2015 at 16:43

Dan, I made some progress. It turns out the bower version defines many global variables (yuk...) directly in the root scope. So the variable PEG (and others) are defined when you include the script. So my strategy will be the following, for the node version: load the PEG npm modules in my main.js file and pass it as a dependency injection to my parser function. As for the the browser version: I will have some kind of main-browser.js which will just assume that the peg.js script is loaded within a script tag and provide the global variable PEG to the parser function. – Balsamic 1/12, 2015 at 17:13

Ah, if that is what causes the issue, you could use the requirejs shim option to tell requirejs which global variable it should expect the pegjs script to set. requirejs.org/docs/api.html#config-shim – Vardon 2/12, 2015 at 11:17