Running tests against code examples in a README.md?

Question

Does anyone know of an open-source project or program for running tests against code examples in README.md?

A perennial problem, my documentation tends to drift out of date with the code. For example, a code snippet in the README.md will no longer work with the current version, and isn't caught until a new developer on-boards onto the project. Is it possible to include README.md code snippets in my test suite?

For example, the usage of say.nancat with sample params:

# $ node

> const say = require('say')
> say.nancat('grumpy is best')
'grumpy is best'

The program would initialize an environment with the '#' (not shown in README.md because the context is assumed), run the '>' line and pass/fail based on the next line. Simular to doctests in python.

Loads of people have the problem of keeping README.md and other docs current with the code, so I was hoping there was an off-the-shelf solution. I've looked (DuckDuckGo) to no avail.

Apivorous

Answer 1

Maybe byexample is what you are looking for.

It is a tool to run the snippets of code (aka examples) in a text file and check their outputs. It is like Python's doctests but it works for Javascript, Ruby, Python and others (even for C and C++).

The Javascript examples can be written in a README.md like:

```javascript
1 + 2

out:
3
```

or like:

```javascript
> 1 + 2
3
```

Then, you run them from the command line:

$ byexample -l javascript README.md
[PASS] Pass: 2 Fail: 0 Skip: 0

And that's it. The full documentation of the tool can be found here and here, and the particular comments and limitations for Javascript is here.

Disclaimer: I'm the author of byexample and I created it for the same reason that rmharrison wrote in his question.

Like him, my documentation was "out of sync" in time to time and the only way to notice that was running the examples by hand. For that reason I created this tool to automatically check and validate the docs.

It is really useful to me; I really hope that it would be useful to others.

Balladeer

Thank you! This is exactly what I was looking for. – Apivorous

Answer 2

This possibly should be achieved in the opposite way. Examples should exist as files that can be linted and tested. Their contents can be injected into README.md on documentation build with any template engine.

E.g. custom includeJs helper function can be defined to render

{{ includeJs('foo.js') }}

to respective Markdown:

**foo.js**

```javascript
/* foo.js contents */
```

Depending on how much snippets have in common, the documentation could possibly be parsed first to uniformly generate files from existing snippets.

E.g.

```
# $ node

> const say = require('say')
> say.nancat('grumpy is best')
'grumpy is best' 
```

could be transformed into

// grumpy-is-best.js

const say = require('say')
say.nancat('grumpy is best')

Priorate

Thanks for your kind response. So to your knowledge there isn't an off-the-shelf tool; rather, I must roll my own documentation build step with injection from snippets? – Apivorous

You're welcome. Yes, I would go with doc build step. I'm not aware of such tool and would be surprised if it exists because this would be very odd and complicated approach. It's like writing an application in rich text editor. – Priorate

Answer 3

Try markdown-doctest:

npm install markdown-doctest

Insert this in your markdown file (i.e. README.md):

```js
var a = 5;

var b = 10;

console.log(a + c);
```

And run markdown-doctest:

$ markdown-doctest
x..

Failed - README.md:32:17
evalmachine.<anonymous>:7
console.log(a + c);
                ^

ReferenceError: c is not defined

Puma

Answer 4

For Python, there is exdown, a small helper tool of mine. Install with

pip install exdown

and test your snippets with

import exdown
import pytest

@pytest.mark.parametrize("string, lineno", exdown.extract("README.md"))
def test_readme(string, lineno):
    exec(string)

Cocoa

Answer 5

Another approach is to name the script sections. E.g. ```example1.sh ...

This way you can just use awk to extract the scripts:

awk '/```example1.sh/{f=1;next} /```/{f=0} f' README.md

Example of extracting and testing a bash script section:

testscr=$(awk '/```example1.sh/{f=1;next} /```/{f=0} f' README.md)
eval "$testscr"

Prole