Capture and execute multiline code and incorporate result in raku

Question

This is a markdown document example.md I have:

## New language

Raku is a new language different from Perl.

## what does it offer
+ Object-oriented programming including generics, roles and multiple dispatch
+ Functional programming primitives, lazy and eager list evaluation, junctions, autothreading and hyperoperators (vector operators)
+ Parallelism, concurrency, and asynchrony including multi-core support
+ Definable grammars for pattern matching and generalized string processing
+ Optional and gradual typing



This code will be evaluated.


```{raku evaluate=TRUE}
4/5
```



Rakudo is a compiler for raku programming language. Install it and you're all set to run raku programs!

This code will be evaluated.

```{raku evaluate=TRUE}
say "this is promising";
say $*CWD;
```



This code will **not** be evaluated.


```{raku evaluate=FALSE}
say "Hello world";
```

which I want to convert into example.md as shown below with the code and output within it.

## New language

Raku is a new language different from Perl.

## what does it offer
+ Object-oriented programming including generics, roles and multiple dispatch
+ Functional programming primitives, lazy and eager list evaluation, junctions, autothreading and hyperoperators (vector operators)
+ Parallelism, concurrency, and asynchrony including multi-core support
+ Definable grammars for pattern matching and generalized string processing
+ Optional and gradual typing



This code will be evaluated.

Code:
```{raku evaluate=TRUE}
4/5
```

Output:
```
0.8
```

Rakudo is a compiler for raku programming language. Install it and you're all set to run raku programs!

This code will be evaluated.

Code:
```{raku evaluate=TRUE}
say "this is promising";
say $*CWD;
```

Output:
```
this is promising
"C:\Users\suman".IO
```

This code will **not** be evaluated.

Code:
```{raku evaluate=FALSE}
say "Hello world";
```

---

What I want to accomplish is:

• capture the code between backticks{raku evaluate} and backticks
• execute the code if evaluate is TRUE
• insert the code and output back into the document

---

What I tried to do:

1. Capture multiline code and evaluate expression

my $array= 'example.md'.IO.slurp;

#multiline capture code chunk and evaluate separately
if $array~~/\`\`\`\{raku (.*)\}(.*)\`\`\`/ {
    #the first capture $0 will be evaluate
    if $0~~"TRUE"{
        #execute second capture which is code chunk which is captured in $1
        }else {
       # don't execute code
        };
};

2. create a temp.p6 file and write code chunk $1 from above into it

my $fh="temp.p6".IO.spurt: $1;

3. execute the chunk if $0 is TRUE

my $output= q:x/raku temp.p6/ if $0==TRUE

4. integrate all this into final example.md while we create intermediate example_new.md

my $fh-out = open "example_new.md", :w; # Create a new file

# Print out next file, line by line
for "$file.tex".IO.lines -> $line {

    # write output of code to example_new.md

}
$fh-out.close;

# copy
my $io = IO::Path.new("example_new.md");
$io.copy("example.md");

# clean up
unlink("example.md");

# move
$io.rename("example.md");

I am stuck in the first step. Any help?

Monometallic

The two examples of example.md are exactly the same - were they supposed to be different? – Enroll

@DeanTaylor Thanks for pointing out. I have fixed it. – Monometallic

@raiph Thank you. I want to clear some confusions. Original example.md file will have code chunks. Its won't have Code: and Output:. It is present only in the output example.md file. If evaluate==TRUE in code chunk, then the code is executed. In final document whether code is executed or not, Code: will be present before chunk and if chunk executed, the output will have Output: before it. Could you please update the code accordingly at glot.io ? Currently its not throwing output as expected. Thank you. – Monometallic

Hi @SumanKhanal. Thanks for commenting. I've updated my answer and the code at glot.io. Lmk if that works for you. TIA. – Hygro

Please reduce the examples to relevant parts. – Noyade

@Hygro Exactly what I wanted ! Thanks for your assistance. – Monometallic

Answer 1

Code that accomplishes "What I want to accomplish"

You can run this code against your data with glot.io.

use v6;

constant $ticks = '```';

my regex Search {
  $ticks '{raku evaluate=' $<evaluate>=(TRUE|FALSE) '}'
  $<code>=[<!before $ticks> .]*
  $ticks
}

sub Replace ($/) {
  "Code:\n" ~ $ticks ~ $<code> ~ $ticks ~
    ($<evaluate> eq 'TRUE'
      ?? "\n\n" ~ 'Output:' ~ "\n" ~ $ticks ~ "\n" ~ Evaluate($<code>) ~ $ticks
      !! '');
}

sub Evaluate ($code) {
  my $out; my $*OUT = $*OUT but role { method print (*@args) { $out ~= @args } }
  use MONKEY; my $eval-result = EVAL $code;
  $out // $eval-result ~ "\n"
}

spurt
  'example_new.md',
  slurp('example.md')
    .subst: &Search, &Replace, :g;

Explanation

Starting at the bottom and then working upwards:

• The .subst method substitutes parts of its invocant string that need to be replaced and returns the revised string. .subst's first argument is a matcher; it can be a string, or, as here, a regex -- &Search¹. .subst's second argument is a replacement; this can also be a string, or, as here, a Callable -- &Replace. If it's a Callable then .subst passes the match from the matcher as a match object² as the first argument to the Callable. The :g adverb directs .subst to do the search/replace repeatedly for as many matches as there are in the invocant string.

• slurp generates a string in one go from a file. No need for open, using handles, close, etc. Its result in this case becomes the invocant of the .subst explained above.

• spurt does the opposite, generating a file in one go from a string, in this case the results of the slurp(...).subst... operation.

• The Evaluate routine generates a string that's the output from evaluating the string of code passed to it. To capture the result of evaluation it temporarily modifies Raku's STDOUT variable $*OUT, redirecting prints (and thus also says etc.) to the internal variable $out before EVALing the code. If the EVAL results in anything being printd to $out then that is returned; if not, then the result of the EVAL is returned (coerced to a string by the ~). (A newline is appended in this second scenario but not the first because that is what's needed to get the correctly displayed result given how you've "specified" things by your example.)

• The Replace routine is passed a match object from a call of the Code regex. It reconstructs the code section (without the evaluate bit) using the $<code> capture. If the $<evaluate> capture is 'TRUE' then it also appends a fresh Output: section using the Evaluate routine explained above to produce the code's output.

• The Code regex matches a code section. It captures the TRUE or FALSE setting from the evaluate directive into a capture named $<evaluate> and the code into a capture named $<code>.

Footnotes

¹ To pass a routine (a regex is a routine) rather than call it, it must be written with a sigil (&foo), not without (foo).

² It does this even if the matcher was merely a string!

Hygro

Great answer! Is there an easy modification of the code that allows the modified context (environment) between the code cells to be shared? E.g. assigned variables or loaded packages. – Xenolith

Please see this screenshot: imgur.com/5MyoD4a . For the last cell I get the message "Variable '$answer' is not declared". – Xenolith

I resolved these non-persistent state problems using a greatly reduced version for the class Sandbox of the package Jupyter::Kernel. – Xenolith

Hi @AntonAntonov. I ended up spending a couple hours playing around with various options. I introduced a new declarative repl. This rewrote each Code: section with that declarative as if it were entered into a REPL. Each line of code prefixed with > , then output, and with declarative effects retained. But I concluded you might want other stuff like proper handling of exceptions, and that what I was doing would quite likely better be done using the new repl function Liz has recently introduced. I'd love to see what you came up with if/when you're ready to share it. – Hygro

Thank your efforts and update! Here is a GitHub repository with my solution: Raku-Text-CodeProcessing. – Xenolith

"[...] using the new repl function Liz has recently introduced." -- I did not know about that new repl function. :) Is there a place I can find out more about it? – Xenolith

@AntonAntonov I found this. – Hygro

Answer 2

There are two ways to execute the code and capture the output:

• You can write it to a tempfile and use my $result = qqx{perl6 $filename} to spawn a separate process
• You can execute the code in the same interpreter using EVAL, and use IO::Capture::Simple to capture STDOUT:

my $re = regex {
    ^^ # logical newline
    '```{perl6 evaluate=' (TRUE|FALSE) '}'
    $<code>=(.*?)
    '```'
}

for $input.match(:global, $re) -> $match {
    if $match[0] eq 'TRUE' {
        use IO::Capture::Simple;
        my $result = capture_stdout {
            use MONKEY-SEE-NO-EVAL;
            EVAL $match<code>;
        }
        # use $result now
    }
}

Now you just need to switch from match to subst and return the value from that block that you want to substitute in, and then you're done.

I hope this gives you some idea how to proceed.

Wobble

Thank you. Your code worked fine but while evaluating 4/5 in one chunk, its throwing Any as output. What could be the reason its not 0.8? – Monometallic

The code I posted just looks at output to STDOUT, and 4/5 doesn't produce any output. Change it say 4/5. You could also check return value from EVAL, but my experience with IRC bots is that it's confusing if both output and return value are printed. – Wobble

Answer 3

You can try this regex:

```{perl6 evaluate=(?<evaluate>[^}]+)}\s+(?<code>[^`]+)

You will get three results from your sample text, each result contains two named groups, the first is evaluate, containing the flag and the second one codeis the code.

Have a look at the regex-demo:
https://regex101.com/r/EsERkJ/1

Since I don't know perl, i can't help you with the implementation :(

Meatiness

Perl6 has a redesigned regex engine. (Which makes it easier to understand and write once you get used to it.) For example (?<a>…) is now written $<a> = (…). (You can get back to the Perl5 style of regex using the :P5 adverb.) Also every non-letter character and space must be escaped if you want it to match literally. Character classes are written differently as well. "{perl6 evaluate=" $<evaluate> = (<-[}]>+) "}" (A regex is treated as code with a different base syntax. They can have parameters and variables.) – Sanmicheli