Your workflow when adding a new feature to a translator

0. Decide the syntax and semantics of the new feature

When you are adding a new feature to a translator, you have to decide the syntax and semantics of the new feature.

1. Try writing different program examples of the new feature playing with the expressiveness of different syntactical approaches and also considering possible semantic variants. For instance, if you are adding while loops to a language, you can try a syntax like while or while do etc. You can also consider if the while loop will return a value or not. Or if a while creates a new scope or not:

➜  functions-solution git:(while) cat test/data/input/while.calc 
a = 0,
b = while a < 10 do { # or while a < 10 {
  print(a),
  a = a +1
},
print(b) # 10

discuss with other people. Decide the syntax and semantics for the new feature.

2. Add new rules to your grammar to support the new feature.
3. Avoid ambiguity in the grammar. Check with npx jison src/grammar.jison that the grammar is not ambiguous.
4. Modify the lexical analyzer if new tokens appear.
5. Decide the name and parameters of the function that will be used to build the AST for the new feature.

For instance, when the new feature is while loops, you can add the following rule to the grammar:

  | WHILE e '{' e '}'   { $$ = buildWhileExpression($e1, $e2); }

In the example, a new token appears, WHILE, and a new function like buildWhileExpression will be needed.

1. Write a simplistic example of the new feature

Now you write one or more program examples according to the selected syntax. For the while loops, you can write (future) very simple tests for the new feature in the original language

➜  functions-solution git:(while) cat test/data/input/while.calc 
a = 0,
b = while a < 10 {
  print(a),
  a = a +1
},
print(b) # 10

2. Write the JS code you expect for such example

Start writing by hand the translation you expect for such example.

You don't need to do everything, only the specific part of the code that you are working on. In the example below you concentrate on lines 10-20 below that define the translation scheme for the while loop in the code above.

➜  functions-solution git:(while) npx prettier test/data/expectedjs/while.js 
#!/usr/bin/env node
const { Complex, print, } = require("/Users/casianorodriguezleon/campus-virtual/2223/pl2223/practicas/functions/functions-solution/src/support-lib.js");
/* End of support code */
 
let $a, $b;
(
    ($a = Complex("0")),
    (
        $b = (
        () => {
            let result = false;
            while ($a.lessThan(Complex("10"))) {
                result = (
                  print($a), 
                  ($a = $a.add(Complex("1")))
                );
            }
            return result;
        })()
    )
),
print($b);

3. Use tools to learn the shape of the AST to be generated

In this step you

1. Will use tools to generate the AST for the code you wrote by hand in the previous step. Typically you will use any parser available for your target language. In this case our target language is JavaScript.
2. Most parsers allow to generate the AST in JSON format.
3. Then use tools to navigate the AST and learn the shape of the AST for the construct you are interested in.
4. Detect the node you are interested in and its properties.

I'll show it here using version 4.1.0 of compast to generate the AST for the code above but you can use the AST Explorer or any JS parser for the job.

✗ npm i -g compact-js-ast@4.1.0
✗ compast --version
4.1.0

Here are the options supported in this version of compast:

✗ compast --help
Usage: compast [options] [filename]

Converts a JS program into a JSON or YML AST format

Arguments:
  filename                    file with the original code

Options:
  -V, --version               output the version number
  -o, --output <filename>     file name of the json putput program
  -p, --program <JS program>  JS program is given in the command line
  -jw --whites <string>       string '  ' Specifies the number of whites for formatting the object
                              (default: "  ")
  -e --hide <fieldnames...>   List of AST fields to omit (default: [])
  -f --hideFile <fileName>    File with a line per AST fields to omit
  -j --json                   output in JSON format (default is YML
  -n --no-parse               do not parse the code, assume the input is already an AST in json format
  -a --all                    output all fields
  -l --location               omit only location fields
  -h, --help                  display help for command

Now generate the AST for the code you expect for the while loop in the code above. The -j option is used to generate the AST in JSON format. The -o option is used to specify the output file.

✗ compast -j test/data/expectedjs/while.js -o test/data/expectedAST/while.json

VSCode

1. Edit the JSON with VSCode and get familiar with the different nodes and their properties.
2. By navigating in the JSON you can detect that the node you are interested in is the CallExpression node with callee an ArrowFunctionExpression.

jq and jless

There are command line tools that can help you navigate JSON in the command line.

Namely we can use the jq language to specify the node of the AST we are interested in:

✗ jq '.body[2].expression.expressions[0].expressions[1].right' test/data/expectedAST/while.json
| jless

Notice the pipe to the jless a utility, which can be used to fold/unfold and navigate any JSON in the command line:

4. Substitute the fixed parts by your AST parameters

Now you are in condition to start writing the buildWhileExpression function that has to build the AST producing the designed translation.

buildWhileExpression(test, body) {
   return <Paste here the output of the JSON produced in the previous step>
}

Substitute the fixed ASTs by your variable part: test and body that were built by the parser:

  | WHILE e '{' e '}'   { $$ = buildWhileExpression($e1, $e2); }

and you are almost done!

5. Consider scope problems and type checking

You have to consider scope problems, type checking and interactions with the remaining code. If you decide that the loop creates a new scope it may alter the code in the scope analysis phase.

6. Add functions to the support library

Is not the case in this example, but sometimes you need to add code to the support library to easy the translation of the new feature. For instace if you are adding strings to the language, you may need to monkey patch the String class to support operators like +.

7. Add tests

Finally, you have to convert the previous example onto a test and new tests for the new feature.

Name the tests after the name of the feature son that can be easily run separately with mocha --grep:

functions-solution git:(while) ✗ npx mocha --grep while test/test.mjs 

  ✔ transpile(while.calc, while.js) (No errors: true) (105ms)
  ✔ transpile(while-while.calc, while-while.js) (No errors: true) (118ms)
  ✔ transpile(while-sum.calc, while-sum.js) (No errors: true) (101ms)
  ✔ transpile(while-nested.calc, while-nested.js) (No errors: true) (123ms)

  4 passing (450ms)