Establish a common terminology for non-terminals

[huperniketes]

I'm working on an application that analyzes source code. The lack of uniform terminology for non-terminals serving the same purpose in other programming languages is quite problematic for me. Perhaps others have written tools which have limited applicability due to the incompatibility between grammars.

There was a previous discussion on establishing a single format for the various grammars #3816.

I hope a collection of grammars for ANTLR4 conforming to a single convention for terminology already exists, or there is a willingness of the community to establish and adhere to one if not.

[kaby76]

Could you explain what you mean by "common terminology"? Is it a naming convention, or are you looking at a common, reusable set of grammar rules?

[huperniketes]

I apologize for not including examples.

The starting (root) symbol for several grammars are unique. For JavaScript/TypeScript it is "program".
For PHP, “htmlDocument”.
C++: “translationUnit”.
Ruby: “prog”.

On the other hand, there are terms which serve the same purpose across some grammars (eg, ifStatement, assignmentExpression, etc.)

I think it would prove helpful to tools and applications which incorporate a grammar from the repo to find they can be used with the grammars of other programming languages or DSL's.

[kaby76]

The starting (root) symbol for several grammars [is] unique. For JavaScript/TypeScript it is "program". For PHP, “htmlDocument”. C++: “translationUnit”. Ruby: “prog”.

That's true, the start rule name isn't consistent. But, since the start rule really should be EOF-terminated, it's not hard to find the start rule in the grammar, rather than assume the start rule name.

Antlr4 does not force start rules to be EOF-terminated, but this can lead to serious parsing problems. I just discovered this in a grammar in the last week. spicelang/spice#661 (comment)

If there are multiple .g4s in the grammar, a topological sort of the grammars can determine which grammars are "top-level" grammars, i.e., those that should be processed by the Antlr4 Tool. Only top-level grammars should be processed, but it is possible to have "start rules" in any .g4. Antlr4 was specifically designed to allow this.

In some grammars, when analysis fails, the desc.xml is used to declare what it should be.

On the other hand, there are terms [that] serve the same purpose across some grammars (e.g., ifStatement, assignmentExpression, etc.)

That's true. https://lionweb.io/ is an effort to specify a common AST meta. Unfortunately, Antlr4 supplants AST construction with the CST.

[huperniketes]

I'm only suggesting use of common terms (ontology) in ANTLR grammars used for programming languages.

What I gather is there's nothing which currently addresses this issue.

[kaby76]

Some syntax is defined to avoid grammatical ambiguity. Java partitions if-statements into "no short form" vs. a "short form" https://docs.oracle.com/javase/specs/jls/se23/html/jls-14.html#jls-14.5. (NB: this has nothing to do with the tertiary operator.) This is done to avoid the "dangling else" problem.

For example,

grammar XXX;

start : ifstmt EOF;
ifstmt : 'if' expr 'then' stmt ('else' stmt)? ;
expr : NUM;
NUM : [0-9]+;
stmt : ID '=' expr | ifstmt;
ID : [a-zA-Z]+;
WS : [ \t\n\r] -> skip;

This grammar is ambiguous for input if 1 then if 9 then a = 2 else b = 3.

$ dotnet trparse --ambig --input 'if 1 then if 9 then a = 2 else b = 3' | dotnet trtree -a
CSharp 0 string success 0.0301816
(start (ifstmt (T__1 "if") (expr (NUM "1")) (T__2 "then") (stmt (ifstmt (T__1 "if") (expr (NUM "9")) (T__2 "then") (stmt (ID "a") (NUM "=") (expr (NUM "2"))) (T__3 "else") (stmt (ID "b") (NUM "=") (expr (NUM "3")))))) (EOF ""))
(start (ifstmt (T__1 "if") (expr (NUM "1")) (T__2 "then") (stmt (ifstmt (T__1 "if") (expr (NUM "9")) (T__2 "then") (stmt (ID "a") (NUM "=") (expr (NUM "2"))))) (T__3 "else") (stmt (ID "b") (NUM "=") (expr (NUM "3")))) (EOF ""))

Java solves this in the JLS by defining "no short" productions. So, grammar XXX would be corrected.

grammar XXX;

start : ifstmt EOF;
ifstmt : 'if' expr 'then' stmtnoshortif ('else' stmt)? ;
ifstmtnoshortif : 'if' expr 'then' stmtnoshortif 'else' stmtnoshortif ;
expr : NUM;
NUM : [0-9]+;
stmt : ID '=' expr | ifstmt;
stmtnoshortif : ID '=' expr | ifstmtnoshortif;
ID : [a-zA-Z]+;
WS : [ \t\n\r] -> skip;

$ dotnet trparse --ambig --input 'if 1 then if 9 then a = 2 else b = 3' | dotnet trtree -a
CSharp 0 string success 0.0156092
10/09-05:31:21 ~/temp2/Generated-CSharp
$ dotnet trparse --input 'if 1 then if 9 then a = 2 else b = 3' | dotnet trtree -a
CSharp 0 string success 0.0142416
(start (ifstmt (T__1 "if") (expr (NUM "1")) (T__2 "then") (stmtnoshortif (ifstmtnoshortif (T__1 "if") (expr (NUM "9")) (T__2 "then") (stmtnoshortif (ID "a") (NUM "=") (expr (NUM "2"))) (T__3 "else") (stmtnoshortif (ID "b") (NUM "=") (expr (NUM "3")))))) (EOF ""))

If I wanted to write a program that analyzes the parse tree to count the number of if-statements, I would have to count not only the number of ifstmt but ifstmtnoshortif.

$ dotnet trparse --input 'if 1 then if 9 then a = 2 else b = 3' | dotnet trxgrep ' //(ifstmt | ifstmtnoshortif)' | dotnet trtext -c
CSharp 0 string success 0.0144677
2

However, this could be easily rewritten to an AST that has only one "ifstmt" node type.

How would you handle this if an if-statement is just called "ifStatement" across all programming language grammars?