New CongoCC features worth checking out

by

CongoCC is a modern successor to JavaCC that is undergoing continuous development. Here are two new features that Jon Revusky brought to my attention.

Cardinality Constraints

When writing a grammar, the question always arises: Should I write a simple grammar and address semantics in a second step, or should I overload the grammar with details? I’ve borrowed a simple example here from the Rocinante project.

Proto :
#if rosi
  Syntax
  Package
  (Import)*
  ( Option | Message | Enum )*
#else
  ( Syntax | Package | Import | Option | Message | Enum | <SEMICOLON>)*
#endif
  // Service
  <EOF>
;

The Proto production of the grammar describes the basic structure of a Protocol Buffer definition. In the original, it consists of an arbitrary sequence of Syntax, Package, Import, Option, Message, and Enum expansions. This lax approach to the grammar bothered me from the very beginning, so here is an alternative format that can be used for a ROSI variant. Here, Syntax and Package are required, and their positions in the file are fixed; they are followed by optional Import expansions, and only then do arbitrary sequences of Option, Message, and Enum sections appear. This form is more appealing to a Java developer than the original form, but perhaps that’s just my impression.

Of course, the grammar form shown above has a weakness that must be addressed elsewhere. Although specifying only one Syntax and one Package makes sense, any number is allowed here. The existing Rocinante grammar solves this problem with the following productions.

Package #PackageProduction :
{
  Node packageNode;
}
  <PACKAGE>
#if rosi
#else
  {
    if (!"".equals(packageName)) {
      throw new ParseException("only one package production allowed");
    }
  }
#endif
  ( packageNode = <FULL_IDENTIFIER> | packageNode = <IDENTIFIER> )
  <SEMICOLON>
  {
    packageName = packageNode + ".";
  }
;

Syntax #SyntaxProduction :
  <SYNTAX>
#if rosi
#else
  {
    if (protoVersion != null)  {
      throw new ParseException("only one syntax production allowed");
    }
  }
#endif
  <ASSIGN>
  <STRING_LITERAL>
  {
    protoVersion = switch (lastConsumedToken.toString()) {
#if rosi
      case "\"rosi1\"" -> ProtoVersion.ROSI1;
#endif
      case "\"proto2\"" -> ProtoVersion.PROTO2;
      case "\"proto3\"" -> ProtoVersion.PROTO3;
      default -> throw new ParseException(lastConsumedToken);
    };
  }
  <SEMICOLON>
;

Both implementations use the same pattern. They check whether the corresponding value has already been stored in the parser—that is, whether it is no longer null—and, if so, throw a ParseException. The ROSI variant does not need to take any action in this case, since its grammar does not allow for multiple Syntax and Package expansions.

With the Cardinality Constraints in CongoCC, this is no longer necessary.

Proto :
#if rosi
  Syntax
  Package
  (Import)*
  ( Option | Message | Enum )*
#else
  ( &&Syntax | &Package | Import | Option | Message | Enum | <SEMICOLON>)+
#endif
  // Service
  <EOF>
;

The difference from the original rule is the use of the prefixes && before Syntax and & before Package. These, along with three other variants, limit the frequency of expansions within a loop. In our case, a Syntax expansion must occur exactly once, and a Package expansion is optional and may occur at most once.

That’s not the exact protocol buffer syntax, but this is how we can introduce both prefixes.

Since our production now controls cardinality much more effectively, we can simplify things in the other two production.

Package #PackageProduction :
{
  Node packageNode;
}
  <PACKAGE>
  ( packageNode = <FULL_IDENTIFIER> | packageNode = <IDENTIFIER> )
  <SEMICOLON>
  {
    packageName = packageNode + ".";
  }
;

Syntax #SyntaxProduction :
  <SYNTAX>
  <ASSIGN>
  <STRING_LITERAL>
  {
    protoVersion = switch (lastConsumedToken.toString()) {
#if rosi
      case "\"rosi1\"" -> ProtoVersion.ROSI1;
#endif
      case "\"proto2\"" -> ProtoVersion.PROTO2;
      case "\"proto3\"" -> ProtoVersion.PROTO3;
      default -> throw new ParseException(lastConsumedToken);
    };
  }
  <SEMICOLON>
;

Since both productions can only be applied once, the corresponding checks can be removed, making both productions much easier to read.

A similar problem arose in the EnumBody production. Here, any number of Option and EnumField expansions may occur, but at least one EnumField expansion must occur.

EnumBody :
  <LBRACE>
#if rosi
  ( Option )*
  ( EnumField )+
#else
  ( Option | EnumField | <SEMICOLON> )*
  #endif
// reserved
  <RBRACE>
;

The ROSI variant solved this using a stricter syntax that expands every Option first and then every EnumField. In the original version, this must be checked in a later step. With the new Cardinality Constraints, this is now much easier.

EnumBody :
  <LBRACE>
#if rosi
  ( Option )*
  ( EnumField )+
#else
  ( Option | &1:&EnumField | <SEMICOLON> )*
  #endif
// reserved
  <RBRACE>
;

The prefix &1:& now forces the production to ensure that EnumField must be expanded at least once; otherwise, the production is not satisfied. It is interesting to note here that the loop with * is accepted because it must, in fact, be executed at least once. You can find more information about this feature at https://discuss.congocc.org/d/77-new-feature-cardinality-constraints

Contextual Keywords

Another new feature is Contextual Keywords. A test for the first feature immediately provided me with an example of this feature. The Protocol Buffer documentation includes the following example: https://protobuf.dev/programming-guides/enum/ for an enum definition.

syntax = "proto3";

enum Enum {
  A = 0;
  B = 1;
}

message Msg {
  repeated Enum enum = 1;
}

Here, an enum is defined whose type is named Enum, and it is used as an attribute enum in the message.

It’s no coincidence that this example doesn’t appear as-is in the Java section on known issues. It would likely be another known issue.

The problem with the original Rosinante grammar is that enum cannot be used as an identifier because it is recognized as a token.

TOKEN #KeyWord :
  < SYNTAX : "syntax">
| < IMPORT: "import" >
| < WEAK: "weak" >
| < PUBLIC: "public" >
| < PACKAGE: "package" >
| < OPTION: "option" >
| < MESSAGE: "message" >
| < ENUM: "enum" >
| < REPEATED: "repeated" >
| < OPTIONAL: "optional" >
;

This is a bit annoying because there’s no reason to prohibit the text enum elsewhere. CongoCC now offers the option to recognize such keywords only where they are expected in the grammar. Elsewhere, they are simply identifiers. This is a somewhat simplified explanation, but grammars and their productions have never been simple.

In CongoCC, there are two ways to define such keywords: either inline as strings in single quotes 'enum' or within a CONTEXTUAL block. I prefer the former because it lets you see all Contextual Keywords at a glance, rather than having them pop up unexpectedly in the productions.

CONTEXTUAL #ContextualKeyWord :
  < ENUM: "enum" >
;

With this variant, enum is now recognized as an identifier within messages and could be used as an attribute name if Java did not prohibit it. More information about this feature can be found at https://discuss.congocc.org/d/72-new-feature-contextual-aka-soft-keywords.

Those were the two new features in CongoCC that I wanted to introduce. If you’re still using JavaCC, please stop using it once and for all!

Leave a Comment