“It doesn’t stop being magic just because you know how it works”
Terry Pratchett
Sometimes you develop things while on vacation because you’ve always wanted to try them out but don’t really see any practical use for them. That’s the case with FreshMarker APT. APT stands for Annotation Processed Templating and simply means applying the FreshMarker template syntax to Java code in a way similar to String Templates.
As everyone probably knows, String Templates were introduced as a pre-release preview feature in JDK 21 but were removed again in JDK 23 for various reasons.
String interpolation(String husband, String employer, String wife) {
return STR."""
My husband, \{husband}, has passed away
Dear \{employer},
I am writing to inform you with great sorrow that my husband, \{husband}, has passed away.
This is a difficult time for our family, and I wanted to ensure you were aware of his passing.
Sincerely,
\{wife}
""";
}
This is a simple example of a string template using the STR string processor. This processor can evaluate expressions within the curly braces and insert the result. Therefore, when called with the appropriate parameters, this method returns the following text.
My husband, Willy Loman, has passed away
Dear Mr. Howard Wagner,
I am writing to inform you with great sorrow that my husband, Willy Loman, has passed away.
This is a difficult time for our family, and I wanted to ensure you were aware of his passing.
Sincerely,
Linda Loman
How can you replicate something like this using FreshMarker and Java‘s built-in tools? A simple method call would be the easiest way.
String interpolation(String husband, String employer, String wife) {
return FreshMarkerHelper.process(
"""
My husband, ${husband}, has passed away
Dear ${employer},
I am writing to inform you with great sorrow that my husband, ${husband}, has passed away.
This is a difficult time for our family, and I wanted to ensure you were aware of his passing.
Sincerely,
${wife}
""",
Map.of("husband", husband, "employer", employer, "wife", wife));
}
In this case, we simply call the static method of a helper class within our method. This is too trivial for a quick exercise during vacation, but we’ll come back to this static method later.
What I have in mind is the following approach using an annotation:
String interpolation(String husband, String employer, String wife) {
@FreshMarker
String message = """
My husband, ${husband}, has passed away
Dear ${employer},
I am writing to inform you with great sorrow that my husband, ${husband}, has passed away.
This is a difficult time for our family, and I wanted to ensure you were aware of his passing.
Sincerely,
${wife}
""";
return message;
}
The variable declaration is annotated with @FreshMarker, which means that the message variable will not contain the specified string, but rather the result of the FreshMarker evaluation using the local variables and method parameters.
This task is performed in the background by an annotation processor. There have been several posts on annotation processors and their use as Hamcrest Matcher Generators, Enum Converter Generators, or Diff Evaluator Processors.
This annotation processor differs slightly from the processors described so far. Those processors all generated additional Java source code, such as MapStruct, whereas the new processor, like Lombok, inserts code directly into the modified class.
@SupportedAnnotationTypes("*")
public class FreshMarkerProcessor extends AbstractProcessor {
@Override
public SourceVersion getSupportedSourceVersion() {
return SourceVersion.latestSupported();
}
@Override
public boolean process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) {
if (roundEnv.processingOver() || trees == null) {
return false;
}
FreshMarkerScanner freshMarkerScanner = new FreshMarkerScanner();
for (Element rootElement : roundEnv.getRootElements()) {
TreePath path = trees.getPath(rootElement);
if (path == null) {
continue;
}
var cut = path.getCompilationUnit();
new FreshMarkerScanner().scan(new TreePath(cut), null);
}
return true;
}
}
The FreshmarkerProcessor is annotated with @SupportedAnnotationTypes("*") to ensure that it actually considers all classes, since the annotation processor does not recognize annotations on local variables. The getSupportedSourceVersion method ensures that the annotation processor always works with the latest Java version.
The actual work starts in the process method. It is passed a set of annotations to be processed and the RoundEnvironment. At the beginning of the method, it checks whether processing has already completed or whether a required initialization failed. In either case, processing is terminated with false, and another annotation processor can try its luck. Since our annotation processor is annotated with @SupportedAnnotationTypes("*"), the set of annotations is empty, and we retrieve all our top-level elements via roundEnv.getRootElements(). These top-level elements are processed by a TreePathScanner.
private class FreshMarkerScanner extends TreePathScanner<Void, Void> {
@Override
public Void visitVariable(VariableTree node, Void unused) {
if (hasAnnotation(node)) {
rewrite(getCurrentPath());
}
return super.visitVariable(node, unused);
}
}
The TreePathScanner implements the Visitor Pattern and traverses the entire Abstract Syntax Tree (AST) of the parsed class. However, since we are only interested in the subtree of the variable, we override only the visitVariable method. Here, we check whether the variable is annotated with @FreshMarker and then manipulate the AST.
The way an annotation processor works is entirely different from, say, working with reflections. Here, we do not process structural elements of a class; instead, we still work on the source code level. This is clearly illustrated by the hasAnnotation method.
private boolean hasAnnotation(VariableTree node) {
for (AnnotationTree ann : node.getModifiers().getAnnotations()) {
CharSequence name = switch (type) {
case IdentifierTree id -> "org.freshmarker.apt." + id.getName();
case MemberSelectTree ms -> ms.toString();
default -> "";
};
if ("org.freshmarker.apt.FreshMarker".contentEquals(name)) {
return true;
}
}
return false;
}
The node parameter is of type VariableTree and describes the entire variable declaration. The expression node.getModifiers().getAnnotations() returns a list of the annotations found, each of which is defined as an AnnotationTree. The corresponding annotation type is also defined as a tree. In this case, it can be an IdentifierTree for @Freshmarker or a MemberSelectTree for @org.freshmarker.apt.FreshMarker. To ensure we correctly identify the annotation in both cases, we check the name against "org.freshmarker.apt.FreshMarker" in each instance. For the IdentifierTree, we add the prefix "org.freshmarker.apt.". As you can see, we work only on elements of the AST; classes or interfaces, as we know them from reflection, are completely absent.
After identifying the locations where we want to modify the class, we must now manipulate the AST so that the generated class contains our code. There is something to keep in mind here. There is no official API for manipulating the AST; what libraries like Lombok or our little tool do is use an internal library because it just happens to work. Changes to the Java compiler could break such tools at any time. Another interesting note about Lombok: in order to access many of the internal classes, we must enable their modules for our application.
--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED
These switches are required not only by our project to compile our tool, but also by any project that wants to use our tool. Anyone who has worked with Lombok before won’t remember having to configure these switches. From this, we can conclude two things. First, Java’s module system isn’t yet as secure as it should be, and second, Lombok uses various workarounds to bypass the module system. Let’s wish the OpenJDK team the best of luck in closing all these gaps soon.
But for now, let’s start by modifying the class.
private void rewrite(TreePath variablePath) {
Tree leaf = variablePath.getLeaf();
if (!(leaf instanceof JCVariableDecl varDecl)) {
printErrorMessage("@FreshMarker can only be applied to local variable declarations", variablePath);
return;
}
if (varDecl.init == null) {
printErrorMessage("@FreshMarker variable must have an initializer string literal", variablePath);
return;
}
JCExpression unwrapped = unwrapParens(varDecl.init);
if (unwrapped instanceof JCTree.JCLiteral literal && literal.value instanceof String templateString) {
java.util.List<String> localNames = collectPrecedingLocals(variablePath);
boolean isStatic = isEnclosingMethodStatic(variablePath);
boolean isShared = isSharedAnnotation(variablePath);
varDecl.init = buildHelperCall(templateString, localNames, isStatic, isShared, varDecl.pos);
return;
}
printErrorMessage("@FreshMarker initializer must be a plain string literal", variablePath);
}
First, we check whether it is a variable declaration with an initializer. If it is not, there will be a meaningful error message in the compiler output. Then we take a closer look at the initializer, and if it’s a string literal, we first collect all local variables preceding the variable declaration and determine whether the variable declaration is inside a static or non-static method, and whether we should reuse the FreshMarker configuration. Generating the configuration over and over again is wasteful, but there may be reasons to do so. Then we take all this information and build a new initializer in the buildhelperCall method.
private JCExpression buildHelperCall(String template, java.util.List<String> localNames, boolean isStatic, boolean isShared, int pos) {
treeMaker.pos = pos;
JCExpression helperClass = qualifiedName(pos, "org", "freshmarker", "apt", "FreshMarkerHelper");
String methodName = isShared ? "processShared" : "process";
JCExpression processMethod = treeMaker.Select(helperClass, names.fromString(methodName));
JCExpression templateLiteral = treeMaker.Literal(template);
JCExpression instanceRef = isStatic
? treeMaker.Literal(com.sun.tools.javac.code.TypeTag.BOT, null)
: treeMaker.Ident(names._this);
JCExpression mapArg = buildMapOf(localNames, pos);
return treeMaker.Apply(List.nil(), processMethod, List.of(templateLiteral, instanceRef, mapArg));
}
In this method, we construct an expression that represents a method call to our utility class shown at the beginning. If we have set the isShared flag, we call the FreshMarkerHelper#processShared method; otherwise, we call the FreshMarkerHelper#process method. Both call an identical private method—one with a lazily created, shared instance, and the other with a freshly created one. Unlike the method we used above, both methods have an additional parameter that passes in the calling instance. This allows the templates to access the class’s attributes as well. For this parameter, it was important to know whether the method was static. In that case, null must be passed because an instance cannot be accessed. Finally, the new initializer is created and returned.
It’s very clear from this code that the classes used here weren’t really designed to manipulate an AST after the fact. Perhaps it’s time to consider a clean API.