<!DOCTYPE html>
<html>
<head>
<title>Annotation File Utilities</title>
</head>
<body>
<h1>Annotation File Utilities</h1>
<p>Contents:</p>
<!-- start toc. do not edit; run html-update-toc instead -->
<ul>
<li><a href="#motivation">Motivation</a>
<ul>
<li><a href="#jaif-file">External storage of annotations</a></li>
<li><a href="#annotation-file-utilities-description">Annotation File Utilities</a></li>
</ul></li>
<li><a href="#installation">Installation</a>
<ul>
<li><a href="#viewing-source">Building from source</a></li>
</ul></li>
<li><a href="#using">Using the Annotation File Utilities</a>
<ul>
<li><a href="#insert-annotations">Insert-annotations</a></li>
<li><a href="#extract-annotations">Extract-annotations</a></li>
<li><a href="#insert-annotations-to-source">Insert-annotations-to-source</a></li>
</ul></li>
<li><a href="#implementation">Design and Implementation Details</a>
<ul>
<li><a href="#asmx">Asmx</a></li>
<li><a href="#scene-lib">Scene-lib</a>
<ul>
<li><a href="#bytecode-insertion">Bytecode Insertion</a></li>
<li><a href="#bytecode-extraction">Bytecode Extraction</a></li>
</ul></li>
<li><a href="#source-insertion">Annotation-file-utilities</a></li>
</ul></li>
<li><a href="#feedback">Feedback and bug reports</a>
<ul>
<li><a href="#changelog">Changelog</a></li>
</ul></li>
</ul>
<!-- end toc -->
<hr />
<h2 id="motivation">Motivation</h2>
<p>
Java annotations are meta-data about Java program elements, as in
“<code><mark>@Deprecated</mark> class Date
{ ... }</code>” or “<code>List<<mark>@NonNull</mark> String></code>”.
Ordinarily, Java annotations are
written in the
source code of a <code>.java</code> Java source file. When
<code>javac</code> compiles the source code, it inserts the annotations in
the resulting <code>.class</code> file (as
“attributes”).
</p>
<!--
I've moved text from here to the annotation-file-format document, but feel
free to improve that document.
-->
<h3 id="jaif-file">External storage of annotations</h3>
<p>
Sometimes, it is convenient to specify
the annotations outside the source code or the <code>.class</code> file.
The document
“Annotation File Format Specification” (<a
href="annotation-file-format.pdf">PDF</a>, <a
href="annotation-file-format.html">HTML</a>)
defines a textual format for annotations,
and it also motivates reasons why such a file format is necessary in
addition to the <code>.java</code> and <code>.class</code> formats. The
file format supports both the declaration annotations and type annotations.
</p>
<p>
An annotation file
conventionally has the extension <code>.jaif</code> (for Java Annotation Index
File).
The <a href="#scene-lib"><code>scene-lib</code></a> sub-project provides
API methods for building and manipulating annotation files.
</p>
<h3 id="annotation-file-utilities-description">Annotation File Utilities</h3>
<p>
Programmers need to be able to transfer annotations
between the three possible locations for annotations — source code, class files,
and annotation files. Programmers will want to extract
annotations from source and class files to an annotation file in order to easily
read annotations, while various tools will only read annotations from
source and class files. The Annotation File Utilities provide three tools
to read and write annotation files.
</p>
<ul>
<li> <a href="#insert-annotations"><code>insert-annotations</code></a> reads annotations from an annotation file
and inserts them into a class file</li>
<li> <a href="#extract-annotations"><code>extract-annotations</code></a> reads annotations from a class file
and writes them out to an annotation file</li>
<li> <a href="#insert-annotations-to-source"><code>insert-annotations-to-source</code></a> reads annotations from an
annotation file and inserts them into a Java source file</li>
</ul>
<p>
A diagram showing the interactions betweeen these tools is below.
</p>
<div>
<img src="figures/tool-relations.svg" alt="Relationships between AFU tools" />
</div>
<p>
There is no
<code>extract-annotations-from-source</code> tool: one can
compile the source code and then use
<code>extract-annotations</code> to read the annotations from the class
file.
</p>
<hr />
<h2 id="installation">Installation</h2>
<p>
The following instructions assume either a Linux or Windows system using a command-line environment.
</p>
<p>
The current release is Annotation File Utilities version
<!-- afu-version -->3.6.44, 03 Aug 2017<!-- /afu-version -->.
</p>
<ol>
<li>
Download
<a href="annotation-tools-3.6.44.zip"><!-- annotation-tools-zip-version -->annotation-tools-3.6.44.zip<!-- /annotation-tools-zip-version --></a>.
</li>
<li>
Creade a directory named
<code>annotation-tools</code> by
unpacking the distribution zipfile (a standard place to do this is in a
directory <code>~/jsr308/</code>):
<pre><code>unzip annotation-tools-3.6.44.zip</code></pre>
The <code>annotation-tools</code> directory must be a sibling of the <code>jsr308-langtools</code>
directory (available at <a href="https://checkerframework.org/jsr308/">https://checkerframework.org/jsr308/</a>).
Alternatively, Unix (including Linux and MacOS) users may set the <code>LANGTOOLS</code> environment variable to
the location of the <code>jsr308-langtools</code> directory:
<pre><code>export LANGTOOLS=<em>/path/to</em>/jsr308-langtools</code></pre>
</li>
<li>
Add the <code>annotation-file-utilities</code> directory to your path.
<ul>
<li>
For <b>Unix</b> (including Linux and MacOS), add the directory to your PATH
environment variable. If your shell is sh or bash, add to your
<code>~/.bashrc</code> or <code>~/.bash_profile</code> file:
<pre><code>export PATH=${PATH}:<em>/path/to</em>/annotation-tools/annotation-file-utilities/scripts</code></pre>
<!-- Omitted to save space and simplify instructions
For csh/tcsh, add to ~/.tcshrc or ~/.cshrc or ~/.login:
<pre><code>setenv PATH=${PATH}:/path/to/annotation-file-utilities/</code></pre>
-->
</li>
<li>
For <b>Windows</b>, add the directory to your
<code>PATH</code> system
variable by going to
<pre><code> Control Panel -> System -> Advanced -> Environment Variables </code></pre>
From there, find the <code>PATH</code> variable under “System variables”
and append to it the directory <code><em>path\to</em>\annotatation-tools\annotation-file-utilities\scripts</code>.
</li>
</ul>
</li>
</ol>
<!-- I think this is obvious, so I have commented it out -MDE. -->
<!--
<p>
To update the annotation file utilities, simply download the most recent <code>annotation-tools.zip</code> file from this website and replace the existing <code>annotation-tools.zip</code>, then extract the file just as when you first installed it. As long as you followed the above instructions, no further work needs to be done.
</p>
-->
<h3 id="viewing-source">Building from source</h3>
<p>
The annotation file utilities are pre-compiled (a jar file is included in
the distribution), so most users do not need to compile it themselves.
</p>
<p>
There are two ways to obtain the source code.
Source code is provided in the
<a href="https://github.com/typetools/annotation-tools/releases">distribution</a>.
Alternately, see the source code repository at
<a href="https://github.com/typetools/annotation-tools">https://github.com/typetools/annotation-tools</a>.
</p>
<p>
To compile, run
<code>ant jarfile</code> from the <code>annotation-file-utilities</code>
subdirectory. If the <code>javac</code> and <code>java</code> programs are
not on your execution path, see the notes near the top of
<code>annotation-file-utilities/tests/Makefile</code>.
</p>
<hr />
<h2 id="using">Using the Annotation File Utilities</h2>
<p>
To use the tools, simply run them from the command-line with the
appropriate arguments. The following instructions are for running the
tools on a Linux/Unix/MacOS machine.
The tools work identically on Windows, except
the extension <code>.bat</code> needs to be appended to the tool name (for
example, Windows users would execute <code>insert-annotations.bat</code>
instead of <code>insert-annotations</code>).
</p>
<p>
For all the tools, arguments starting with a single
‘<code>@</code>’ are recognized as argument files
(<code>argfiles</code>), the contents of which get expanded into the
command line. (Initial <code>@@</code> represents a literal
<code>@</code> in the argument text.) For additional details of argfile
processing, refer to Oracle's
<a href="https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html"><code>javac</code></a>
documentation.
</p>
<h3 id="insert-annotations"> Insert-annotations </h3>
<p>
To insert annotations specified by an annotation file into a class file, use the
insert-annotations tool. Running:
</p>
<pre><code>insert-annotations mypackage.MyClass indexFile.jaif</code></pre>
<p>
will read in all the annotations from the annotation file
<code>indexFile.jaif</code> and insert those annotations pertaining to
<code>mypackage.myClass</code> into the class file for
<code>mypackage.MyClass</code>, outputting the final class file to
<code>mypackage.MyClass.class</code> in the present working directory.
Note that the class file for <code>mypackage.MyClass</code> must be located
on your classpath.
</p>
<p>
Multiple pairs of class and index files (in that order) can be specified
on a single command line; if the program exits normally, the results
are the same as if the program were run once for each pair of arguments
in sequence.
Run:
</p>
<pre><code>insert-annotations --help</code></pre>
<p>
for full usage information.
</p>
<h3 id="extract-annotations"> Extract-annotations </h3>
<p>
To extract annotations from a class file and write them to an annotation file,
use the extract-annotations tool. Running:
</p>
<pre><code>extract-annotations mypackage.MyClass</code></pre>
<p>
will locate the class file for <code>mypackage.MyClass</code>, read all
annotations from it, and write the results in annotation file format to
<code>mypackage.MyClass.jaif</code> in the present working directory. Note
that <code>mypackage.MyClass</code> must be located on your classpath.
</p>
<p>
Multiple classes can be specified on a single command line; if the
program exits normally, the results are the same as if the program were
run once for each class in sequence.
Run:
</p>
<pre><code>extract-annotations --help</code></pre>
<p>
for full usage information.
</p>
<h3 id="insert-annotations-to-source">Insert-annotations-to-source </h3>
<p>
To insert annotations specified by an annotation file into a Java source file,
use the insert-annotations-to-source tool. Running:
</p>
<pre><code>insert-annotations-to-source index1.jaif index2.jaif mypackage/MyClass.java yourpackage/YourClass.java</code></pre>
<p>
will read all the annotations from <code>index1.jaif</code> and
<code>index2.jaif</code>, insert them (when applicable) into their
appropriate locations in <code>mypackage/MyClass.java</code> and
<code>yourpackage/YourClass.java</code>, and write the results to
<code>annotated/mypackage/MyClass.java</code> and
<code>annotated/mypackage/MyClass.java</code>, respectively.
</p>
<p>
Index and source files can be specified in any order, mixing the two
file types freely; if the source files have no overlapping definitions
and the program exits normally, the results are the same as if the
program were run once for each source file, with <em>all</em> JAIFs
given for each run.
Run:
</p>
<pre><code>insert-annotations-to-source --help</code></pre>
<p>
for full usage information.
</p>
<p>
If you wish to insert annotations into method bodies, you must have the
associated class <code>mypackage.MyClass.class</code> on your classpath.
You can insert annotations on class/field/method declarations and
signatures without the class on your classpath.
</p>
<p>
In general (but see below for exceptions), your source code needs to
contain the locations that your
<code>.jaif</code> file references. In particular, if the
<code>.jaif</code> file contains annotations for a type parameter, but the
source code uses a raw type, then you will get an error such as
</p>
<pre>
Found class Edge, but unable to insert @checkers.nullness.quals.Nullable:
@checkers.nullness.quals.Nullable (nl=true) @ [GenericArrayLocationCriterion at ( [TYPE_ARGUMENT(0)] ), ...
</pre>
<p>
In this case, you should add type arguments, such as changing
</p>
<pre> public void pushNonezeroRing(Stack stack, Hashtable seen) {</pre>
<p>to</p>
<pre> public void pushNonezeroRing(Stack<Edge> stack, Hashtable<Edge, ?> seen) {</pre>
<p>
In a few cases, insert-annotations-to-source will generate "default"
code to provide a location for an annotation. These include
</p>
<ul>
<li>method and constructor <a href="http://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.4.1-220">receivers;</a></li>
<li>type parameter bounds (<code>extends Object</code>);</li>
<li>type casts for expressions</li>
<li>nullary constructor definitions; and</li>
<li>explicit constructor invocations for literal arrays (e.g., <code>{"a", "b"}</code> becomes <code>new String[] {"a", "b"}</code>).</li>
</ul>
<p>
The command-line options are as follows:
</p>
<!-- start options doc (DO NOT EDIT BY HAND) -->
<ul>
<li id="optiongroup:General-options">General options
<ul>
<li id="option:outdir"><b>-d</b> <b>--outdir=</b><i>directory</i>. Directory in which output files are written. [default annotated/]</li>
<li id="option:in-place"><b>-i</b> <b>--in-place=</b><i>boolean</i>. If true, overwrite original source files (making a backup first).
Furthermore, if the backup files already exist, they are used instead
of the .java files. This behavior permits a user to tweak the .jaif
file and re-run the annotator.
<p>
Note that if the user runs the annotator with --in-place, makes edits,
and then re-runs the annotator with this --in-place option, those
edits are lost. Similarly, if the user runs the annotator twice in a
row with --in-place, only the last set of annotations will appear in
the codebase at the end.
<p>
To preserve changes when using the --in-place option, first remove the
backup files. Or, use the <code>-d .</code> option, which makes (and
reads) no backup, instead of --in-place. [default false]</li>
<li id="option:abbreviate"><b>-a</b> <b>--abbreviate=</b><i>boolean</i>. Abbreviate annotation names [default true]</li>
<li id="option:comments"><b>-c</b> <b>--comments=</b><i>boolean</i>. Insert annotations in comments [default false]</li>
<li id="option:omit-annotation"><b>-o</b> <b>--omit-annotation=</b><i>string</i>. Omit given annotation</li>
<li id="option:nowarn"><b>--nowarn=</b><i>boolean</i>. Suppress warnings about disallowed insertions [default false]</li>
<li id="option:convert-jaifs"><b>--convert-jaifs=</b><i>boolean</i>. Convert JAIFs to new format [default false]</li>
<li id="option:help"><b>-h</b> <b>--help=</b><i>boolean</i>. Print usage information and exit [default false]</li>
</ul>
</li>
<li id="optiongroup:Debugging-options">Debugging options
<ul>
<li id="option:verbose"><b>-v</b> <b>--verbose=</b><i>boolean</i>. Verbose (print progress information) [default false]</li>
<li id="option:debug"><b>--debug=</b><i>boolean</i>. Debug (print debug information) [default false]</li>
<li id="option:print-error-stack"><b>--print-error-stack=</b><i>boolean</i>. Print error stack [default false]</li>
</ul>
</li>
</ul>
<!-- end options doc -->
<hr />
<h2 id="implementation">Design and Implementation Details</h2>
<p>This section describes some high level-design and implementation
details of the Annotation File Utilities, including the different
components of the Annotation File Utilities and how they fit together.
It is intended for someone who is beginning work on the Annotation File
Utilities or is curious about how the Annotation File Utilities work.</p>
<p>The Annotation File Utilities is composed of three sub-projects:
<code>asmx</code>, <code>scene-lib</code>, and
<code>annotation-file-utilities</code>. The <code>asmx</code>
sub-project provides methods for reading and writing class files. The
<code>scene-lib</code> sub-project represents a <code>.jaif</code> file
and inserts and extracts annotations to/from bytecode. The
<code>annotation-file-utilities</code> sub-project inserts annotations
into source code.</p>
<h3 id="asmx" class="subsection">Asmx</h3>
<p>The <code>asmx</code> sub-project provides methods for reading and
writing class files. <code>asmx</code> is based on an old version
(2.2.2) of the <a href="http://asm.ow2.org/">ASM Framework</a>. It has been modified to
allow it to read and write <a href="https://checkerframework.org/jsr308/specification/java-annotation-design.html">JSR 308</a>
annotations in bytecode. However, it is far behind the current ASM
version (5.0), which provides support for type annotations. We should
discard the custom <code>asmx</code> in the Annotation File Utilities
and adapt the Annotation File Utilities to use the official, supported
version of ASM.</p>
<h3 id="scene-lib" class="subsection">Scene-lib</h3>
<p><code>scene-lib</code> is an interface to a <code>.jaif</code> file.
It reads in and writes out <code>.jaif</code> files and provides an
internal representation of a <code>.jaif</code> file to access and
manipulate.</p>
<p>Internally, a <code>.jaif</code> file is represented by the
<code>scene-lib/src/annotations/el/AScene.java</code> class. The
<code>AScene</code> class (or “annotated scene”) roughly
parallels the root of an abstract syntax tree. An <code>AScene</code>
has a number of classes (<code>AClass</code>) as children. Each class
has a number of methods (<code>AMethod</code>), fields
(<code>AElement</code>), etc. as children. All of these classes are
related in the type hierarchy shown below.</p>
<div>
<img src="figures/scene-lib-type-hierarchy.svg" alt="scene-lib type hierarchy">
</div>
<p>Each class in the type hierarchy has one or more fields to hold
annotations for the different components of the class. For example, the
<code>AMethod</code> class has the following fields: bounds, return
type, receiver parameters, and throws clause. Each of these fields
holds the annotations stored on that part of the method. For details on
the remainder of the classes in the type hierarchy, and their
respective fields, see the documentation for each file in
<code>scene-lib/src/annotations/el/</code>.</p>
<p>An <code>AScene</code> instance can be created in two ways. An empty
<code>AScene</code> can be created by calling the <code>AScene</code>
constructor, or an <code>AScene</code> can be created by parsing an
existing <code>.jaif</code> file. Once an <code>AScene</code> is
created, annotations can be added to it by adding them to the correct
fields of the children. An <code>AScene</code> can also be output to
create a new <code>.jaif</code> file.</p>
<h4 id="bytecode-insertion" class="subsubsection">Bytecode Insertion</h4>
<p>Annotations can be inserted into bytecode by executing the
<code>annotation-file-utilities/scripts/insert-annotations</code>
script. This script takes one or more ⟨class name,
<code>.jaif</code> file⟩ pairs as arguments. The annotations
specified in the <code>.jaif</code> file are inserted into the
classfile directly before the <code>.jaif</code> file in the argument
list.</p><p>First, each <code>.jaif</code> file is parsed into an
<code>AScene</code> (as described in
<a href="#scene-lib">Scene-lib</a>). Then,
<code>asmx/src/org/objectweb/asm/ClassReader.java</code> parses the
classfile. As it is parsing the classfile, it passes the parsed
bytecode off to the
<code>scene-lib/src/annotations/io/classfile/ClassAnnotationSceneWriter.java</code>
class. This class has a reference to the <code>AScene</code> parsed
from the <code>.jaif</code> file. As this class receives the parsed
bytecode it inserts the relevant annotations from the
<code>AScene</code> in the bytecode and then writes the bytecode back
out.</p>
<h4 id="bytecode-extraction" class="subsubsection">Bytecode Extraction</h4>
<p>Annotations can be extracted from bytecode by executing the
<code>annotation-file-utilities/scripts/extract-annotations</code>
script. This script takes one or more class names as arguments and
outputs the annotations found in those classes to <code>.jaif</code>
files.</p><p>First, an empty <code>AScene</code> is constructed to store
the annotations.
<code>asmx/src/org/objectweb/asm/ClassReader.java</code> parses the
classfile and passes the parsed bytecode off to the
<code>scene-lib/src/annotations/io/classfile/ClassAnnotationSceneReader.java</code>
class. This class filters out the annotations in the bytecode and adds
them to the correct part of the <code>AScene</code>. After this, the
<code>AScene</code> is output to a <code>.jaif</code> file.</p>
<h3 id="source-insertion" class="subsection">Annotation-file-utilities</h3>
<p>The <code>annotation-file-utilities</code> sub-project inserts annotations into source
code. It can be run by executing the
<code>annotation-file-utilities/scripts/insert-annotations-to-source</code>
script. The script takes one or more <code>.jaif</code> files, followed by one or more <code>.java</code> source files as arguments. The annotations in the <code>.jaif</code> files are inserted into the <code>.java</code> source files.</p><p>First, an instance of
<code>annotation-file-utilities/src/annotator/specification/IndexFileSpecification.java</code>
is created. Its <code>parse</code> method parses the <code>.jaif</code>
file into an <code>AScene</code> (as described in
<a href="#scene-lib">Scene-lib</a>). The <code>parse</code> method
calls the <code>parseScene</code> method, which traverses through the
<code>AScene</code> and creates an
<code>annotation-file-utilities/src/annotator/specification/CriterionList.java.</code>
A <code>CriterionList</code> identifies a unique AST node that is the
location of an insertion. It contains objects that implement the
<code>annotation-file-utilities/src/annotator/find/Criterion.java</code>
interface.
Each <code>Criterion</code> has an <code>isSatisifiedBy</code> method —
a predicate that takes an AST node and returns <code>true</code> if the
AST node satisfies the <code>Criterion</code> and <code>false</code>
otherwise. To determine if a given node matches a
<code>CriterionList</code>, the node is passed to all of the
<code>Criterion</code>s in the <code>CriteriaList</code>. If every
<code>Criterion</code> returns <code>true</code> then it is match. If
one or more <code>Criterion</code>s return <code>false</code> then it is
not a match. The various <code>Criterion</code> classes are in the
<code>annotation-file-utilities/src/annotator/find/</code> directory.
For example, take the following source code:</p>
<pre class="verbatim">package afu.example;
public class Test {
public void m(boolean b, int i) {
// ...
}
}
</pre>
<p>The <code>CriterionList</code> to specify the location of the
<code>i</code> parameter contains the following
<code>Criterion</code>s:</p>
<ul class="itemize"><li class="li-itemize">
<code>InPackageCriterion("afu.example")</code>
</li><li class="li-itemize"><code>InClassCriterion("Test")</code>
</li><li class="li-itemize"><code>InMethodCriterion("m(ZI)V")</code>
</li><li class="li-itemize"><code>ParamCriterion(1)</code>
</li></ul>
<p>After this <code>CriterionList</code> is built up an
<code>annotation-file-utilities/src/annotator/find/Insertion.java</code>
is created. An <code>Insertion</code> stores an
<code>annotation-file-utilities/src/annotator/find/Criteria.java</code>
(which is created from a <code>CriterionList</code>) and the text to be
inserted. All of these <code>Insertion</code>s are then added to a
list. The Java compiler then is called to parse the Java source into an
abstract syntax tree. This is followed by a call to the
<code>getPositions</code> method of
<code>annotation-file-utilities/src/annotator/find/TreeFinder.java</code>,
which scans through each node of the abstract syntax trees. For each node,
it runs through the <code>Criteria</code> for each un-matched
<code>Insertion</code>. If at least one of the <code>Criteria</code>
does not match, then this is not the correct place for the
<code>Insertion</code> and the <code>Insertion</code> will be checked
at the remaining nodes of the tree. If all of the <code>Criteria</code>
match, then this node is the correct place for the
<code>Insertion</code>. It is removed from the list of un-matched
<code>Insertion</code>s and the position where to insert the
<code>Insertion</code> is determined. This position is the integer
index in the file where the <code>Insertion</code> should be inserted.
After the positions are found for all of the <code>Insertion</code>s,
the <code>Insertion</code> text is inserted into the file. This happens
backwards, with <code>Insertion</code>s at the end of the file (i.e.
with higher positions) being inserted first. If <code>Insertion</code>s
were instead inserted from the beginning of the file then a single
<code>Insertion</code> would invalidate all of the positions for the
following <code>Insertion</code>s.</p>
<p>If there are remaining <code>Insertion</code>s that were not matched
to a node in the abstract syntax tree then an error message is
displayed.</p>
<hr />
<h2 id="feedback"> Feedback and bug reports </h2>
<p>
To submit a bug report or request a new feature, use the
<a href="https://github.com/typetools/annotation-tools/issues">issue
tracker</a>. When reporting a bug, please include exact instructions in
how to reproduce it, and please also attach relevant input files. This
will let us resolve the issue quickly.
</p>
<p>
You can also reach the developers at
<a href="mailto:annotation-tools-dev@googlegroups.com">annotation-tools-dev@googlegroups.com</a>.
But please use the
<a href="https://github.com/typetools/annotation-tools/issues">issue
tracker</a> for bug reports and feature requests.
</p>
<h3 id="changelog">Changelog</h3>
<p>
The <a href="changelog.html">changelog</a> describes what is new in each release.
</p>
<hr/>
</body>
</html>
<!-- LocalWords: utils bashrc tcsh tcshrc cshrc classpath
-->