View the Project on GitHub TiarkRompf/cs352

Project 1: Scanner

Due 11:59pm Sunday Sept 13

Useful Links


Implement a scanner for the Mojo language, which takes input from the standard input stream (System.in), writes error messages to the standard error stream (System.err), and writes output to the standard output stream (System.out). The scanner should print each recognized token and its corresponding lexeme on the standard output one line at a time. Input files can be any sequence of Mojo tokens. Use the following template for your JavaCC specification file Parser.jj:

public class Parser {

  int comment, pragma;
  public static void main(String[] args) {
    SimpleCharStream stream = new SimpleCharStream(System.in);
    ParserTokenManager scanner = new ParserTokenManager(stream);
    while (true) {
      try {
        Token token = scanner.getNextToken();
        for (Token t = token.specialToken; t != null; t = t.specialToken)
          System.out.println(tokenImage[t.kind] + " " + t);
        if (token.kind == EOF) break;
        System.out.println(tokenImage[token.kind] + " " + token);
      } catch (TokenMgrError e) {

Please note that it is normal to get an error if you attempt to run this file as-is without first defining at least one token. JavaCC is not a compiler; it generates code similar to other tools like Lex and Yacc. JavaCC will create several .java files which you must compile using javac (it is a good idea to create a Makefile for this!). To run your scanner simply invoke the class containing main(), ParserTokenManager, to read from standard input (System.in):

java ParserTokenManager



An optional Makefile has been provided for you to get started on your project. As necessary, you may modify this and carry it forward for future projects (as you build parts of your compiler). You may access it by doing the following from inside the folder you are using for your project:

    cp /homes/cs352/Fall15/project1/Makefile ./

JavaCC Example Code

JavaCC example code is available on the JavaCC website and also in the JavaCC install. It is encouraged that you look at some of the example code to familiarize yourself with the features of JavaCC, starting with the simple examples provided. To copy the example code into your directory, do the following: cp /homes/cs352/javacc/

Mojo Example Code & Token Output

Download the example code and token output for this project into your project folder by doing the following:

    cp /homes/cs352/Fall15/project1/tests-project1.tar.gz ./
    tar -xvf tests-project1.tar.gz

In the folder called "output", you can see the output of a canonical implemntation of the tokenizer. The test examples are valid Mojo programs.


You should turn in just your JavaCC file (Parser.jj), which implements your scanner, using the turnin command available on CS Unix machines.

To turn in your project, make sure that you are in the directory that contains your Parser.jj file. Then type the following:

turnin -c cs352 -p project1 Parser.jj

To verify your turned-in work, do:

turnin -c cs352 -p project1 -v


Grading will be done using automated scripts to compare your output with a model solution. You should ensure that the output file consists solely of tokens and lexeme strings, one per line of output, as they are scanned from the input. White space and comments should be silently discarded.

Some of (but not all) of the test input that we will use to grade your scanners will be released to you later in the week of Aug 31. Please note that we will not release all of our test material. It is your responsibility to follow the documentation.