CorefEval

The CorefEval utility computes a coreference score using a version of Marc Vilain's MUC-6 coreference scoring algorithm.  It is invoked by

jet -CorefEval  system-response-collection key-collection

where system-response-collection is the system response and key-collection is the coreference key;  both are collections of XML-annotated files.  Each collection file should contain a list of the file names of the XML-annotated document files, one file name per line (the two files should contain the same number of lines).  File names may either be absolute paths or relative paths;  relative paths are interpreted relative to the directory containing the collection.

The individual document files are marked for coreference with the tags <mention entity=n> ... </mention>, where n is an integer.
Sequences enclosed in tags with the same value of n are coreferential.

When the system response and key are compared, mentions match if the last character of the mention in the response and key are the same.  Mentions in response or key which do not have a corresponding mention in the other document are ignored.  The scorer reports a recall and precision score for each document and for the entire collection.