CorefEval
The CorefEval utility computes a coreference
score using a version of Marc Vilain's MUC-6 coreference scoring
algorithm. It is invoked by
jet -CorefEval system-response-collection
key-collection
where system-response-collection is
the system response and key-collection
is the coreference key; both are collections of
XML-annotated files. Each collection file should contain a list
of the file names of the XML-annotated document
files, one file name per line (the two files should contain the same
number of lines). File names may either be absolute
paths or relative paths; relative paths are interpreted relative
to the directory containing the collection.
The individual document files are marked
for coreference with the tags <mention entity=n> ... </mention>, where n is an integer.
Sequences enclosed in tags with the same value of n are coreferential.
When the system response and key are compared, mentions match if the last character of the mention in
the response and key are the same. Mentions in response or key
which do not have a corresponding mention in the other document are
ignored. The scorer reports a recall and precision score for each
document and for the entire collection.