-
Registration: Contestants should email Charles Ortiz
charles.ortiz@nuance.com stating their intent to enter the contest
no later than July 1, 2016.
The contest itself will be held at
IJCAI 2016 in New York City, July 12, 2016.
-
Input format: Contestant programs will receive their input
in the form of an .xml file. An example file may be found at
http://www.cs.nyu.edu/faculty/davise/papers/WSCExample.xml.
The structure of the .xml should be self-explanatory on inspection
of this file.
-
Problems: All problems have the following form. There is a text of a
single sentence or a few sentences that contains one or more pronouns
with ambiguous referents. Each problem asks about one such pronoun. The
pronoun that is the subject of the problem is demarcated in the XML
input with the tag < pron > < /pron >. (In viewing the XML file in a web
browser, the pronoun appears in boldface.) After the text, there is a short
excerpt from the text containing the pronoun that is the subject of the problem
and a few words that occur on one side or the other; this is for the benefit
human viewers. Finally, a list of possible referents is given, labelled
"A", "B", "C" ... For example:
Babar wonders how he can get new clothing. Luckily, a very rich old man
who has always been fond of little elephants understands right away
that he is longing for a fine suit.
As he likes to make people happy, he gives him his wallet.
he is longing for a fine suit
- Babar
- old man
There may be multiple problems that use the same text but ask about different
pronouns, as with problems 2-5 in the example file.
-
Number of rounds and questions:
There will be one or two rounds in the contest. Each round will
consist of at least 60 questions.
The two rounds differ in the source of the texts.
In the first round, the texts
are "Pronoun Disambiguation Problems"; that is, they are drawn from
actual texts, possibly with some editing. In the second round, each text is
one half of a Winograd schema. A detailed discussion and justification
is given in (Morgenstern, Davis, and Ortiz, 2016).
Only contestants who achieve at least 90% in the first round
will be allowed to compete in the
second round. If no contestants qualify, then the second round will not be
given.
-
Competing:
Each contestant must be represented by an individual who is present. The
representative must bring a laptop on which the entry will run. Any
commercially sold portable computer is acceptable.
-
Internet access:
Limited access to the Internet will be permitted, under certain
circumstances. Please contact the organizers for details.
-
Output format:
The output of your program should be a plain text file named
TeamName-output.txt. The format to be followed is illustrated in the file
http://www.cs.nyu.edu/faculty/davise/papers/WSOutput.txt.
For each problem, there will be four lines in the output, separated by
line breaks:
Line 1: Problem number, and echo of text of problem.
Line 2: Echo of the excerpt for the problem.
Line 3: "Answer" problemNumber.answerNumber answer
Line 4: Blank, as a separator between problems
For example, if the problem above is problem 2 in the input, then
the corresponding four lines of the output file
would be as follows:
Babar wonders how he can get new clothing. Luckily, a very rich old man who has always been fond of little elephants understands right away that he is longing for a fine suit. As he likes to make people happy, he gives him his wallet.
he is longing for a fine suit
Answer 2.A Babar
At the end of the file, there should be a comma-separated list of all the
answers in order. E.g.
A, A, B, B, A, A, B
The submission will be graded on the final list of answers. The remaining
material is there for human inspection.
No problems should be omitted. Any problem that is omitted will be marked
as wrong, so it always pays to guess.
-
Publication:
In the three weeks following the competition, researchers with winning
or potentially winning entries will be expected to submit to WSC
organizers a paper explaining the algorithms, knowledge sources,
and knowledge structures used. These papers will be posted on the
commonsensereasoning.org website. Publication on the commonsensereasoning.org
website does not preclude any other publication. Entries not submitting
such a paper will be disqualified.
If, in the judgement of the contest committee, the description of the program
in this paper is entirely inadequate or implausible as an explanation of
the success of the program, then the team involved will be asked to demonstrate
in detail that the behavior of the program is in fact that described in the
paper.
The aim of this contest is to advance science; all results obtained
must be reproducible, and communicable to the public. As such, any
winning entry is encouraged to furnish to the organizers of the
Winograd Schema Challenge Competition its source code and executable
code, and to use open source databases or knowledge bases or make its
databases and knowledge structures available for independent
verification of results.
-
Prizes:
The grand prize of $25,000 will be awarded to the first team to achieve
a score of 90% in both rounds of the contest. If more than one team
accomplishes this, the prize will be awarded to the team with the higher
score in the second round. If tied in the second round, the prize will go
to the team with the higher score in the first round. If both rounds are
ties, the prize will be split.
At IJCAI-2016 three smaller prizes, of $1000, $750, and $500
will be awarded to the top three programs that score over 65% on the
first round of the contest.
-
Contest commitee: The contest has been designed and will
be administered by a committee consisting of: Leora Morgenstern
leora.morgenstern@leidos.com; Ernest Davis davise@cs.nyu.edu; and
Charles Ortiz charles.ortiz@nuance.com. The committee's decisions on
all matters relating to the contest is final. Any questions should be addressed
to them. We gratefully acknowledge the assistance of Hector Levesque and
of Gary Marcus.