The city councilmen refused the demonstrators a permit because
they [feared/advocated] violence.
Who [feared/advocated] violence?
Answers: The city councilmen/the demonstrators.
Comment: This was the version published in (Winograd 1972).
Winograd's earlier Ph.D. thesis (1970) has it in a slightly different
form:
"The city councilmen refused to give the women a permit for a demonstration
because they [feared/advocated] violence".
Winograd observes that finding the correct reference resolution would matter
if one were translating the sentence into French, because of the gender of
"they".
The trophy doesn't fit into the brown suitcase because it's
too [small/large].
What is too [small/large]?
Answers:The suitcase/the trophy.
Joan made sure to thank Susan for all the help she had [given/received].
Who had [given/received] help?
Answers: Susan/Joan.
Paul tried to call George on the phone, but he wasn't
[successful/available].
Who was not [successful/available]?
Answers: Paul/George
The lawyer asked the witness a question, but he was reluctant to
[answer/repeat] it .
Who was reluctant to [answer/repeat] the question?
Answers: The witness/the lawyer.
The delivery truck zoomed by the school bus because it was going so
[fast/slow].
What was going so [fast/slow]?
Answers: The truck/the bus
Frank felt [vindicated/crushed] when his longtime rival Bill revealed that
he was the winner of the competition.
Who was the winner of the competition?
Answers: Frank/Bill
The man couldn't lift his son because he was so [weak/heavy].
Who was [weak/heavy]?
Answers: The man/the son.
The large ball crashed right through the table because it was made of
[steel/styrofoam].
What was made of [steel/styrofoam]?
Answers: The ball/the table.
John couldn't see the stage with Billy in front of him because he is so
[short/tall]. Who is so [short/tall]?
Answers: John/Billy.
Tom threw his schoolbag down to Ray after he reached the [top/bottom] of the
stairs.
Who reached the [top/bottom] of the stairs?
Answers: Tom/Ray.
Although they ran at about the same speed, Sue beat Sally because she had such
a [good/bad] start.
Who had a [good/bad] start?
Answers: Sue/Sally.
The sculpture rolled off the shelf because it wasn't [anchored/level].
What wasn't [anchored/level]?
Answers: The sculpture/the shelf.
Sam's drawing was hung just above Tina's and it did look much better with
another one [below/above] it.
Which looked better?
Answers: Sam's drawing/Tina's drawing.
Anna did a lot [better/worse]
than her good friend Lucy on the test because she had
studied so hard.
Who studied hard?
Answers: Anna/Lucy
The firemen arrived [after/before] the police because they were coming from so
far away.
Who came from far away?
Answers: The firemen/the police.
Frank was upset with Tom because the toaster he had [bought from/sold to]
him didn't work.
Who had [bought/sold] the toaster?
Answers: Frank/Tom.
Jim [yelled at/comforted] Kevin because he was so upset.
Who was upset?
Answers: Jim/Kevin.
The sack of potatoes had been placed [above/below]
the bag of flour, so it had to be
moved first.
What had to be moved first?
Answers: The sack of potatoes/the bag of flour.
Pete envies Martin [because/although] he is very successful.
Who is very successful?
Answers: Martin/Pete.
I was trying to balance the bottle upside down on the table, but I
couldn't do it because it was so [top-heavy/uneven].
What was [top-heavy/uneven]?
Answers: the bottle/the table.
I spread the cloth on the table in order to [protect/display] it.
To [protect/display] what?
Answers: the table/the cloth.
The older students were bullying the younger ones, so we [rescued/punished]
them.
Whom did we [rescue/punish]?
Answers: The younger students/the older students.
I poured water from the bottle into the cup until it was [full/empty].
What was [full/empty]?
Answers: The cup/the bottle.
Susan knows all about Ann's personal problems because she is
[nosy/indiscreet].
Who is [nosy/indiscreet]?
Answers: Susan/Anne.
Sid explained his theory to Mark but he couldn't [convince/understand] him.
Who did not [convince/understand] whom?
Answer Pair A:
Sid did not convince Mark/Mark did not convince Sid.
Answer Pair B:
Sid did not understand Mark/Mark did not understand Sid.
Comment: Winograd schema in the broad sense, since the
question asks about two pronouns.
Susan knew that Ann's son had been in a car accident, [so/because] she told her
about it.
Who told the other about the accident?
Answers: Susan/Ann.
Comment: There is a large literature on "implicit
causality" and "implicit consequentiality" biases (see, for example,
(Hartshorne, 2013), (Kehler et al. 2008)), in which a comprehender obtains
a preference for the antecedent of a pronoun in a second clause based on the
content of the first clause and the connective, before seeing the
content of the second clause. These are often associated with "so" and
"because". For example seeing:
"Sally frightened Mary because she ..." people interpret "she" as Sally.
"Sally loves Mary because she ..." people interpret "she" as Mary.
"Sally frightened Mary, so she ..." people interpret "she" as Mary.
"Sally loves Mary, so she ..." people interpret "she" as Sally.
If there are biases that favor the correct disambiguation in our example here,
and if the direction of the biases can be determined purely from linguistic
characteristics of the first clause e.g. from the use of the verb "knew",
then this is not a valid Winograd schema, because an automated understander
could do better than chance on this without employing world knowledge. However,
I currently have no reason to think that this is the case.
Joe's uncle can still beat him at tennis, even though he is 30 years
[older/younger].
Who is [older/younger]?
Answers:Joe's uncle/Joe.
Comment: Note the following results of Google search (1/9/11).
"uncle older" 14,500,000
"uncle younger" 9,630,000
"older" 357,000,000
"younger" 93,000,000
Thus "uncle" is not particularly associated with "older".
The police left the house and went into the garage, [where/after] they found
the murder weapon.
Where did they find the murder weapon?
Answers: In the garage/in the house.
Comment: "after" is perhaps slightly awkward here.
Winograd schema in the broad sense, since there is no ambiguous pronoun.
The painting in Mark's living room shows an oak tree.
It is to the right of [the bookcase/a house].
What is to the right of [the bookcase/a house]?
Answers: The painting/the tree.
There is a gap in the wall. You can see the garden [through/behind] it.
You can see the garden [through/behind] what?
Answers: The gap/the wall.
Comment. The syntax of the question is strictly incorrect, but
``[Behind/through] what can you see the garden?'' is
almost incomprehensible.
The drain is clogged with hair. It has to be [cleaned/removed].
What has to be [cleaned/removed]?
Answers: The drain/the hair.
My meeting started at 4:00 and I needed to catch the train at 4:30,
so there wasn't much time. Luckily, it was [short/delayed], so it worked out.
What was [short/delayed]?
Answers: The meeting/the train.
There is a pillar between me and the stage, and I can't [see/see around]
it.
What can't I [see/see around]?
Answers: The stage/the pillar.
They broadcast an announcement, but a subway came into the
station and I couldn't [hear/hear over] it.
What couldn't I [hear/hear over]?
Answers: The announcement/the subway.
In the middle of the outdoor concert, the rain started falling, [and/but] it
continued until 10.
What continued until 10?
Answers: The rain/the concert.
I used an old rag to clean the knife, and then I put it in the [drawer/trash].
What did I put in the [drawer/trash]?
Answers: The knife/the rag.
Ann asked Mary what time the library closes, [but/because] she had forgotten.
Who had forgotten?
Answers:Mary/Ann.
I took the water bottle out of the backpack so that it would be [lighter/handy].
What would be [lighter/handy]?
Answers: The backpack/the bottle.
I couldn't put the pot on the shelf because it was too [high/tall].
What was too [high/tall]?
Answers: The shelf/the pot.
Comment:
The Google query ``high pot" gives about 10 times as many pages
as ``tall pot" (search 9/8/11)
for reasons unconnected with the meanings in this sentence.
I'm sure that my map will show this building; it is very [famous/good].
What is [famous/good]?
Answers: The building/the map.
Comment: ``Detailed'' would of course be a better word to describe the
map, but it would certainly be Googlable, and probably solvable by
selectional restriction. One might suppose that "famous building" would be
a more frequent combination than "famous map" but Google search (5/11/2012)
suggests the reverse:
"building": 2.2 billion
"famous building": 250 million
"map": 1.4 billion
"famous map": 500 million
Bob paid for Charlie's college education. He is very [generous/grateful].
Who is [generous/grateful]?
Answers: Bob/Charlie.
Bob paid for Charlie's college education, but now Charlie acts as though
it never happened. He is very [hurt/ungrateful].
Who is [hurt/ungrateful]?
Answers: Bob/Charlie
Bob was playing cards with Adam and was way ahead. If Adam hadn't had a
sudden run of good luck, he would have [won/lost].
Who would have [won/lost]?
Answers: Bob/Adam.
Adam can't leave work here until Bob arrives to replace him. If Bob had
left home for work on time, he would be [here/gone] by this time.
Who would be [here/gone]?
Answers: Bob/Adam
If the con artist has succeeded in fooling Sam, he would have [gotten/lost] a
lot of money.
Who would have [gotten/lost] the money?
Answers: The con artist/Sam.
It was a summer afternoon, and the dog was sitting in the middle of the
lawn. After a while, it got up and moved to a spot under the tree,
because it was
[hot/cooler].
What was [hot/cooler]?
Answers: The dog/The spot under the tree.
Comment:
An earlier version of this page omitted "to a spot" in the second sentence,
but that was problematic. Thanks to Peter Schueller for discussion.
The cat was lying by the mouse hole waiting for the mouse, but it was too
[cautious/impatient].
What was too [cautious/impatient]?
Answers: The mouse/the cat.
Anne gave birth to a daughter last month. She is a very charming
[woman/baby].
Who is a very charming [woman/baby]?
Answers: Anne/Anne's daughter.
Alice tried frantically to stop her daughter from [chatting/barking] at
the party, leaving us to wonder why she was behaving so strangely.
Who was behaving strangely?
Answers: Alice/Alice's daughter.
I saw Jim yelling at some guy in a military uniform with a huge red
beard. I don't know [who/why] he was, but he looked very unhappy.
Who looked very unhappy?
Answers: The guy in the uniform/Jim.
Comment:
If you stop the second sentence at "I don't know [who/why] he was" you
still have the same ambiguity, but it becomes very difficult to ask the
question. "I don't know who who was?'' is perhaps best.
Of course, either way there is a reading in
which the two ``he'' refer to different people, but it seems to me that that's
considerably less preferred.
The fish ate the worm. It was [tasty/hungry].
What was [tasty/hungry]?
Answers: The worm/the fish.
I was trying to open the lock with the key, but someone had filled the
keyhole with chewing gum, and I couldn't get it [in/out].
What couldn't I get [in/out]?
Answers: The key/the chewing gum.
The dog chased the cat, which ran up a tree. It waited at the [top/bottom].
Which waited at the [top/bottom]?
Answers: The cat/the dog.
In the storm, the tree fell down and crashed through the roof of my
house. Now, I have to get it [removed/repaired].
What has to be [removed/repaired]?
Answers: The tree/the roof.
The customer walked into the bank and stabbed one of the tellers. He was
immediately taken to the [emergency room/police station].
Who was taken to the [emergency room/police station]?
Answers: The teller/the customer.
John was doing research in the library when he heard a man humming and
whistling. He was very [annoyed/annoying].
Who was [annoyed/annoying]?
Answers:John/the hummer.
John was jogging through the park when he saw a man juggling watermelons.
He was very [impressed/impressive].
Who was [impressed/impressive]?
Answers: John/the juggler.
Bob collapsed on the sidewalk. Soon he saw Carl coming to help. He was
very [ill/concerned].
Who was [ill/concerned]?
Answers: Bob/Carl.
Sam and Amy are passionately in love, but Amy's parents are unhappy
about it, because they are [snobs/fifteen].
Who are [snobs/fifteen]?
Answers: Amy's parents/Sam and Amy.
Mark told Pete many lies about himself, which Pete included in his book.
He should have been more [truthful/skeptical].
Who should have been more [truthful/skeptical]?
Answers: Mark/Pete.
Joe has sold his house and bought a new one a few miles away. He will
be moving [out of/into] it on Thursday.
Which house will he be moving [out of/into]?
Answers: The old house/the new house.
Many people start to read Paul's books and can't put them down. They
are [gripped/popular] because Paul writes so well.
Who or what are [gripped/popular]?
Answers: The readers/the books.
Comment:
A lot of synonyms for ``gripped" (e.g. ``fascinated") can be resolved
by selectional restrictions. ``Gripped" cannot because it is polysemic with
physical grasping.
Mary took out her flute and played one of her favorite pieces. She
has [loved/had] it since she was a child.
What has Mary [loved/had] since she was a child?
Answers: The piece/the flute.
Sam pulled up a chair to the piano, but it was broken, so he
had to [stand/sing] instead.
What was broken?
Answers: The chair/the piano.
Since it was raining, I carried the newspaper [over/in] my backpack to
keep it dry. What was I trying to keep dry?
Answers: The backpack/the newspaper.
Comment:
The form of the question is quite far from the text, but there
doesn't seem to be a better way to formulate this question.
Sara borrowed the book from the library because she needs it for an
article she is working on. She [reads/writes] it when she gets home from work.
What does Sara [read/write] when she gets home from work?
Answers: The book/the article.
This morning, Joey built a sand castle on the beach, and put a toy
flag in the highest tower, but this afternoon [a breeze/the tide]
knocked it down.
What did the [breeze/tide] knock down?
Answers: The flag/the sand castle.
Comment: Perhaps a little overly delicate.
Jane knocked on Susan's door, but there was no answer. She was
[out/disappointed].
Who was [out/disappointed]?
Answers: Susan/Jane.
Comment: Note that Jane is also "out" (of the house or room);
The disambiguation in this case involves issues of textual coherence. A
simplified version is given in #114.
Jane knocked on the door, and Susan answered it. She invited her to come
[out/in].
Who invited whom?
Answers: Jane invited Susan/Susan invited Jane.
Sam took French classes from Adam, because he was [eager/known] to
speak it fluently.
Who was [eager/known] to speak French fluently?
Answers: Sam/Adam.
The path to the lake was blocked, so we couldn't [reach/use] it.
What couldn't we [reach/use]?
Answers: The lake/the path.
The sun was covered by a thick cloud all morning, but luckily, by the time
the picnic started, it was [gone/out].
What was [gone/out]?
Answers: The cloud/the sun.
We went to the lake, because a shark had been seen at the ocean beach, so
it was a [dangerous/safer] place to swim.
Which was a [dangerous/safer] place to swim?
Answers: The beach/the lake.
Sam tried to paint a picture of shepherds with sheep, but they ended up
looking more like [dogs/golfers].
What looked like [dogs/golfers]?
Answers:the sheep/the shepherds.
Comment: It may be objected that this can be solved
using the distance in the semantic hierarchy rather than reasoning
about the actual visual appearance. Example 137
below avoids that objection, but is perhaps more difficult for the
human reader. In any case, a program that reasons that similar looking
objects are apt to be close in the semantic hierarchy is, I would say,
solving the problem in a perfectly reasonable way. A program that reasons
ignores the "looks like" relation and just reasons, "The referent of
an anaphora is likely to be close in a semantic hierarchy to some nearby
word in the sentence" is not solving the problem in a reasonable way,
and is employing a fragile heuristic that is easily broken by
rearranging the sentence. For example, if we reword the
above "Sam tried to paint a picture of shepherds with sheep,
but they ended up looking to Wilma
more like [dogs/golfers]" where this heuristic will in all cases look
for a referent semantically close to "Wilma".
Mary tucked her daughter Anne into bed, so that she could [sleep/work].
Who is going to [sleep/work]?
Answers: Anne/Mary.
Fred and Alice had very warm down coats, but they were not [enough/prepared]
for the cold in Alaska.
Who or what were not [enough/prepared] for the cold?
Answers: The coats/Fred and Alice.
Comment: 6 years after adding this, I am concerned that
it may be solvable with selectional restrictions; "enough" is perhaps a little
anomalous as applied to people and "prepared" a little anomalous as
applied to coats. However, neither is impossible, and my feeling that these
restrictions, if real at all, would be hard either to find statistically
or to characterize in terms of features, so I have left the example.
Thomson visited Cooper's grave in 1765. At that date he had been
[dead/travelling] for five years.
Who had been [dead/travelling] for five years?
Answers: Cooper/Thomson
Jackson was greatly influenced by Arnold, though he lived two centuries
[earlier/later].
Who lived [earlier/later]?
Answers: Arnold/Jackson.
Tom's daughter Eva is engaged to Dr. Stewart, who is his partner.
The two [doctors/lovers] have known one another for ten years.
Which two people have known one another for ten years?
Answers: Tom and Dr. Stewart / Eva and Dr. Stewart.
Comment: Winograd schema in the broad sense, since
there is no ambiguous pronoun.
I can't cut that tree down with that axe; it is too [thick/small].
What is too [thick/small]?
Answers: The tree/the axe.
The foxes are getting in at night and attacking the chickens. I shall have
to [guard/kill] them.
What do I have to [guard/kill]?
Answers: The chickens/the foxes.
The foxes are getting in at night and attacking the chickens. They have
gotten very [bold/nervous].
What has gotten [bold/nervous]?
Answers: The foxes/the chickens.
Fred covered his eyes with his hands, because the wind was blowing sand
around. He [opened/lowered] them when the wind stopped.
What did Fred [open/lower]?
Answers: His eyes/his hands.
The actress used to be named Terpsichore, but she changed it to Tina a
few years ago, because she figured it was [easier/too hard] to pronounce.
Which name was [easier/too hard] to pronounce?
Answers: Tina/Terpsichore.
Comment:
This question can be answered without seeing the text, just seeing
the question and the choice of possible answers. I don't see that this is
a defect, but it is certainly an anomaly.
Fred watched TV while George went out to buy groceries. After an hour he
got [up/back].
Who got [up/back]?
Answers: Fred/George.
Fred was supposed to run the dishwasher, but he put it off, because
he wanted to watch TV. But the show turned out to be boring, so he
changed his mind and turned it [on/off].
What did Fred turn [on/off]?
Answers: The dishwasher/the television.
Fred is the only man still alive who remembers my great-grandfather. He
[is/was] a remarkable man.
Who [is/was] a remarkable man?
Answers: Fred/my great-grandfather.
Fred is the only man alive who still remembers my father as an infant.
When Fred first saw my father, he was twelve [years/months] old.
Who was twelve [years/months] old?
Answers: Fred/my father.
In July, Kamtchatka declared war on Yakutsk. Since Yakutsk's army was much
better equipped and ten times larger, they were [victorious/defeated]
within weeks.
Who was [victorious/defeated]
Answers: Yakutsk/Kamchatka.
Comment: Winograd schema in the broad sense, since
with "defeated" there is no plural to serve as referent of "they".
Elizabeth moved her company from Sparta to Troy to save money on
taxes; the taxes are much [higher/lower] there.
Where are the taxes [higher/lower]?
Answers: In Sparta/In Troy
Comment: Winograd schema in the broad sense, since the
ambiguous word is not a pronon.
Esther figures that she will save shipping costs if she builds her
factory in Springfield instead of Franklin, because [most/none]
of her customers live there.
In which town do [most/none] of Esther's customers live?
Answers: Springfield/Franklin.
Comment: Winograd schema in the broad sense, since the
ambiguous word is not a pronon.
Look! There is a [shark/minnow] swimming right below that duck! It had better
get away to safety fast!
What needs to get away to safety?
Answer Pair A: The shark/The duck.
Answer Pair B: The minnow/the duck.
Comment: The pair of possible answers depends on the choice
of words, since the special and alternate words are possible referents.
This also occurs in schema #142.
There are too many deer in the park, so the park service brought in a
small pack of wolves. The population should [increase/decrease] over the
next few years.
Which population will [increase/decrease]?
Answers: The wolves/the deer.
Comment:Winograd schema in the broad sense, since there is no ambiguous
pronoun.
Archaeologists have concluded that humans lived in Laputa 20,000 years
ago. They hunted for [deer/evidence] on the river banks.
Who hunted for [deer/evidence]?
Answers: The prehistoric humans/the archaeologists.
The scientists are studying three species of fish that have recently
been found living in the Indian Ocean. They [appeared/began] two years ago.
Who or what [appeared/began] two years ago?
Answers: The fish/the scientists.
The journalists interviewed the stars of the new movie. They were very
[cooperative/persistent], so the interview lasted for a long time.
Who was [cooperative/persistent]?
Answers: The stars/the journalists
The police arrested all of the gang members. They were trying to [run/stop]
the drug trade in the neighborhood.
Who was trying to [run/stop] the drug trade?
Answers: The gang/the police.
Comment: Hopefully the reader is not too cynical.
I put the cake away in the refrigerator. It has a lot of [butter/leftovers]
in it.
What has a lot of [butter/leftovers]?
Answers: The cake/the refrigerator.
Sam broke both his ankles and he's walking with crutches. But a month
or so from now they should be [better/unnecessary].
What should be [better/unnecessary]?
Answers: The ankles/the crutches.
When the sponsors of the bill got to the town hall, they were surprised
to find that the room was full of opponents. They were very much in the
[majority/minority].
Who were in the [majority/minority]?
Answers: The opponents /the sponsors.
Everyone really loved the oatmeal cookies; only a few people liked the
chocolate chip cookies. Next time, we should make [more/fewer] of them.
Which cookie should we make [more/fewer] of, next time?
Answers: The oatmeal cookies/the chocolate chip.
We had hoped to place copies of our newsletter on all the chairs in the
auditorium, but there were simply [not enough / too many] of them.
There are [too many/not enough] of what?
Answers: chairs/copies of the newsletter.
I stuck a pin through a carrot. When I pulled the pin out, it [left/had]
a hole.
What [left/had] a hole?
Answers: The pin/the carrot.
Note: You might think this is Googlable, but in fact, ``pin left a hole"
finds two results and ``pin leaves a hole" finds 66, whereas ``pin has a hole"
has 11,800 (because of equipment pins, not sewing pins). The phrase
"carrot has a hole" has 2 results and "carrot left a
hole" has none. (8/17/10)
I couldn't find a spoon, so I tried using a pen to stir my
coffee. But that turned out to be a bad idea, because it got full of
[ink/coffee].
What got full of [ink/coffee]?
Answers:The coffee/the pen.
Comment: The statistical associations give the backward answer here:
``ink'' is associated with ``pen'' and ``coffee'' is associated with
``coffee''. Of course, a contestant could use a backward rule here: Since
the challenge designers have excluded examples where statistics give
the right answer, if you find a statistical relation, guess that
the answer runs opposite to it. But that seems very risky.
Steve follows Fred's example in everything. He [admires/influences] him hugely.
Who [admires/influences] whom?
Answers: Steve admires Fred/Fred influences Steve.
The table won't fit through the doorway because it is too [wide/narrow].
What is too [wide/narrow]?
Answers: The table/the doorway.
Grace was happy to trade me her sweater for my jacket. She thinks it looks
[great/dowdy] on her.
What looks [great/dowdy] on Grace?
Answers: The jacket/the sweater.
Bill thinks that calling attention to himself was rude [to/of] Bert.
Who called attention to himself?
Answers: Bill/Bert.
Comment: Winograd schema in the broad sense. The essential
issue is the subject of "calling" which is not a pronoun.
John [hired/hired himself out to] Bill to take care of him.
Who is taking care of whom?
Answers: Bill is taking care of John/John is
taking care of Bill.
Comment: Winograd schema in the broad sense. The essential
issue is the subject of "taking care" which is not a pronoun.
John [promised/ordered] Bill to leave, so an hour later he left.
Who left?
Answers: John/Bill.
Sam Goodman's biography of the Spartan general Xenophanes
conveys a vivid sense of the difficulties he faced in his
[childhood/research].
Who faced difficulties?
Answers: Xenophanes/Sam.
Comment:
It is quite possible that "biography" is correlated with "research".
But even if that correlation is detected, there is another non-trivial
step to realize that the research is associated with the author rather
than the subject of the biography.
Emma's mother had died long ago, and her [place/education] had been
[taken/managed] by an excellent woman as governess.
Whose [place/education] had been [taken/managed]?
Answers: Emma's mother/Emma.
Comment:
The first version is adapted from a sentence in the second paragraph
of Jane Austen's Emma. The original is
Her mother had died too long ago for her to have more than an indistinct
remembrance of her caresses; and her place had been taken by an excellent
woman as governess, who had fallen little short of a mother in affection.
Note that the original presents two further difficult disambiguation problems,
with the second and third "her"'s in the sentence.
Winograd schema in the broad sense, since two separated words are replaced.
Jane knocked on Susan's door but she did not [answer/get an answer].
Who did not [answer/get an answer]?
Answers: Susan/Jane
Comment:
Simplified version of #69. Like #69, it relies on
textual coherence for disambiguation; it is, after all, equally true that Jane
did not answer, and that Susan did not get an answer.
Joe paid the detective after he [received/delivered] the final
report on the case.
Who [received/delivered] the final report?
Answers: Joe/the detective.
Beth didn't get angry with Sally, who had cut her off, because she
stopped and [counted to ten/apologized].
Who [counted to ten/apologized]?
Answers: Beth/Sally
Jim signaled the barman and gestured toward his [empty
glass/bathroom key].
Whose [empty glass/bathroom key]?
Answers: Jim/the barman.
Dan took the rear seat while Bill claimed the front because his "Dibs!"
was [quicker/slow].
Whose "Dibs" was [quicker/slow]?
Answers: Bill/Dan
Tom said "Check" to Ralph as he [took/moved] his bishop.
Whose bishop did Tom [take/move]?
Answers: Ralph's/Tom's
As Andrea in the crop duster passed over Susan, she could see the landing
[strip/gear].
Who could see the landing [strip/gear]?
Answers: Andrea/Susan
Tom gave Ralph a lift to school so he wouldn't have to [walk/drive alone].
Who wouldn't have to [walk/drive alone]?
Answers: Ralph/Tom
Bill passed the half-empty plate to John because
he was [full/hungry].
Who was [full/hungry]?
Answers: Bill/John
Bill passed the gameboy to John because his turn was [over/next].
Whose turn was [over/next]?
Answers: Bill/John
The man lifted the boy onto his [bunk bed/shoulders].
Whose [bunk bed/shoulders]?
Answers: The boy's/the man's.
[Patting/Stretching] her back, the woman smiled at the girl.
Whose back did the woman [pat/stretch]?
Answers: The girl's/the woman's
Billy cried because Toby wouldn't [share/accept] his toy.
Who owned the toy?
Answers: Toby/Billy
Lily spoke to Donna, breaking her [concentration/silence].
Whose [concentration/silence]?
Answers: Donna/ Lily
When Tommy dropped his ice cream, Timmy giggled, so father gave him a
[stern/sympathetic] look.
Who got the look from father?
Answers: Timmy/Tommy
As Ollie carried Tommy up the long winding steps, his legs [dangled/ached].
Whose legs [dangled/ached]?
Answers: Tommy/Ollie
The father carried the sleeping boy in his [arms/bassinet].
Whose [arms/bassinet]?
Answers: The father/the boy
Comment: The concern has been raised that there may be
a statistical association between "sleeping boy" and "bassinet".
The woman held the girl against her [chest/will].
Whose [chest/will]?
Answers: The woman's/the girl's
Pam's parents came home and found her having sex with her boyfriend,
Paul. They were [embarrassed/furious] about it.
Who were [embarrassed/furious]?
Answers: Pam and Paul/Pam's parents.
Dr. Adams informed Kate that she had [cancer/retired]
and presented several options for future treatment.
Who had [cancer/retired]?
Answers: Kate/Dr. Adams
Dan had to stop Bill from toying with the injured bird.
He is very [compassionate/cruel].
Who is [compassionate/cruel]?
Answers: Dan/Bill
George got free tickets to the play, but he gave them to Eric
[because/even though] he was [particularly/not particularly] eager to
see it.
Who [was / was not] eager to see the play?
Answers:
"because" & "particularly": Eric.
"because" & "not particularly": George
"even though" & "particularly": George
"even though" & "not particularly": Eric
Comment: The format here is non-standard; this is a
cross-over,
like a two-way light switch. I am a little doubtful about the intelligibility
of "even though" with "not particularly", but the rest seem fine to me.
David Bender points out that the difficulty in that case may be due to the
fact that it is hard to find a motivation for George. Winograd schema in
the broad sense.
Jane gave Joan candy because she [was/wasn't] hungry.
Who [was/wasn't] hungry?
Answers: Joan/Jane.
Comment: From the interesting collection,
Linguistic Problems and Complexities . Similar to the previous
example, but more elegant wording.
I tried to paint a picture of an orchard, with lemons in the
lemon trees, but they came out looking more like [light bulbs /
telephone poles].
What looked like [light bulbs / telephone poles]?
Answers: The lemons / the trees
Comment: Similar to example
75 above, but not solvable by
closeness in a semantic hierarchy. However, I am not entirely confident
how easy this disambiguation is for the human reader.
James asked Robert for a favor but he [refused/was refused].
Who [refused/was refused]?
Answers: Robert/James
Kirilov ceded the presidency to Shatov because he was [more/less]
popular.
Who was [more/less] popular?
Answers: Shatov/Kirilov
Comment: In (Rahman and Ng, 2012), this example is given
with the names Medvedev and Putin. I have changed the names, because with the
real names, one can solve the problem by Googling to see which politician
is actually more popular.
Emma did not pass the ball to Janie although she [was open/saw that she
was open].
Who [was open/saw that the other player was open]?
Answers: Janie/Emma
Comment: Modified from example IV, table 1 in (Rahman and
Ng, 2012). The original text is "Emma did not pass the ball to Janie although
she [was open/should have]''; however, with ``should have'' this can be
disambiguated on syntactic grounds.
Joe saw his brother skiing on TV last night but the fool didn't
[recognize him/have a coat on]
Who is the fool?
Answers: Joe/Joe's brother.
Comment: From (Lenat 2008).
Winograd schema in the broad sense. (The ambiguity is in the
noun "fool", not in a pronoun.)
I put the [heavy book/butterfly wing] on the table and it broke.
What broke?
Answer Pair A: The table/The book
Answer Pair B: The butterfly wing/The table
Comment: From (Winograd, 1971). Thanks to Charlie Ortiz
for finding this. As in schema #93, the answer pair depends on the
choice of words.
Madonna fired her trainer because she [slept with/couldn't stand]
her boyfriend.
Who [slept with/couldn't stand] whose boyfriend?
Answer: The trainer slept with Madonna's boyfriend /
Madonna couldn't stand the trainer's boyfriend.
Comment: Non-standard form. The first
variant, with "slept with", is slightly
modified from a
headline in People magazine.
Note that for both sentences there are four possible answers:
[Madonna, trainer] x [Madonna's/trainer's boyfriend]. Note also that the
correct interpretation in the first sentence runs counter to default
constraints on sleeping with people; by default, one sleeps with one's own
boyfriend and not with other people's boyfriends. In the Victorian
era, and many other times and places, the interpretation that Madonna
fired the trainer because the trainer slept with her own boyfriend
would have been the more plausible reading.
Carol believed that Rebecca [suspected / regretted] that she had
stolen the watch.
Who is suspected of stealing the watch? / Who stole the watch?
Answer: Carol / Rebecca
This book introduced Shakespeare to [Ovid/Goethe]; it was a major
influence on his writing.
Whose writing was influenced?
Answer: Shakespeare/Goethe
Comment: A little too recondite
This book introduced Shakespeare to [Ovid/Goethe]; it was a fine
selection of his writing.
A fine selection of whose writing?
Answer: Ovid/ Shakespeare
Alice looked for her friend Jade in the crowd. Since she always
[has good luck/wears a red turban], Alice spotted her quickly.
Who always [has good luck/wears a red turban]
Answer: Alice/Jade
During a game of tag, Ethan [chased/ran from] Luke because he was "it".
Who was "it"?
Answer: Ethan/Luke
At the Loebner competition
the judges couldn't figure out which respondents
were the chatbots because they were so [advanced/stupid].
Who were so [advanced/stupid]?
Answer: the chatbots/the judges
The user changed his password from "GrWQWu8JyC" to "willow-towered Canopy
Huntertropic wrestles" as it was easy to [remember/forget].
What was easy to [remember/forget]?
Answer:
the password "GrWQWu8JyC"/the password " willow-towered
Canopy Huntertropic wrestles")
