A Collection of Winograd Schemas

Outdated, archived version. Archived on November 14, 2014.

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd (1972)

The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.

In his paper, ``The Winograd Schema Challenge'' Hector Levesque (2011) proposes to assemble a set of such Winograd schemas that are

• easily disambiguated by the human reader (ideally, so easily that the reader does not even notice that there is an ambiguity);
• not solvable by simple techniques such as selectional restrictions;
• Google-proof; that is, there is no obvious statistical test over text corpora that will reliably disambiguate these correctly.

The set would then be presented as a challenge for AI programs, along the lines of the Turing test. The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art.

This online document presents a collection of 133 Winograd schemas. Item (1) is the original schema due to Terry Winograd; items (2) --- (19) were constructed by Hector Levesque; (20) --- (109), (114), (115), (135) and (137) by Ernest Davis; (110)---(113) by Ray Jackendoff; (116) --- (133) by David Bender; (136) from the web page Linguistic Problems and Complexities by the TANKA group at U. Ottawa; (138) --- (140) from (Rahman and Ng, 2012), and (141) from (Lenat, 2008). Item (114) is adapted by Ernest Davis from a sentence in Emma by Jane Austen.

We have made a deliberate effort to cover a wide range of world knowledge and a wide range of linguistic features. At the same time, many of the schemas were designed in closely related pairs; these are generally adjacent in the schema. There did not seem to be any serious drawback to including pairs of closely related schemas.

In some cases where we were uncertain whether the schema was Google-proof, we have done some experiments with searches using Google's count of result pages. These counts, however, are notoriously unreliable (Lapata and Keller, 2005) so these ``experiments'' should be taken with several grains of salt.

For the ease of machine reading, the text of each example is bracketed with the HTML tag < text > < /text >. The question is bracked with < question > < /question >. The answer is bracketed with < answer > < /answer >. Comments are bracketed with < remark > < /remark >. These tags are invisible in a browser.

There is also an XML version of this collection, in the same format that would be used in an actual running of a challenge. The following differences between the XML collection and the collection below should be noted:

• Item 113 here, which involves two concurrent wording changes, and is thus not in standard format, is omitted in the XML file.
• Item 135 here, which involves two independent wording changes, has been broken into three cases in the XML file.

Altaf Rahman and Vincent Ng, in their paper (2012) ``Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge" applied a variety of corpus-based machine learning techniques to the problem of disambiguating twin sentences. Over a corpus of sentences developed by 30 undergraduate students, they achieved an overall accuracy of 73%. That corpus of sentences is here.

The collection is open, and readers are invited to submit contributions, which will be added if the editor considers that they satisfy the constraints and are meaningfully different from the schemas already in the collection. Email submissions to davise@cs.nyu.edu.

For comparison, there is also a small collection of examples that, though interesting, in one way or another fail to meet the bar of WS schemas. To avoid confusion, these are placed on a separate page.

Acknowledgements:Thanks to Hector Levesque, Leora Morgenstern, Ray Jackendoff, David Bender, and Wei Xu.

References

Evan Ackerman, Can Winograd Schemas Replace Turing Test for Defining Human-Level AI? IEEE Spectrum, July 29, 2014.

J. Hartshorne, What is implicit causality? Language, Cognition, and Neuroscience, 2013.

A. Kehler, L. Kertz, H. Rohde, and J. Elman, Coherence and Coreference Revisited, Journal of Semantics, 25(1), 2008, 1-44.

M. Lapata and F. Keller, Web-based models for natural language processing, ACM Transactions on Speech and Language Processing 2:1, 2005.

Doug Lenat, The Voice of the Turtle: Whatever Happened to AI? AI Magazine, 29:2, 2008, 11-22.

Hector Levesque, The Winograd Schema Challenge, Commonsense-2011.

Hector Levesque, Ernest Davis, and Leora Morgenstern, The Winograd Schema Challenge, KR-2012. An expanded version of the previous item.

Hector Levesque, On Our Best Behaviour, IJCAI Research Excellence Award Presentation, 2013.

Gary Marcus, Why Can't My Computer Understand Me? The New Yorker August 18, 2013.

Altaf Rahman and Vincent Ng, Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge EMNLP, 2012.

Peter Schüller, Tackling Winograd Schemas by Formalizing Reference Theory in Knowledge Graphis KR 2014.

Terry Winograd, Understanding Natural Language, Academic Press, 1972.

Winograd Schema Challenge

AAAI-2015 Workshop: Beyond the Turing Test

Workshop Web Site

Submissions to workshop

Beyond the Turing Test: The Winograd Schema Challenge, h+ Magazine, July 28, 2014.

1. The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. Who [feared/advocated] violence?
   Answers: The city councilmen/the demonstrators.

2. The trophy doesn't fit into the brown suitcase because it's too [small/large]. What is too [small/large]?
   Answers:The suitcase/the trophy.

3. Joan made sure to thank Susan for all the help she had [given/received]. Who had [given/received] help?
   Answers: Susan/Joan.

4. Paul tried to call George on the phone, but he wasn't [successful/available]. Who was not [successful/available]?
   Answers: Paul/George

5. The lawyer asked the witness a question, but he was reluctant to [answer/repeat] it . Who was reluctant to [answer/repeat] the question?
   Answers: The witness/the lawyer.

6. The delivery truck zoomed by the school bus because it was going so [fast/slow]. What was going so [fast/slow]?
   Answers: The truck/the bus

7. Frank felt [vindicated/crushed] when his longtime rival Bill revealed that he was the winner of the competition. Who was the winner of the competition?
   Answers: Frank/Bill

8. The man couldn't lift his son because he was so [weak/heavy]. Who was [weak/heavy]?
   Answers: The man/the son.

9. The large ball crashed right through the table because it was made of [steel/styrofoam] What was made of [steel/styrofoam]?
   Answers: The ball/the table.

10. John couldn't see the stage with Billy in front of him because he is so [short/tall]. Who is so [short/tall]?
    Answers: John/Billy.

11. Tom threw his schoolbag down to Ray after he reached the [top/bottom] of the stairs. Who reached the [top/bottom] of the stairs?
    Answers: Tom/Ray.

12. Although they ran at about the same speed, Sue beat Sally because she had such a [good/bad] start. Who had a [good/bad] start?
    Answers: Sue/Sally.

13. The sculpture rolled off the shelf because it wasn't [anchored/level]. What wasn't [anchored/level]?
    Answers: The sculpture/the shelf.

14. Sam's drawing was hung just above Tina's and it did look much better with another one [below/above] it. Which looked better?
    Answers: Sam's drawing/Tina's drawing.

15. Anna did a lot [better/worse] than her good friend Lucy on the test because she had studied so hard. Who studied hard?
    Answers: Anna/Lucy

16. The firemen arrived [after/before] the police because they were coming from so far away. Who came from far away?
    Answers: The firemen/the police.

17. Frank was upset with Tom because the toaster he had [bought from/sold to] him didn't work. Who had [bought/sold] the toaster?
    Answers: Frank/Tom.

18. Jim [yelled at/comforted] Kevin because he was so upset. Who was upset?
    Answers: Jim/Kevin.

19. The sack of potatoes had been placed [above/below] the bag of flour, so it had to be moved first. What had to be moved first?
    Answers: The sack of potatoes/the bag of flour.

20. Pete envies Martin [because/although] he is very successful. Who is very successful?
    Answers: Martin/Pete.

21. I was trying to balance the bottle upside down on the table, but I couldn't do it because it was so [top-heavy/uneven]. What was [top-heavy/uneven]?
    Answers: the bottle/the table.

22. I spread the cloth on the table in order to [protect/display] it. To [protect/display] what?
    Answers: the table/the cloth.

23. The older students were bullying the younger ones, so we [rescued/punished] them. Whom did we [rescue/punish]?
    Answers: The younger students/the older students.

24. I poured water from the bottle into the cup until it was [full/empty]. What was [full/empty]?
    Answers: The cup/the bottle.

25. Susan knows all about Ann's personal problems because she is [nosy/indiscreet]. Who is [nosy/indiscreet]?
    Answers: Susan/Anne.

26. Sid explained his theory to Mark but he couldn't [convince/understand] him. Who did not [convince/understand] whom?
    Answers: Sid did not convince Mark/Mark did not understand Sid.

27. Susan knew that Ann's son had been in a car accident, [so/because] she told her about it. Who told the other about the accident?
    Answers: Susan/Ann.

    Comment: There is a large literature on "implicit causality" and "implicit consequentiality" biases (see, for example, (Hartshorne, 2013), (Kehler et al. 2008)), in which a comprehender obtains a preference for the antecedent of a pronoun in a second clause based on the content of the first clause and the connective, before seeing the content of the second clause. These are often associated with "so" and "because". For example seeing:

    "Sally frightened Mary because she ..." people interpret "she" as Sally.
    "Sally loves Mary because she ..." people interpret "she" as Mary.
    "Sally frightened Mary, so she ..." people interpret "she" as Mary.
    "Sally loves Mary, so she ..." people interpret "she" as Sally.
    

    If there are biases that favor the correct disambiguation in our example here, and if the direction of the biases can be determined purely from linguistic characteristics of the first clause e.g. from the use of the verb "knew", then this is not a valid Winograd schema, because an automated understander could do better than chance on this without employing world knowledge. However, I currently have no reason to think that this is the case.

28. Joe's uncle can still beat him at tennis, even though he is 30 years [older/younger]. Who is [older/younger]?
    Answers:Joe's uncle/Joe.

    Comment: Note the following results of Google search (1/9/11).

                              "uncle older"    14,500,000 
                              "uncle younger"   9,630,000 
                              "older"         357,000,000 
                              "younger"        93,000,000
    

    Thus "uncle" is not particularly associated with "older".

29. The police left the house and went into the garage, [where/after] they found the murder weapon. Where did they find the murder weapon?
    Answers: In the garage/in the house.

    Comment: "after" is perhaps slightly awkward here.

30. The painting in Mark's living room shows an oak tree. It is to the right of [the bookcase/a house]. What is to the right of [the bookcase/a house]?
    Answers: The painting/the tree.

31. There is a gap in the wall. You can see the garden [through/behind] it. You can see the garden [through/behind] what?
    Answers: The gap/the wall.

    Comment. The syntax of the question is strictly incorrect, but ``[Behind/through] what can you see the garden?'' is almost incomprehensible.

32. The drain is clogged with hair. It has to be [cleaned/removed]. What has to be [cleaned/removed]?
    Answers: The drain/the hair.

33. My meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was [short/delayed], so it worked out. What was [short/delayed]?
    Answers: The meeting/the train.

34. There is a pillar between me and the stage, and I can't [see/see around] it. What can't I [see/see around]?
    Answers: The stage/the pillar.

35. They broadcast an announcement, but a subway came into the station and I couldn't [hear/hear over] it. What couldn't I [hear/hear over]?
    Answers: The announcement/the subway.

36. In the middle of the outdoor concert, the rain started falling, [and/but] it continued until 10. What continued until 10?
    Answers: The rain/the concert.

37. I used an old rag to clean the knife, and then I put it in the [drawer/trash]. What did I put in the [drawer/trash]?
    Answers: The knife/the rag.

38. Ann asked Mary what time the library closes, [but/because] she had forgotten. Who had forgotten?
    Answers:Mary/Ann.

39. I took the water bottle out of the backpack so that it would be [lighter/handy]. What would be [lighter/handy]?
    Answers: The backpack/the bottle.

40. I couldn't put the pot on the shelf because it was too high/tall. What was too [high/tall]?
    Answers: The shelf/the pot.

    Comment: The Google query ``high pot" gives about 10 times as many pages as ``tall pot" (search 9/8/11) for reasons unconnected with the meanings in this sentence.

41. I'm sure that my map will show this building; it is very [famous/good]. What is [famous/good]?
    Answers: The building/the map.

    Comment: ``Detailed'' would of course be a better word to describe the map, but it would certainly be Googlable, and probably solvable by selectional restriction. One might suppose that "famous building" would be a more frequent combination than "famous map" but Google search (5/11/2012) suggests the reverse:

    "building": 2.2 billion
    "famous building": 250 million
    "map": 1.4 billion
    "famous map": 500 million

42. Bob paid for Charlie's college education. He is very [generous/grateful]. Who is [generous/grateful]?
    Answers: Bob/Charlie.

43. Bob paid for Charlie's college education, but now Charlie acts as though it never happened. He is very [hurt/ungrateful]. Who is [hurt/ungrateful]?
    Answers: Bob/Charlie

44. Bob was playing cards with Adam and was way ahead. If Adam hadn't had a sudden run of good luck, he would have [won/lost]. Who would have [won/lost]?
    Answers: Bob/Adam.

45. Adam can't leave work here until Bob arrives to replace him. If Bob had left home for work on time, he would be [here/gone] by this time. Who would be [here/gone]?
    Answers: Bob/Adam

46. If the con artist has succeeded in fooling Sam, he would have [gotten/lost] a lot of money. Who would have [gotten/lost] the money?
    Answers: The con artist/Sam.

47. It was a summer afternoon, and the dog was sitting in the middle of the lawn. After a while, it got up and moved to a spot under the tree, because it was [hot/cooler]. What was [hot/cooler]?
    Answers: The dog/The spot under the tree.

    Comment: An earlier version of this page omitted "to a spot" in the second sentence, but that was problematic. Thanks to Peter Schueller for discussion.

48. The cat was lying by the mouse hole waiting for the mouse, but it was too [cautious/impatient]. What was too cautious/impatient?
    Answers: The mouse/the cat.

49. Anne gave birth to a daughter last month. She is a very charming [woman/baby]. Who is a very charming [woman/baby]?
    Answers: Anne/Anne's daughter.

50. At the party, Amy and her friends were [chatting/barking]; her mother was frantically trying to make them stop. It was very strange behavior. Who was behaving strangely?
    Answers: Amy's mother/Amy and her friends.

    Comment: Admittedly, a strange example, but I think a valid WS. Note that the question does not directly ask about the antecedent of ``It'', but phrasing it so that it does requires a clumsy answer: ``What was strange behavior?'' ``Amy's mother trying to stop her and her friends from chatting/Amy and her friends barking.''

51. I saw Jim yelling at some guy in a military uniform with a huge red beard. I don't know [who/why] he was, but he looked very unhappy. Who looked very unhappy?
    Answers: The guy in the uniform/Jim.

    Comment: If you stop the second sentence at "I don't know [who/why] he was" you still have the same ambiguity, but it becomes very difficult to ask the question. "I don't know who who was?'' is perhaps best. Of course, either way there is a reading in which the two ``he'' refer to different people, but it seems to me that that's considerably less preferred.

52. The fish ate the worm. It was [tasty/hungry]. What was [tasty/hungry]?
    Answers: The worm/the fish.

53. I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn't get it [in/out]. What couldn't I get [in/out]?
    Answers: The key/the chewing gum.

54. The dog chased the cat, which ran up a tree. It waited at the [top/bottom]. Which waited at the [top/bottom]?
    Answers: The cat/the dog.

55. In the storm, the tree fell down and crashed through the roof of my house. Now, I have to get it [removed/repaired] What has to be [removed/repaired]?
    Answers: The tree/the roof.

56. The customer walked into the bank and stabbed one of the tellers. He was immediately taken to the [emergency room/police station]. Who was taken to the [emergency room/police station]?
    Answers: The teller/the customer.

57. John was doing research in the library when he heard a man humming and whistling. He was very [annoyed/annoying]. Who was [annoyed/annoying]?
    Answers:John/the hummer.

58. John was jogging through the park when he saw a man juggling watermelons. He was very [impressed/impressive] Who was [impressed/impressive]?
    Answers: John/the juggler.

59. Bob collapsed on the sidewalk. Soon he saw Carl coming to help. He was very [ill/concerned]. Who was [ill/concerned]?
    Answers: Bob/Carl.

60. Sam and Amy are passionately in love, but Amy's parents are unhappy about it, because they are [snobs/fifteen]. Who are [snobs/fifteen]?
    Answers: Amy's parents/Sam and Amy.

61. Mark told Pete many lies about himself, which Pete included in his book. He should have been more [truthful/skeptical]. Who should have been more [truthful/skeptical]?
    Answers: Mark/Pete.

62. Joe has sold his house and bought a new one a few miles away. He will be moving [out of/into] it on Thursday. Which house will he be moving [out of/into]?
    Answers: The old house/the new house.

63. Many people start to read Paul's books and can't put them down. They are [gripped/popular] because Paul writes so well. Who or what are gripped/popular?
    Answers: The readers/the books.

    Comment: A lot of synonyms for ``gripped" (e.g. ``fascinated") can be resolved by selectional restrictions. ``Gripped" cannot because it is polysemic with physical grasping.

64. Mary took out her flute and played one of her favorite pieces. She has [loved/had] it since she was a child. What has Mary [loved/had] since she was a child?
    Answers: The piece/the flute.

65. Sam pulled up a chair to the piano, but it was broken, so he had to [stand/sing] instead. What was broken?
    Answers: The chair/the piano.

66. Since it was raining, I carried the newspaper [over/in] my backpack to keep it dry. What was I trying to keep dry?
    Answers: The backpack/the newspaper.

    Comment: The form of the question is quite far from the text, but there doesn't seem to be a better way to formulate this question.

67. Sara borrowed the book from the library because she needs it for an article she is working on. She [reads/writes] it when she gets home from work. What does Sara [read/write] when she gets home from work?
    Answers: The book/the article.

68. This morning, Joey built a sand castle on the beach, and put a toy flag in the highest tower, but this afternoon the [wind/tide] knocked it down. What did the [wind/tide] knock down?
    Answers: The flag/the sand castle.

    Comment: Perhaps a little overly delicate.

69. Jane knocked on Susan's door, but there was no answer. She was [out/disappointed]. Who was [out/disappointed]?
    Answers: Susan/Jane.

    Comment: Note that Jane is also "out" (of the house or room); The disambiguation in this case involves issues of textual coherence. A simplified version is given in #115.

70. Jane knocked on the door, and Susan answered it. She invited her to come [out/in]. Who invited whom?
    Answers: Jane invited Susan/Susan invited Jane.

71. Sam took French classes from Adam, because he was [eager/known] to speak it fluently. Who was [eager/known] to speak French fluently?
    Answers: Sam/Adam.

72. The path to the lake was blocked, so we couldn't [reach/use] it. What couldn't we [reach/use]?
    Answers: The lake/the path.

73. The sun was covered by a thick cloud all morning, but luckily, by the time the picnic started, it was [gone/out]. What was [gone/out]?
    Answers: The cloud/the sun.

74. We went to the lake, because a shark had been seen at the ocean beach, so it was a [dangerous/safer] place to swim. Which was a [dangerous/safer] place to swim?
    Answers: The beach/the lake.

75. Sam tried to paint a picture of shepherds with sheep, but they ended up looking more like [dogs/golfers]. What looked like [dogs/golfers]?
    Answers:the sheep/the shepherds.

    Comment: It may be objected that this can be solved using the distance in the semantic hierarchy rather than reasoning about the actual visual appearance. Example 137 below avoids that objection, but is perhaps more difficult for the human reader. In any case, a program that reasons that similar looking objects are apt to be close in the semantic hierarchy is, I would say, solving the problem in a perfectly reasonable way. A program that reasons ignores the "looks like" relation and just reasons, "The referent of an anaphora is likely to be close in a semantic hierarchy to some nearby word in the sentence" is not solving the problem in a reasonable way, and is employing a fragile heuristic that is easily broken by rearranging the sentence. For example, if we reword the above "Sam tried to paint a picture of shepherds with sheep, but they ended up looking to Wilma more like [dogs/golfers]" where this heuristic will in all cases look for a referent semantically close to "Wilma".

76. Mary tucked her daughter Anne into bed, so that she could [sleep/work]. Who is going to [sleep/work]?
    Answers: Anne/Mary.

77. Fred and Alice had very warm down coats, but they were not [enough/prepared] for the cold in Alaska. Who or what were not [enough/prepared] for the cold?
    Answers: The coats/Fred and Alice.

78. Thomson visited Cooper's grave in 1765. At that date he had been [dead/travelling] for five years. Who had been [dead/travelling] for five years?
    Answers: Cooper/Thomson

79. Jackson was greatly influenced by Arnold, though he lived two centuries [earlier/later]. Who lived [earlier/later]?
    Answers: Arnold/Jackson.

80. Tom's daughter Eva is engaged to Dr. Stewart, who is his partner. The two [doctors/lovers] have known one another for ten years. Which two people have known one another for ten years?
    Answers: Tom and Dr. Stewart / Eva and Dr. Stewart.

81. I can't cut that tree down with that axe; it is too [thick/small]. What is too [thick/small]?
    Answers: The tree/the axe.

82. The foxes are getting in at night and attacking the chickens. I shall have to [guard/kill] them. What do I have to [guard/kill]?
    Answers: The chickens/the foxes.

83. The foxes are getting in at night and attacking the chickens. They have gotten very [bold/nervous]. What has gotten [bold/nervous]?
    Answers: The foxes/the chickens.

84. Fred covered his eyes with his hands, because the wind was blowing sand around. He [opened/lowered] them when the wind stopped. What did Fred [open/lower]?
    Answers: His eyes/his hands.

85. The actress used to be named Terpsichore, but she changed it to Tina a few years ago, because she figured it was [easier/too hard] to pronounce. Which name was [easier/too hard] to pronounce?
    Answers: Tina/Terpsichore.

    Comment: This question can be answered without seeing the text, just seeing the question and the choice of possible answers. I don't see that this is a defect, but it is certainly an anomaly.

86. Fred watched TV while George went out to buy groceries. After an hour he got [up/back]. Who got [up/back]?
    Answers: Fred/George.

87. Fred was supposed to run the dishwasher, but he put it off, because he wanted to watch TV. But the show turned out to be boring, so he changed his mind and turned it [on//off]. What did Fred turn [on/off]?
    Answers: The dishwasher/the television.

88. Fred is the only man still alive who remembers my great-grandfather. He [is/was] a remarkable man. Who [is/was] a remarkable man?
    Answers: Fred/my great-grandfather.

89. Fred is the only man alive who still remembers my father as an infant. When Fred first saw my father, he was twelve [years/months] old. Who was twelve [years/months] old?
    Answers: Fred/my father.

90. In July, Kamtchatka declared war on Yakutsk. Since Yakutsk's army was much better equipped and ten times larger, they were [victorious/defeated] within weeks. Who was [victorious/defeated]
    Answers: Yakutsk/Kamchatka.

91. Elizabeth moved her company from Sparta to Troy to save money on taxes; the taxes are much [higher/lower] there. Where are the taxes [higher/lower]?
    Answers: In Sparta/In Troy

92. Esther figures that she will save shipping costs if she builds her factory in Springfield instead of Franklin, because [most/none] of her customers live there. In which town do [most/none] of Esther's customers live?
    Answers: Springfield/Franklin.

93. Look! There is a [shark/minnow] swimming right below that duck! It had better get away to safety fast! What needs to get away to safety?
    Answers: The duck/the minnow.

94. There are too many deer in the park, so the park service brought in a small pack of wolves. The population should [increase/decrease] over the next few years. Which population will [increase/decrease]?
    Answers: The wolves/the deer.

95. Archaeologists have concluded that humans lived in Laputa 20,000 years ago. They hunted for [deer/evidence] on the river banks. Who hunted for [deer/evidence]?
    Answers: The prehistoric humans/the archaeologists.

96. The scientists are studying three species of fish that have recently been found living in the Indian Ocean. They [appeared/began] two years ago. Who or what [appeared/began] two years ago?
    Answers: The fish/the scientists.

97. The journalists interviewed the stars of the new movie. They were very [cooperative/persistent], so the interview lasted for a long time. Who was [cooperative/persistent]?
    Answers: The stars/the journalists

98. The police arrested all of the gang members. They were trying to [run/stop] the drug trade in the neighborhood. Who was trying to [run/stop] the drug trade?
    Answers: The gang/the police.

    Comment: Hopefully the reader is not too cynical.

99. I put the cake away in the refrigerator. It has a lot of [butter/leftovers] in it. What has a lot of [butter/leftovers]?
    Answers: The cake/the refrigerator.

100. Sam broke both his ankles and he's walking with crutches. But a month or so from now they should be [better/unnecessary]. What should be [better/unnecessary]?
     Answers: The ankles/the crutches.

101. When the sponsors of the bill got to the town hall, they were surprised to find that the room was full of opponents. They were very much in the [majority/minority]. Who were in the [majority/minority]?
     Answers: The sponsors/the opponents.

102. Everyone really loved the oatmeal cookies; only a few people liked the chocolate chip cookies. Next time, we should make [more/fewer] of them. Which cookie should we make [more/fewer] of, next time?
     Answers: The oatmeal cookies/the chocolate chip.

103. We can't put a copy of the program on every chair in the theater. There are [too many/not enough] of them. There are [too many/not enough] of what?
     Answers: chairs/copies of the program.

104. I stuck a pin through a carrot. When I pulled the pin out, it [left/had] a hole. What [left/had] a hole?
     Answers: The pin/the carrot. Note: You might think this is Googlable, but in fact, ``pin left a hole" finds two results and ``pin leaves a hole" finds 66, whereas ``pin has a hole" has 11,800 (because of equipment pins, not sewing pins). The phrase "carrot has a hole" has 2 results and "carrot left a hole" has none. (8/17/10)

105. I couldn't find a spoon, so I tried using a pen to stir my coffee. But that turned out to be a bad idea, because it got full of [ink/coffee]. What got full of [ink/coffee]?
     Answers:The coffee/the pen.

     Comment: The statistical associations give the backward answer here: ``ink'' is associated with ``pen'' and ``coffee'' is associated with ``coffee''. Of course, a contestant could use a backward rule here: Since the challenge designers have excluded examples where statistics give the right answer, if you find a statistical relation, guess that the answer runs opposite to it. But that seems very risky.

106. Steve follows Fred's example in everything. He [admires/influences] him hugely. Who [admires/influences] whom?
     Answers: Steve admires Fred/Fred influences Steve.

107. The table won't fit through the doorway because it is too [wide/narrow]. What is too [wide/narrow]?
     Answers: The table/the doorway.

108. Grace was happy to trade me her sweater for my jacket. She thinks it looks [great/dowdy] on her. What looks [great/dowdy] on Grace?
     Answers: The jacket/the sweater.

109. Bill thinks that calling attention to himself was rude [to/of] Bert. Who called attention to himself?
     Answers: Bill/Bert.

110. John [hired/hired himself out to] Bill to take care of him. Who is taking care of whom?
     Answers: Bill is taking care of John/John is taking care of Bill.

111. John [promised/ordered] Bill to leave, so an hour later he left. Who left?
     Answers: John/Bill.

112. Sam Goodman's biography of the Spartan general Xenophanes conveys a vivid sense of the difficulties he faced in his [childhood/research]. Who faced difficulties?
     Answers: Xenophanes/Sam.

     Comment: It is quite possible that "biography" is correlated with "research". But even if that correlation is detected, there is another non-trivial step to realize that the research is associated with the author rather than the subject of the biography.

113. Emma's mother had died long ago, and her [place/education] had been [taken/managed] by an excellent woman as governess. Whose [place/education] had been [taken/managed]?
     Answers: Emma's mother/Emma.

     Comment: The first version is adapted from a sentence in the second paragraph of Jane Austen's Emma. The original is

     Her mother had died too long ago for her to have more than an indistinct remembrance of her caresses; and her place had been taken by an excellent woman as governess, who had fallen little short of a mother in affection.

     Note that the original presents two further difficult disambiguation problems, with the second and third "her"'s in the sentence.

114. Jane knocked on Susan's door but she did not [answer/get an answer]. Who did not [answer/get an answer]?
     Answers: Susan/Jane

     Comment: Simplified version of #69. Like #69, it relies on textual coherence for disambiguation; it is, after all, equally true that Jane did not answer, and that Susan did not get an answer.

115. Joe paid the detective after he [received/delivered] the final report on the case. Who [received/delivered] the final report?
     Answers: Joe/the detective.

116. Beth didn't get angry with Sally, who had cut her off, because she stopped and [counted to ten/apologized]. Who [counted to ten/apologized]?
     Answers: Beth/Sally

117. Jim signaled the barman and gestured toward his [empty glass/bathroom key]. Whose [empty glass/bathroom key]?
     Answers: Jim/the barman.

118. Dan took the rear seat while Bill claimed the front because his "Dibs!" was [quicker/slow]. Whose "Dibs" was [quicker/slow]?
     Answers: Bill/Dan

119. Tom said "Check" to Ralph as he [took/moved] his bishop. Whose bishop did Tom [take/move]?
     Answers: Ralph's/Tom's

120. As Andrea in the crop duster passed over Susan, she could see the landing [strip/gear]. Who could see the landing [strip/gear]?
     Answers: Andrea/Susan

121. Tom gave Ralph a lift to school so he wouldn't have to [walk/drive alone]. Who wouldn't have to [walk/drive alone]?
     Answers: Ralph/Tom

122. Bill passed the half-empty plate to John because he was [full/hungry]. Who was [full/hungry]?
     Answers: Bill/John

123. Bill passed the gameboy to John because his turn was [over/next]. Whose turn was [over/next]?
     Answers: Bill/John

124. The man lifted the boy onto his [bunk bed/shoulders]. Whose [bunk bed/shoulders]?
     Answers: The boy's/the man's.

125. [Patting/Stretching] her back, the woman smiled at the girl. Whose back did the woman [pat/stretch]?
     Answers: The girl's/the woman's

126. Billy cried because Toby wouldn't [share/accept] his toy. Who owned the toy?
     Answers: Toby/Bill

127. Lily spoke to Donna, breaking her [concentration/silence]. Whose [concentration/silence]?
     Answers: Lily/Donna

128. When Tommy dropped his ice cream, Timmy giggled, so father gave him a [stern/sympathetic] look. Who got the look from father?
     Answers: Timmy/Tommy

129. As Ollie carried Tommy up the long winding steps, his legs [dangled/ached]. Whose legs [dangled/ached]?
     Answers: Tommy/Ollie

130. The father carried the sleeping boy in his [arms/bassinet]. Whose [arms/bassinet]?
     Answers: The father/the boy
     Comment: The concern has been raised that there may be a statistical association between "sleeping boy" and "bassinet".

131. The woman held the girl against her [chest/will]. Whose [chest/will]?
     Answers: The woman's/the girl's

132. Pam's parents came home and found her having sex with her boyfriend, Paul. They were [embarrassed/furious] about it. Who were [embarrassed/furious]?
     Answers: Pam and Paul/Pam's parents.

133. Dr. Adams informed Kate that she had [cancer/retired] and presented several options for future treatment. Who had [cancer/retired]?
     Answers: Kate/Dr. Adams

134. Dan had to stop Bill from toying with the injured bird. He is very [compassionate/cruel]. Who is [compassionate/cruel]?
     Answers: Dan/Bill

135. George got free tickets to the play, but he gave them to Eric [because/even though] he was [particularly/not particularly] eager to see it. Who [was / was not} eager to see the play?
     Answers:
     "because" & "particularly": Eric.
     "because" & "not particularly": George
     "even though" & "particularly": George
     "even though" & "not particularly": Eric
     Comment: The format here is non-standard; this is a cross-over, like a two-way light switch. I am a little doubtful about the intelligibility of "even though" with "not particularly", but the rest seem fine to me. David Bender points out that the difficulty in that case may be due to the fact that it is hard to find a motivation for George.

136. Jane gave Joan candy because she [was/wasn't] hungry. Who [was/wasn't] hungry?
     Answers: Joan/Jane.

     Comment: From the interesting collection, Linguistic Problems and Complexities . Similar to the previous example, but more elegant wording.

137. I tried to paint a picture of an orchard, with lemons in the lemon trees, but they came out looking more like [light bulbs / telephone poles]. What looked like [light bulbs / telephone poles]?
     Answers: The lemons / the trees

     Comment: Similar to example 75 above, but not solvable by closeness in a semantic hierarchy. However, I am not entirely confident how easy this disambiguation is for the human reader.

138. James asked Robert for a favor but he [refused/was refused]. Who [refused/was refused]?
     Answers: Robert/James

139. Kirilov ceded the presidency to Shatov because he was [more/less] popular. Who was [more/less] popular?
     Answers: Shatov/Kirilov

     Comment: In (Rahman and Ng, 2012), this example is given with the names Medvedev and Putin. I have changed the names, because with the real names, one can solve the problem by Googling to see which politician is actually more popular.

140. Emma did not pass the ball to Janie although she [was open/saw that she was open]. Who [was open/saw that the other player was open]?
     Answers: Emma/Janie

     Comment: Modified from example IV, table 1 in (Rahman and Ng, 2012). The original text is "Emma did not pass the ball to Janie although she [was open/should have]''; however, with ``should have'' this can be disambiguated on syntactic grounds.

141. Joe saw his brother skiing on TV last night but the fool didn't [recognize him/have a coat on] Who is the fool?
     Answer: Joe/Joe's brother.

Multilingual Winograd Schemas

French

Les conseillers municipaux ont refusé de donner un permis aux manifestants énervés parce qu'ils [avaient peur/ont preconisé] de la violence. Qui [avait peur/ont preconisé] de la violence?
Answers: les conseillers municipaux/ les manifestants énervés
Comment: From Slate.fr

Chinese (translated by Wei Xu).