A Collection of Winograd Schemas

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd (1972)

The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.

In his paper, ``The Winograd Schema Challenge'' Hector Levesque (2011) proposes to assemble a set of such Winograd schemas that are

• easily disambiguated by the human reader (ideally, so easily that the reader does not even notice that there is an ambiguity);
• not solvable by simple techniques such as selectional restrictions
• Google-proof; that is, there is no obvious statistical test over text corpora that will reliably disambiguate these correctly.

The set would then be presented as a challenge for AI programs, along the lines of the Turing test. The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art.

This online document presents a collection of 114 Winograd schemas. Item (1) is the original schema due to Terry Winograd; items (2) --- (19) were constructed by Hector Levesque; (20) --- (110) by Ernest Davis; and (111)---(114) by Ray Jackendoff.

We have made a deliberate effort to cover a wide range of world knowledge and a wide range of linguistic features. At the same time, many of the schemas were designed in closely related pairs; these are generally adjacent in the schema. There did not seem to be any serious drawback to including pairs of closely related schemas.

In some cases where we were uncertain whether the schema was Google-proof, we have done some experiments with searches using Google's count of result pages. These counts, however, are notoriously unreliable (Lapata and Keller, 2005) so these ``experiments'' should be taken with several grains of salt.

For the ease of machine reading, the text of each example is bracketed with the HTML tag < text > < /text >. The question is bracked with < question > < /question >. The answer is bracketed with < answer > < /answer >. Comments are bracketed with < remark > < /remark >. These tags are invisible in a browser.

The collection is open, and readers are invited to submit contributions, which will be added if the editor considers that they statisfy the constraints and are meaningfully different from the schemas already in the collection. Email submissions to davise@cs.nyu.edu.

Acknowledgements:Thanks to Hector Levesque, Leora Morgenstern, and Ray Jackendoff.

References

Hector Levesque, The Winograd Schema Challenge Commonsense-2011.

Terry Winograd, Understanding Natural Language, Academic Press, 1972.

1. The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. Who [feared/advocated] violence?
   Answers: The city councilmen/the demonstrators.

2. The trophy would not fit into the brown suitcase because it was too [small/large]. What was too [small/large]?
   Answers:The suitcase/the trophy.

3. Joan made sure to thank Susan for all the help she had [given/received]. Who had [given/received] help?
   Answers: Susan/Joan.

4. Paul tried to call George on the phone, but he was not [successful/available]. Who was not [successful/available]
   Answers: Paul/George

5. The lawyer asked the witness a question, but he was reluctant to [answer/repeat] it . Who was reluctant to [answer/repeat] the question?
   Answers: The witness/the lawyer.

6. The delivery truck zoomed by the school bus because it was going so [fast/slow]. What was going so [fast/slow]?
   Answers: The truck/the bus

7. Frank felt [vindicated/crushed] when his longtime rival Bill revealed that he was the winner of the competition. Who was the winner of the competition?
   Answers: Frank/Bill

8. The man could not lift his son because he was so [weak/heavy]. Who was [weak/heavy]?
   Answers: The man/the son.

9. The large ball crashed right through the table because it was made of [steel/styrofoam] What was made of [steel/styrofoam]?
   Answers: The ball/the table.

10. John could not see the stage with Billy in front of him because he is so [short/tall]. Who is so [short/tall]?
    Answers: John/Billy.

11. Tom threw his schoolbag down to Ray after he reached the [top/bottom] of the stairs. Who reached the [top/bottom] of the stairs?
    Answers: Tom/Ray.

12. Although they ran at about the same speed, Sue beat Sally because she had such a [good/bad] start. Who had a [good/bad] start?
    Answers: Sue/Sally.

13. The sculpture rolled off the shelf because it was not [anchored/level]. What was not [anchored/level]?
    Answers: The sculpture/the shelf.

14. Sam's drawing was hung just above Tina's and it did look much better with another one [below/above] it. Which looked better?
    Answers: Sam's drawing/Tina's drawing.

15. Anna did a lot [better/worse] than her good friend Lucy on the test because she had studied so hard. Who studied hard?
    Answers: Anna/Lucy

16. The firemen arrived [after/before] the police because they were coming from so far away. Who came from far away?
    Answers: The firemen/the police.

17. Frank was upset with Tom because the toaster he had [bought from/sold to] him did not work. Who had [bought/sold] the toaster.
    Answers: Frank/Tom.

18. Jim [yelled at/comforted] Kevin because he was so upset. Who was upset?
    Answers: Jim/Kevin.

19. The sack of potatoes had been placed [above/below] the bag of flour, so it had to be moved first. What had to be moved first?
    Answers: The sack of potatoes/the bag of flour.

20. Pete envies Martin [because/although] he is very successful. Who is very successful?
    Answers: Martin/Pete.

21. I was trying to balance the bottle upside down on the table, but I couldn't do it because it was so [top-heavy/uneven]. What was [top-heavy/uneven]?
    Answers: the bottle/the table.

22. I spread the cloth on the table in order to [protect/display] it. To [protect/display] what?
    Answers: the table/the cloth.

23. The older students were bullying the younger ones, so we [rescued/punished] them. Whom did we [rescue/punish]?
    Answers: The younger students/the older students.

24. I poured water from the bottle into the cup until it was [full/empty]. What was [full/empty]?
    Answers: The cup/the bottle.

25. Susan knows all about Ann's personal problems because she is [nosy/indiscreet]. Who is [nosy/indiscreet]?
    Answers: Susan/Anne.

26. Sid explained his theory to Mark but he couldn't [convince/understand] him. Who did not [convince/understand] whom?
    Answers: Sid did not convince Mark/Mark did not understand Sid.

27. Susan knew that Ann's son had been in a car accident, [so/because] she told her about it. Who told the other about the accident?
    Answers: Susan/Ann.

28. Joe's uncle can still beat him at tennis, even though he is 30 years [older/younger]. Who is [older/younger]?
    Answers:Joe's uncle/Joe.

    Comment: Note the following results of Google search (1/9/11).

                              "uncle older"    14,500,000 
                              "uncle younger"   9,630,000 
                              "older"         357,000,000 
                              "younger"        93,000,000
    

    Thus "uncle" is not particularly associated with "older".

29. The police left the house and went into the garage, [where/after] they found the murder weapon. Where did they find the murder weapon?
    Answers: In the garage/in the house.

    Comment: "after" is perhaps slightly awkward here.

30. The painting in Mark's living room shows an oak tree. It is to the right of [the bookcase/a house]. What is to the right of [the bookcase/a house]?
    Answers: The painting/the tree.

31. There is a gap in the wall. You can see the garden [through/behind] it. You can see the garden [through/behind] what?
    Answers: The gap/the wall.

    Comment. The syntax of the question is strictly incorrect, but ``[Behind/through] what can you see the garden?'' is almost incomprehensible.

32. The drain is clogged with hair. It has to be [cleaned/removed]. What has to be [cleaned/removed]?
    Answers: The drain/the hair.

33. My meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn't much time. Luckily, it was [short/delayed], so it worked out. What was [short/delayed]?
    Answers: The meeting/the train.

34. There is a pillar between me and the stage, and I can't [see/see around] it. What can't I [see/see around]
    Answers: The stage/the pillar.

35. They broadcast an announcement, but a subway came into the station and I couldn't [hear/hear over] it. What couldn't I [hear/hear over]?
    Answers:} The announcement/the subway.

36. In the middle of the outdoor concert, the rain started falling, [and/but] it continued until 10. What continued until 10?
    Answers: The rain/the concert.

37. I used an old rag to clean the knife, and then I put it in the [drawer/trash]. What did I put in the [drawer/trash]?
    Answers: The knife/the rag.

38. Ann asked Mary what time the library closes, [but/because] she had forgotten. Who had forgotten?
    Answers:Mary/Ann.

39. I took the water bottle out of the backpack so that it would be [lighter/handy]. What would be [lighter/handy]?
    Answers: The backpack/the bottle.

40. I couldn't put the pot on the shelf because it was too high/tall. What was too [high/tall].
    Answers: The shelf/the pot.

    Comment: The Google query ``high pot" gives about 10 times as many pages as ``tall pot" (search 9/8/11) for reasons unconnected with the meanings in this sentence.

41. I'm sure that my map will show this building; it is very [famous/good]. What is [famous/good]?
    Answers: The building/the map.

    Comment: ``Detailed'' would of course be a better word to describe the map, but it would certainly be Googlable, and probably solvable by selectional restriction.

42. Bob paid for Charlie's college education. He is very [generous/grateful]. Who is [generous/grateful]?
    Answers: Bob/Charlie.

43. Bob paid for Charlie's college education, but now Charlie acts as though it never happened. He is very [hurt/ungrateful]. Who is [hurt/ungrateful]?
    Answers: Bob/Charlie

44. Bob was playing cards with Adam and was way ahead. If Adam hadn't had a sudden run of good luck, he would have [won/lost]. Who would have [won/lost]?
    Answers: Bob/Adam.

45. Adam can't leave work here until Bob arrives to replace him. If Bob had left home for work on time, he would be [here/gone] by this time. Who would be [here/gone]?
    Answers: Bob/Adam

46. If the con artist has succeeded in fooling Sam, he would have [gotten/lost] a lot of money. Who would have [gotten/lost] the money?
    Answers: The con artist/Sam.

47. It was a summer afternoon, and the dog was sitting in the middle of the lawn. After a while, it got up and moved under the tree, because it was [hot/cooler]. What was [hot/cooler]?
    Answers: The dog/under the tree.

    Comment: Admittedly a prepositional phrase is an odd answer to a ``What?'' question, but the phrasing of the text seems natural enough, and the antecedent of ``it'' has got to be ``under the tree.''

48. The cat was lying by the mouse hole waiting for the mouse, but it was too [cautious/impatient]. What was too cautious/impatient?
    Answers: The mouse/the cat.

49. Anne gave birth to a daughter last month. She is a very charming [woman/baby]. Who is a very charming [woman/baby]?
    Answers: Anne/Anne's daughter.

50. At the party, Amy and her friends were [chatting/barking]; her mother was frantically trying to make them stop. It was very strange behavior. Who was behaving strangely?
    Answers: Amy's mother/Amy and her friends.

    Comment: Admittedly, a strange example, but I think a valid WS. Note that the question does not directly ask about the antecedent of ``It'', but phrasing it so that it does requires a clumsy answer: ``What was strange behavior?'' ``Amy's mother trying to stop her and her friends from chatting/Amy and her friends barking.''

51. I saw Jim yelling at some guy in a military uniform with a huge red beard. I don't know [who/why] he was, but he looked very unhappy. Who looked very unhappy?
    Answers: The guy in the uniform/Jim.

    Comment: If you stop the second sentence at "I don't know [who/why] he was" you still have the same ambiguity, but it becomes very difficult to ask the question. "I don't know who who was?'' is perhaps best. Of course, either way there is a reading in which the two ``he'' refer to different people, but it seems to me that that's considerably less preferred.

52. The fish ate the worm. It was [tasty/hungry]. What was [tasty/hungry]?
    Answers: The worm/the fish.

53. I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn't get it [in/out]. What couldn't I get [in/out]?
    Answers: The key/the chewing gum.

54. The dog chased the cat, which ran up a tree. It waited at the [top/bottom] Which waited at the [top/bottom]?
    Answers: The cat/the dog.

55. In the storm, the tree fell down and crashed through the roof of my house. Now, I have to get it [removed/repaired] What has to be [removed/repaired]?
    Answers: The tree/the roof.

56. The customer walked into the bank and stabbed one of the tellers. He was immediately taken to the [emergency room/police station]. Who was taken to the [emergency room/police station]?
    Answers: The teller/the customer.

57. John was doing research in the library when he heard a man humming and whistling. He was very [annoyed/annoying]. Who was [annoyed/annoying]?
    Answers:John/the hummer.

58. John was jogging through the park when he saw a man juggling watermelons. He was very [impressed/impressive] Who was [impressed/impressive]?
    Answers: John/the juggler.

59. Bob collapsed on the sidewalk. Soon he saw Carl coming to help. He was very [ill/concerned]. Who was [ill/concerned]?
    Answers: Bob/Carl.

60. Sam and Amy are passionately in love, but Amy's parents are unhappy about it, because they are [snobs/fifteen]. Who are [snobs/fifteen]?
    Answers: Amy's parents/Sam and Amy.

61. Mark told Pete many lies about himself, which Pete included in his book. He should have been more [truthful/skeptical]. Who should have been more [truthful/skeptical]?
    Answers: Mark/Pete.

62. Joe has sold his house and bought a new one a few miles away. He will be moving [out of/into] it on Thursday. Which house will he be moving [out of/into]?
    Answers: The old house/the new house.

63. Many people start to read Paul's books and can't put them down. They are [gripped/popular] because Paul writes so well. Who or what are gripped/popular?
    Answers: The readers/the books.

    Comment: A lot of synonyms for ``gripped" (e.g. ``fascinated") can be resolved by selectional restrictions. ``Gripped" cannot because it is polysemic with physical grasping.

64. Mary took out her flute and played one of her favorite pieces. She has [loved/had] it since she was a child. What has Mary [loved/had] since she was a child?
    Answers: The piece/the flute.

65. Sam pulled up a chair to the piano, but it was broken, so he had to [stand/sing] instead. What was broken?
    Answers: The chair/the piano.

66. Since it was raining, I carried the newspaper [over/in] my backpack to keep it dry. What was I trying to keep dry?
    Answers: The backpack/the newspaper.

    Comment: The form of the question is quite far from the text, but there doesn't seem to be a better way to formulate this question.

67. Sara borrowed the book from the library because she needs it for an article she is working on. She [reads/writes] it when she gets home from work. What does Sara [read/write] when she gets home from work/
    Answers: The book/the article.

68. This morning, Joey built a sand castle on the beack, and put a toy flag in the highest tower, but this afternoon the [wind/tide] knocked it down. What did the [wind/tide] knock down?
    Answers: The flag/the sand castle.

    Comment: Perhaps a little overly delicate.

69. Jane knocked on Susan's door, but there was no answer. She was [out/disappointed]. Who was [out/disappointed]??
    Answers: Susan/Jane.

70. Jane knocked on the door, and Susan answered it. She invited her to come [out/in]. Who invited whom?
    Answers: Jane invited Susan/Susan invited Jane.

71. Sam took French classes from Adam, because he was [eager/known] to speak it fluently. Who was [eager/known] to speak French fluently?
    Answers: Sam/Adam.

72. The path to the lake was blocked, so we couldn't [reach/use] it. What couldn't we [reach/use]?
    Answers: The lake/the path.

73. The sun was covered by a thick cloud all morning, but luckily, by the time the picnic started, it was [gone/out]. What was [gone/out]?
    Answers: The cloud/the sun.

74. We went to the lake, because a shark had been seen at the ocean beach, so it was a [dangerous/safer] place to swim. Which was a [dangerous/safer] place to swim?
    Answers: The beach/the lake.

75. Sam tried to paint a picture of shepherds with sheep, but they ended up looking more like [dogs/golfers]. What looked like [dogs/golfers]?
    Answers:the sheep/the shepherds.

76. Mary tucked her daughter Anne into bed, so that she could [sleep/work]. Who is going to [sleep/work]?
    Answers: Anne/Mary.

77. Fred and Alice had very warm down coats, but they were not [enough/prepared] for the cold in Alaska. Who or what were not [enough/prepared] for the cold?
    Answers: The coats/Fred and Alice.

78. Thomson visited Cooper's grave in 1765. At that date he had been [dead/travelling] for five years. Who had been [dead/travelling] for five years?
    Answers: Cooper/Thomson

79. Jackson was greatly influenced by Arnold, though he lived two centuries [earlier/later]. Who lived [earlier/later]?
    Answers: Arnold/Jackson.

80. Tom's daughter Eva is engaged to Dr. Stewart, who is his partner. The two [doctors/lovers] have known one another for ten years. Which two people have known one another for ten years?
    Answers: Tom and Dr. Stewart / Eva and Dr. Stewart.

81. I can't cut that tree down with that axe; it is too [thick/small]. What is too [thick/small]?
    Answers: The tree/the axe.

82. The foxes are getting in at night and attacking the chickens. I shall have to [guard/kill] them. What do I have to [guard/kill]?
    Answers: The chickens/the foxes.

83. The foxes are getting in at night and attacking the chickens. They have gotten very [bold/nervous]. What has gotten [bold/nervous]?
    Answers: The foxes/the chickens.

84. Fred covered his eyes with his hands, because the wind was blowing sand around. He [opened/lowered] them when the wind stopped. What did Fred [open/lower]?
    Answers: His eyes/his hands.

85. The actress used to be named Terpsichore, but she changed it to Tina a few years ago, because she figured it was [easier/too hard] to pronounce. Which name was [easier/too hard] to pronounce?
    Answers: Tina/Terpsichore.

    Comment: This question can be answered without seeing the text, just seeing the question and the choice of possible answers. I don't see that this is a defect, but it is certainly an anomaly.

86. Fred watched TV while George went out to buy groceries. After an hour he got [up/back]. Who got [up/back]?
    Answers: Fred/George.

87. Fred was supposed to run the dishwasher, but he put it off, because he wanted to watch TV. But the show turned out to be boring, so he changed his mind and turned it [on//off]. What did Fred turn [on/off]?
    Answers: The dishwasher/the television.

88. Fred is the only man still alive who remembers my great-grandfather. He [is/was] a remarkable man. Who [is/was] a remarkable man?
    Answers: Fred/my great-grandfather.

89. Fred is the only man alive who still remembers my father as an infant. When Fred first saw my father, he was twelve [years/months] old. Who was twelve [years/months] old?
    Answers: Fred/my father.

90. In July, Kamtchatka declared war on Yakutsk. Since Yakutsk's army was much better equipped and ten times larger, they were [victorious/defeated] within weeks. Who was [victorious/defeated]
    Answers: Yakutsk/Kamchatka.

91. Elizabeth moved her company from Sparta to Troy to save money on taxes; the taxes are much [higher/lower] there. Where are the taxes [higher/lower]?
    Answers: In Sparta/In Troy

92. Esther figures that she will save shipping costs if she builds her factory in Springfield instead of Franklin, because [most/none] of her customers live there. In which town do [most/none] of Esther's customers live?
    Answers: Springfield/Franklin.

93. Look! There is a [shark/minnow] swimming right below that duck! It had better get away to safety fast! What needs to get away to safety?
    Answers: The duck/the minnow.

94. There are too many deer in the park, so the park service brought in a small pack of wolves. The population should [increase/decrease] over the next few years. Which population will [increase/decrease]?
    Answers: The wolves/the deer.

95. Archaeologists have concluded that humans lived in Laputa 20,000 years ago. They hunted for [deer/evidence] on the river banks. Who hunted for [deer/evidence]?
    Answers: The prehistoric humans/the archaeologists.

96. The scientists are studying three species of fish that have recently been found living in the Indian Ocean. They [appeared/began] two years ago. Who or what [appeared/began] two years ago?
    Answers: The fish/the scientists.

97. The journalists interviewed the stars of the new movie. They were very [cooperative/persistent], so the interview lasted for a long time. Who was [cooperative/persistent]?
    Answers: The stars/the journalists

98. The police arrested all of the gang members. They were trying to [run/stop] the drug trade in the neighborhood. Who was trying to [run/stop] the drug trade?
    Answers: The gang/the police.

    Comment: Hopefully the reader is not too cynical.

99. I brought the flower pot out to the garage. It is full of [dirt/tools]. What is full of [dirt/tools]?
    Answers: The flower pot/the garage.

100. I put the cake away in the refrigerator. It has a lot of [butter/leftovers] in it. What has a lot of [butter/leftovers]?
     Answers: The cake/the refrigerator.

101. Sam broke both his ankles and he's walking with crutches. But a month or so from now they should be [better/unnecessary]. What should be [better/unnecessary]?
     Answers: The ankles/the crutches.

102. When the sponsors of the bill got to the town hall, they were surprised to find that the room was full of opponents. They were very much in the [majority/minority]. Who were in the [majority/minority]?
     Answers: The sponsors/the opponents.

103. Everyone really loved the oatmeal cookies; only a few people liked the chocolate chip cookies. Next time, we should make [more/fewer] of them. Which cookie should we make [more/fewer] of, next time?
     Answers: The oatmeal cookies/the chocolate chip.

104. We can't put a copy of the program on every chair in the theater. There are [too many/not enough] of them. There are [too many/not enough] of what?
     Answers: chairs/copies of the program.

105. I stuck a pin through a carrot. When I pulled the pin out, it [left/had] a hole. What [left/had] a hole?
     Answers: The pin/the carrot. Note: You might think this is Googlable, but in fact, ``pin left a hole" finds two results and ``pin leaves a hole" finds 66, whereas ``pin has a hole" has 11,800 (because of equipment pins, not sewing pins). The phrase "carrot has a hole" has 2 results and "carrot left a hole" has none. (8/17/10)

106. I couldn't find a spoon, so I tried using a pen to stir my coffee. But that turned out to be a bad idea, because it got full of [ink/coffee]. What got full of [ink/coffee]?
     Answers:The coffee/the pen.

     Comment: The statistical associations give the backward answer here: ``ink'' is associated with ``pen'' and ``coffee'' is associated with ``coffee''. Of course, a contestant could use a backward rule here: Since the challenge designers have excluded examples where statistics give the right answer, if you find a statistical relation, guess that the answer runs opposite to it. But that seems very risky.

107. Steve follows Fred's example in everything. He [admires/influences] him hugely. Who [admires/influences] whom?
     Answers: Steve admires Fred/Fred influences Steve.

108. The table won't fit through the doorway because it is too [wide/narrow]. What is too [wide/narrow]?
     Answers: The table/the doorway.

109. Grace was happy to trade me her sweater for my jacket. She thinks it looks [great/dowdy] on her. What looks [great/dowdy] on Grace?
     Answers: The jacket/the sweater.

110. Bill thinks that calling attention to himself was rude [to/of] Bert. Who called attention to himself?
     Answers: Bill/Bert.

111. John [hired/hired himself out to] Bill to take care of him. Who is taking care of whom?
     Answers: Bill is taking care of John/John is taking care of Bill.

112. John [promised/ordered] Bill to leave, so an hour later he left. Who left?
     Answers: John/Bill.

113. Sam Goodman's biography of the Spartan general Xenophanes conveys a vivid sense of the difficulties he faced in his [childhood/research]. Who faced difficulties?
     Answers: Xenophanes/Sam.

     Comment: It is quite possible that "biography" is correlated with "research". But even if that correlation is detected, there is another non-trivial step to realize that the research is associated with the author rather than the subject of the biography.