Failed examples of Winograd Schemas

We present below a number of attempts at Winograd schemas that seemed promising but could not ultimately be turned into complete valid schemas.

Except where otherwise noted, these failed attempts are due to Ernest Davis. The credit for detecting the problems is shared between him, Hector Levesque, Leora Morgenstern, and David Bender.

  1. The women stopped taking the pills because they were [pregnant/carcinogenic]. Who or what were [pregnant/carcinogenic]?
    Answers: The women/the pills.

    Comment: From (Levesque 2009); deliberately created as a non-example. Solvable by selectional restrictions.

  2. The racecar zoomed by the school bus because it was going so [fast/slow]. What was going so [fast/slow]?
    Answers: [The racecar/the school bus]

    Comment: From (Levesque 2009); deliberately created as a non-example. "Fast" is associated with racecars.

  3. Frank was [jealous/pleased] when Bill said that he was the winner of the competition. Who was the winner?
    Answers: Bill/Frank.

    Comment: From (Levesque 2009); deliberately created as a non-example. The version with "pleased" is genuinely ambiguous (i.e. to the human reader). Frank might well be pleased on learning that Bill was the winner.

  4. Many astronomers are engaged in the search for distant galaxies. They are spread all over the [earth/universe]. What are spread all over the [earth/universe]?
    Answers: The astronomers/the galaxies.

    Comment: Problem: "Galaxies are spread all over the universe" may be Googlable.

  5. Though strangers, Joe and Sue ended up sharing a table at the restaurant. It was [really busy/the only one left/unavoidable/meant to be] What was [really busy/the only one left/unavoidable/meant to be]?
    Answers: The restaurant/the table/sharing the table/Joe and Sue meeting each other.

    Comment: Due to David Bender. This example is charming, but unfortunately almost all the wrong assignments can be excluded by selectional restrictions; e.g. the table cannot be "meant to be"; Joe and Sue sharing a table cannot be "the only one left." Admittedly, formulating a system of selectional restrictions capable of handling all of these is itself no walk in the park.

  6. The scientists have been studying three species of deep water fish. They were not [known/?] until a couple of years ago. What was not [known/?]
    Answers: The fish/the scientists.

    Comment: I can't find a verb that indicates scientists but is not a selectional restriction violation for "species"

  7. Tom's books are full of mistakes. Some of them are quite [foolish/worthless]. What are [foolish/worthless]?
    Answers: The mistakes/the books.

    Comment: "Foolish mistakes" is Googlable.

  8. I tried to keep the dogs out of the kitchen by putting a chair in the middle of the doorway, but it was too [wide/small]. What was too [wide/small]?
    Answers: The doorway/the chair.

    Comment: There are selectional preferences for associating "wide" with "doorway" rather with "chair" and for associating "narrow" with "doorway" rather than "chair" that suffice to give a cheap method for solving this. These are not strict selectional constraints; one can speak of a "wide chair" or a "small doorway" without seeming anomalous. But they are strong enough preferences that, if there is an ambiguity where this preference points in one direction while a spatial or physical inference points in the opposite direction, the resulting text is confusing. For example, in the sentence

    I tried to keep the dogs out of the kitchen by putting a chair in the middle of the doorway, but it was too narrow
    the reader may have difficulty realizing that, geometrically, "it" must refer to the chair, because "narrow" is a more suitable descriptor for a doorway than for a chair. See "Qualitative spatial reasoning in interpreting text and narrative," E. Davis, Spatial Cognition and Computation, to appear.

  9. I brought the flower pot out to the garage. It is full of [dirt/tools]. What is full of [dirt/tools]?
    Answers: The flower pot/the garage.

    Comment: This was actually included as an example in the collection of Winograd schemas for a while. However, a number of readers opined that the version with "tools" was unclear or confusing, because it is not that much more plausible that a garage should be full of tools than that a flower pot should be full of tools.

    Two examples of difficult reference resolution from natural text.

    The following two examples are not Winograd schemas, but they serve to illustrate the very considerable problems of reference resolution in high-quality, naturally occurring text.

  10. The affection of the whole family, the warm attachment of Miss Campbell in particular, was the more honorable to each party from the circumstances of Jane's decided superiority both in beauty and acquirements. That nature had given it in feature could not be unseen by the young woman, nor could her higher powers of mind be unfelt by the parents.
    Does "the young woman" refer to Jane or Miss Campbell?
    Answer: Miss Campbell.

    Comment: From Jane Austen, Emma, part II, chapter 2. Presumably both Miss Campbell and Jane were aware of Jane's superior beauty, but considerations of textual coherence --- not at all easy to characterize --- require that this refers to Miss Campbell. I do not feel confident that an overwhelming majority of human subjects would get this one right.

    Note also that the quoted reference to the text in the question --- a form that we avoid in the real Winograd schemas --- is difficult to avoid here. Asking about content gets pretty stilted: "Which of the young women could not help seeing that nature had given Jane superior features?"

  11. Late in 2009, the novelist Jim Powell found a cache of letter written by George Meredith to his great-great-grandmother Susan Mary Neill. Whose great-great-grandmother?
    Answers: Powell's.

    Comment: This is from the article "A peacock in the attic" by Nicholas Joukovsky and Jim Powell, Times Literary Supplement, July 22, 2011. I myself (Ernie Davis) misinterpreted this on the first reading, apparently letting the rule that people write to their own relatives override the rules that few people have living great-great-grandmothers and that people do not write to deceased relatives. One could easily turn this into Winograd schema format:

    Late in 2009, the novelist Jim Powell found a cache of letter written by the 19th century novelist George Meredith to his [great-great-grandmother/friend] Susan Mary Neill
    except that the sentence with "great-great-grandmother" is too easily misunderstood.

  12. Almanzo turned to Mr. Thompson and asked, "Did you lose a pocketbook?"

    Mr. Thompson jumped. He slapped a hand to his pocket and fairly shouted.

    "Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?"

    "Is this it?" Alamanzo asked.

    From Farmer Boy by Laura Ingalls Wilder. Note that in Almanzo's question, "Is this it?", "this" refers to the wallet that Almanzo is holding out (implicit in the text), whereas "it" refers to Mr. Thompson's wallet. The example is particularly tricky since the two are in fact co-extensional. This example was noticed by Gary Marcus.

Another Classic Example from the History of AI

Eugene Charniak's Ph.D. thesis, Toward a Model of Children's Story Comprehension, (MIT 1972), exactly contemporary with Winograd's thesis, and from the same AI lab, contains (p. 7) the following example of reference resolution, well known at the time:
Janet and Penny went to the store to get presents for Jack. Janet said, "I will get Jack a top." "Don't get Jack a top," said Penny. "He has a top. He will make you take it back."
I do not see any way to turn this into a reasonable WS. However it is a very interesting example, because the referent of "it" in the last sentence, is an entirely hypothetical, or perhaps one should say intensional, top; it is the top that Janet would buy.

On the same page of his thesis, Charniak states that the goal of his thesis is to elucidate the problem ``How do we incorporate common sense knowledge into the process of understanding natural language'' (underlined). That was before Charniak abandoned commonsense reasoning and knowledge representation in favor of statistics (see Charniak's discussion in the preface to his book Statistical Language Learning, 1993).