|T(D1) intersect T(D2)| / |T(D1) union T(D2)| (Jacquard coefficient).

Algorithm for finding most similar pages in a collection (Jacquard coefficient)

for (each document D in collection C) for (each term T in D) write record (T,D) to file F1; sort F1 by T; aggregate records from F1 into form (T,S) where S is the set of documents containing T (discard terms where |S|=1). for (each record (T,S)) for each pair D1,D2 in S write record (D1,D2,1) to file F2; sort F2 by D1,D2; find number of records (D1,D2,1). This is equal to |T(D1) intersect T(D2)|; |T(D1) union T(D2)| = |T(D1)| + |T(D2)| - |T(D1) intersect T(D2)| r = |T(D1) intersect T(D2)| / |T(D1) union T(D2)|; save pairs with largest values of r (or with r greater than threshhold).This can be adapted to the vector model as follows:

Algorithm for finding most similar pages in a collection (vector model)

for (each document D in collection C) for (each term T in D) { W := coefficient of T in normalized vector of D; write record (T,D,W) to file F1; sort F1 by T; aggregate records from F1 into form (T,S) where S is the set of pairs [D,W], where (T,D,W) has been written on F1 for (each record (T,S)) for each (pair [D1,W1], [D2,W2] in S) write record (D1,D2,W1*W2) to file F2; sort F2 by D1,D2; for (each pair of document (D1,D2) written to F2) { similarity(D1,D2) = sum of X over all records [D1,D2,X] in F2; return the most similar pairs of documents, or pairs of documents with similarity greater than threshhold.

Estimating the Jacquard coefficient of sets A and B using random sampling:

Step 1: Choose a value m and a random function P(W) from elements to integers. Step 2: For set X, let V(X) = { x in X | P(x) mod m = 0} be a sample of X. This is called thesketchof X.

Step 3: | V(A) intersect V(B) | / |V(A) union V(B) | is an estimate of the similarity of A and B.

Note: if you just sample A and sample B, get a substantial underestimate of the resemblance. Also note that you can get an independent estimate by using a different random function (or relatively prime modulus), so by using several of these you can get high precision and confidence.

Shingle : K consecutive words.

Fix a shingle size K. Let S(A) be the set of shingles in A and
let S(B) be the set of shingles in B.

Compute |S(D1) intersect S(D2)| / |S(D1) union S(D2)| (Jacquard coefficient).

(Rather than save actual shingle, you can use a fingerprint of the shingle.)

Implementation uses shingle size = 10, m = 25, 40 bit sketch.

High-level algorithm from Broder et al.

Step 1: Normalize by removing HTML and converting to lower-case.

Step 2: Calculate sketch of shingle set of each document.

Step 3: For each pair of documents, compare sketches to see if they
exceed resemblance threshhold.

Step 4: Find connected components.

Further issues:

- Eliminate very common shingles (more than 1000 documents). Almost all automatically generated by standard programs.
- Eliminate truly identical documents by computing and comparing fingerprint of entire document.

Q = set of documents retrieved

R = set of relevant documents

QR -- True positives.

Q(D-R) -- False positives. (Irrelevant documents retrieved)

(D-Q)R -- False negatives. (Relevant documents omitted)

(D-Q)(D-R) -- True negatives. (Irrelevant documents omitted)

Percentage correct = (|QR| + |(D-Q)(D-R)|) / D.

Not a good measure; counts false positives and false negatives equally.

E.g. suppose |R|=3.

Q1 has two relevant documents and three irrelevant documents.

Q2 returns one irrelevant document.

Then both are making the same number of errors (4), but clearly Q1 is
better than Q2.

Standard measures in IR (in fact, in all applications where the objective
is to find a set of solutions):

Precision = |QR| / |Q| -- fraction of retrieved documents that are relevant
= 1 - (fraction of retrieved documents that are false positives).

Recall =
|QR| / |R| -- fraction of relevant documents that are retrieved
= 1 - (fraction of relevant documents that are false negatives).

In the above example Q1 has precision 2/5 and recall 2/3. Q2 has precision and recall = 0.

If Q1 subset Q2, then Recall(Q2) >= Recall(Q1). Prec(Q2) can be
either greater or less than Q2. If you consider the precision over the
first K documents returned for K = 1, 2, ... then the precision goes
up every time d_{K} is relevant and down every time it is
irrelevant, so graph is sawtoothed. But on the whole precision tends to
go down, so there is a trade-off between recall and precision as you get
more documents.

Smoothed precision: Plot precision only at points when documents have been found; interpolate in between. Set precision(0)=1. Then precision can be monotonically decreasing, and will tend to be so except possibly at beginning.

Probabilistic model. Suppose that the
matcher returns a measure of the "quality" of the document for the query.
Suppose that this measured quality has some value in the following
sense:

If q1 > q2, then Prob(d in R | qual(d)=q1) >
Prob(d in R | qual(d) = q2)

Let Q_{T} = { d | qual(d) > = T }.

Then the expected value of precision(Q_{T}) is a decreasing
function of T. The expected value of recall(Q_{T}) is
an increasing function of T, but concave downward.

- Users generally don't care much about recall. (Not entirely clear that precision is exactly the right measure either.)
- Measuring recall involves identifying R. Always difficult; on Web, extremely difficult.
- Doesn't take into account
*order*of answers. - Doesn't take into account
*degree*of relevance. - Two numbers (two curves, as functions of T). Would prefer one number.
- Zero values
- What do you mean by "relevance" anyway?

1/F = average(1/p,1/r)

F = 2pr/(p+r).

If either p or r is small then F is small. If p and r are close then F is about the average of p and r.

Generalized precision: Value of information obtained for user / cost of examining results.

Generalized recall: Value of information obtained / Value of optimal results (or: value of entire Web for user's current need)

** Average precision**: Average of precision at 20% recall, 50%, 80%.
Or average of precision at recall = 0%, 10%, 20% ... 90%, 100%. (Since
recall is does not attain these value exactly, and since recall remains
constant until next relevant document found, and thus same value
of recall can have several values of precision, take max precision
or avg precision, and interpolate. Similarly, precision at recall = 0% is
extrapolated.)

** Precision over first K documents:**
(or average relevance over first K documents).
User model: User will only read first K documents.

** Rank of Kth relevant document ** (or rank such that sum of relevance = K).
User model: User will read until he has gotten K relevant documents
(or documents whose total relevance is K).

** Weighted precision:**
sum_{K in Q} rel(d_{K}) / |Q|.

** Weighted recall:**
sum_{K in Q} rel(d_{K}) /
sum_{K in R} rel(d_{K})

** Order diminishing sum:** Value of search is sum_{K}
rel(d_{K}) * p^{K}. User model: User starts reading
at beginning, at each step continues with probability p.

** Total content precision:** Total information relevant to query
in pages retrieved divided by
|Q| (or divided by total reading time). Of course, this is hard to quantify.
You can, for example, prepare a list of questions on the subject matter,
and measure "Total information" as the fraction of questions that can
be answered from the retrieved texts.

** Total content recall:** Total information relevant to query
in pages retrieved divided by total information relevant to query in Web.

- The crawler has not indexed it.
- The retriever doesn't recognize that it is relevant.

First case is hopeless. (We will talk later about how to estimate the size of this set, but no way to estimate # of relevant documents.)

Second case:

- Broaden query as far as feasible. Include disjunction of lots of related words, synonyms, alternative spellings. Set threshhold as low as possible.
- "Seed" web with relevant documents. Or identify specific subset of documents that you know to be indexed (e.g. articles from specific journals.) See what fraction are retrieved, and extrapolate. (E.g. if 40% of seed documents are retrieved and 200 documents total are retrieved, estimate 500 relevant documents total.) Hard to be confident that seed documents are representative of existing documents, relative to query and engine. Impossible to place seed documents at a collection of sites representative as regards PageRank.
- For accessible subcollection (e.g. newsgroup): randomly sample subcollection and count relevant documents.

- Relevance to query as stated
- Relevance to actual user question. (Includes limitations of query language; user skill at formulating question).
- Interest to user after the fact.
- Originality; new information contained (depends on other docs retrieved or previous knowledge of user).
- Authority: User has confidence in information contained. (Much more an issue in Web than in traditional IR.)

Measured:

- By user questionaire/external judge.
- By clicking through.
- By time spent reading
- By external independent index.

Observational studies: users observed as they use web for their own purposes. Ecologically valid. The more interference, the less ecologically valid. (Just informing users that they are observed alters their behavior; however, there can be privacy issues if they are observed without being informed.)

Controlled experiment: Users carry out task specified by experimenter in controlled setting. Much more information per task, much more demanding of user, possible to design narrowly focussed experiment, less clearly representative of "normal" use.

How are queries chosen? Just taking the most frequent queries probably does not give a good test; these are neither very interesting nor very high-minded. Plus the search engines may well have hand-edited the answers. The same issue arises generally with regard to creating a corpus of benchmark problems. There must be some theory, but I don't know it.

- 1. Web content
- 1.A No information on Web
- 1.B Incorrect/outdated information
- 1.C Information hard to find on site. Page poorly designed, bad links, site too big, etc.
- 1.D Redundancy between web pages.

- 2. Crawler problem
- 2.A Document not indexed
- 2.B URL out of date (document moved)
- 2.C Dead link
- 2.D URL content changed
- 2.E Multiple copies of identical/near-identical page

- 3. Retrieval problem
- 3.A False positive
- 3.B False negative
- 3.C Misjudge importance of page
- 3.D Misordering of pages
- 3.E Wrong page on Web site (e.g. link to internal page when top-level page would be better or vice versa)
- 3.F Engine too slow

- 4. Query language problem
- 4.A Insufficiently expressive
- 4.B Ill-defined.
- 4.C Too complicated / misleading

- 5. User problem
- 5.A Poor choice of query terms
- 5.B Ineffective use of query language
- 5.C Ineffective use of browser
- 5.D Information overload -- too many pages causes user to give up.
- 5.E Impatient
- 5.F Distracted

- 6. Results page problem
- 6.A Confusing format
- 6.B Misleading snippets

- 7. Browsing problem
- 7.A Page cannot be downloaded
- 7.B Page takes too long to download
- 7.C Page cannot be correctly displayed

- 8. Page unsuitable to user
- 8.A Too elementary
- 8.B Too advanced
- 8.C Wrong language
- 8.D Pornographic or objectionable

- 9. Non-text media
- 9.A Poor quality at web site.
- 9.B Browser doesn't know format.
- 9.C Poor quality at browser.
- 9.D Limited textual information. (In almost all real cases, such media are indexed by related text.)
- 9.E Format unsuited to user (e.g. limited vision/hearing).

Tester specifies query; test subject read first 30 pages; labels each page "relevant", "irrelevant", or category of failure (e.g. "bad link"; "too long to download" etc.) This tests separately 3.A possibly combined with 3.E; 2.C; 7.A; 7.B.

Tester is aware (not through search engine) of a valuable page; runs a variety of queries; tabulates fraction of queries for which page is in top 100. Combines 2.A, 3.B, 3.C, 3.D.

Tester specifies list of questions in some subject area to be answered in fixed time period; test subjects use search engine as best they can. This combines pretty much all possible errors.