Non-parametric paired test

Suppose I have two approaches on the same set of subjects (e.g. patients).
Each approach gives a score for the patients (score1 for approach 1
and score2 for approach 2).
e.g. score could be a number or it could be 1 if correct and 0 if incorrect
depending on the application

patient identifier, score1, score2
id5, 10, 12
id6, 14, 13
id23, 8, 4

etc.

Call that dataset Orig.

To do a non-parametric test of significance, we find the difference
between the sum of the score1 values and the sum of the score2 values.
Call that Diff.
Without loss of generality assume Diff > 0.
(If Diff < 0, then let Diff be the difference
between the sum of the score2 values and the sum of the score1 values.)

mycount := 0
Do 10,000 times
Start with Orig.
New := {}
for each row i in Orig
  add to New Orig[i] with prob 0.5
   or Orig[i] with the score 1 and score 2 values swapped with prob 0.5
end for
NewDiff := compute Diff for New
if NewDiff > Diff
  mycount += 1
  

p-value is mycount/10,000

nof_iters = 10000

def pairedtest(n1, n2, nof_iters):
    import random
   
    ndiff = sum([ n1[i]-n2[i] for i in range(len(n1)) ])
    if ndiff < 0:
         nbig = n2
         nsmall = n1
         ndiff = -ndiff
    else:
         nbig = n1
         nsmall = n2
    
    bcount = 0
    for niter in range(nof_iters):
        tdiff = 0
        for i in range(len(nbig)):
            a = random.random()
            if a <0.5:
                tdiff += nsmall[i]-nbig[i]
            else:
                tdiff += nbig[i]-nsmall[i]
        if tdiff > ndiff:
            bcount += 1
    return bcount / nof_iters

for i in range(10):
    print(pairedtest(n1,n2, 10000))
print('-'*40)
for i in range(10):
    print(pairedtest(n2,n1, 10000))


=====

Confidence interval. 

Reformat the above orig dataset to show the differences, e.g.
if the original has 

id5, 10, 12
id6, 14, 13
id23, 8, 4
id27, 9, 13

We will reformat to

id5, -2
id6, +1
id23, +4
id27, -4


http://cs.nyu.edu/cs/faculty/shasha/papers/MeanConf.py
http://cs.nyu.edu/cs/faculty/shasha/papers/MeanConf.vals


>slides
2 1 4 -4