From jimgray@sfbay.ENET.dec.com Wed Dec 29 15:03:49 1993
Received: from CS.NYU.EDU by SHASHA.CS.NYU.EDU (5.61/1.34)
	id AA00278; Wed, 29 Dec 93 15:03:42 -0500
Received: from mts-gw.pa.dec.com by cs.NYU.EDU (5.61/1.34)
	id AA18871; Wed, 29 Dec 93 15:03:37 -0500
Received: by mts-gw.pa.dec.com; id AA28632; Wed, 29 Dec 93 12:03:26 -0800
Message-Id: <9312292003.AA28632@mts-gw.pa.dec.com>
Received: from sfbay.enet; by decpa.enet; Wed, 29 Dec 93 12:03:27 PST
Date: Wed, 29 Dec 93 12:03:27 PST
From: "Jim Gray, Digital dtn:542-3955. 455 Market, 7th fl. SF Ca 94133 tel: 415-882-3955, fax:..x3991  29-Dec-1993 1202" <jimgray@sfbay.ENET.dec.com>
To: shasha@cs.NYU.EDU, ted@squall.cis.ufl.edu
Cc: carr_richard@tandem.com, ranga@nova.ENET.dec.com, gray@sfbay.ENET.dec.com
Apparently-To: carr_richard@tandem.com, ted@squall.cis.ufl.edu,
        shasha@cs.nyu.edu
Subject: 2Q - and dynamic buffer pool sizes.
Status: R

Ted and Dennis:

 I read the 2Q paper with great interest.
 Reading it made the connection for me between LUR/2 and the CLOCK algorithm.

 Briefly, the OS folks have a similar problem: managing LRU is expensive.  
 Rather they often use a CLOCK algorithm which keeps a shift register of
  the last  N (typically 2) reference bits.  When a page is referenced 
  the leading reference bit is set.
  Clock finds a victim page by circulating a rover thru  memory.  As the rover 
  advances it shifts the reference bit right by one and clears the leading bit.
  The victim is the first page with a zero reference register.  This is fast
  and simple.

 I think there is an intersting conncetion between clock and LRU/K.  In fact
 I guess the typical 2-bit clock (the so called second chance algorithm) is
 very similar to LRU/2.

 Now, the more interesting thing is that "good" database systems grow and shirnk
 their buffer pool in response to memory glut or pressure and buffer pressure
 or glut.

 The idea is to use WSCLOCK (See Richard Carr's thesis).  Have a global clock
 for the whole machine.  When the memory manager is about to steal a page from
 the buffer pool, it has a callback asking the buffer manager to free that page.
 This causes the pool to shirink.   When the velocity of the clock goes above
 a certain threshold (say 3 pages per search or one clock cycle per 5 minutes or..)
 then the buffer manager grows the pool, exerting pressure on the memory manager.

 In this way, the DB buffer manager can cooperate with the OS memory manager.

 If the customer buys an extra GB of ram, the buffer manager will quickly consume
 it.  A batch job runs at night when there is little other pressure, then the 
 buffer manager will grow its pool. On the other hand, when the daytime load
 creates memory pressure for "ordinary" memory, the OS will steal pages back from
 the buffer manager for "real" work.

 The Tandem system works this way (thanks to Richard Carr) and I am trying to get
 the DEC system (Rdb) to work this way.

 So, here are some interesting questions:
  Is 2Q better than clock?
  What is the analog of the clock velocity to decide memory pressure 
   and drive automatic buffer growth?
Jim
 
  


From shasha@SHASHA.CS.NYU.EDU Wed Dec 29 15:23:53 1993
Received: from CS.NYU.EDU by SHASHA.CS.NYU.EDU (5.61/1.34)
	id AA00348; Wed, 29 Dec 93 15:23:52 -0500
Received: from SHASHA.CS.NYU.EDU by cs.NYU.EDU (5.61/1.34)
	id AA18907; Wed, 29 Dec 93 15:23:50 -0500
Received: by SHASHA.CS.NYU.EDU (5.61/1.34)
	id AA00341; Wed, 29 Dec 93 15:22:10 -0500
Date: Wed, 29 Dec 93 15:22:10 -0500
From: shasha@SHASHA.CS.NYU.EDU (Dennis Shasha)
Message-Id: <9312292022.AA00341@SHASHA.CS.NYU.EDU>
To: jimgray@sfbay.ENET.dec.com, shasha@cs.NYU.EDU, ted@squall.cis.ufl.edu
Subject: Re:  2Q - and dynamic buffer pool sizes.
Cc: carr_richard@tandem.com, gray@sfbay.ENET.dec.com, ranga@nova.ENET.dec.com
Status: R

Dear Jim and Richard,

Thanks very much for your note and questions.
We are currently comparing 2Q with various clock algorithms
(though we seem to have reinvented the second change one)
and are trying to incorporate 2Q and LRU/k type ideas into the
clock algorithms.
We haven't thought about changing buffer pool sizes yet.

If you have histories of buffer page accesses, we would be most grateful.
Also, we would like two copies of Richard's thesis if possible.
Our addresses are:

Theodore Johnson
Computer and Information
  Sciences Department
301 CSE
University of Florida
Gainesville, FL 32611

and



Dennis Shasha
Department of Computer Science
Courant Institute of Mathematical Sciences
New York University
251 Mercer Street
New York, N.Y. 10012-1185

Thanks very much for the feedback,
Dennis

From jimgray@sfbay.ENET.dec.com Wed Dec 29 16:52:24 1993
Received: from mts-gw.pa.dec.com by SHASHA.CS.NYU.EDU (5.61/1.34)
	id AA00568; Wed, 29 Dec 93 16:52:21 -0500
Received: by mts-gw.pa.dec.com; id AA05123; Wed, 29 Dec 93 13:51:38 -0800
Message-Id: <9312292151.AA05123@mts-gw.pa.dec.com>
Received: from sfbay.enet; by decpa.enet; Wed, 29 Dec 93 13:51:39 PST
Date: Wed, 29 Dec 93 13:51:39 PST
From: "Jim Gray, Digital dtn:542-3955. 455 Market, 7th fl. SF Ca 94133 tel: 415-882-3955, fax:..x3991  29-Dec-1993 1350" <jimgray@sfbay.ENET.dec.com>
To: ted@squall.cis.ufl.edu
Cc: shasha@shasha.cs.nyu.edu, carr_richard@tandem.com
Apparently-To: carr_richard@tandem.com, shasha@shasha.cs.nyu.edu,
        ted@squall.cis.ufl.edu
Subject: RE: Re:  2Q - and dynamic buffer pool sizes.
Status: R

Ted and Dennis:
 Great!
 I had discussed your work with Richard this morning and he commented
  that showing the 2Q is better than 2nd chance is a significant result.
 I expect you can get Richard's thisis from him (it was publshed by 
  American Elsivere.)
 Looks like you are onto something here.  Have fun!
Jim

--------------
From:	DECPA::"ted@squall.cis.ufl.edu" "Ted Johnson"    29-DEC-1993 13:19
To:	sfbay::jimgray
CC:	
Subj:	Re:  2Q - and dynamic buffer pool sizes.

Jim,
Thanks for the input, I'm glad to hear that you found our work useful.

We are following up the 2Q study by looking at algorithms
that can't perform list manipulation on every reference - and use
reference bits instead.

I'll send you a copy of the report when its written (a couple of months).
In the mean time:

1) is 2Q better than clock?
   - If you keep some reference history, clock is competitive with LRU.
     2Q beats both.
     caveat: this is a preliminary result, I need to run more experiments.
2) what is the analog of clock velocity (in 2Q)?
   - interesting question, I'll think about it.

Please do send Carr's thesis, to the address that Dennis gave.
        Ted Johnson

% ====== Internet headers and postmarks (see DECWRL::GATEWAY.DOC) ======
% Received: by mts-gw.pa.dec.com; id AA03220; Wed, 29 Dec 93 13:19:42 -0800
% Received:  from localhost  by squall.cis.ufl.edu (8.6.4/4.12) id QAA01328; Wed, 29 Dec 
1993 16:19:41 -050
% Date: Wed, 29 Dec 1993 16:19:41 -0500
% From: "Ted Johnson" <ted@squall.cis.ufl.edu>
% Message-Id: <199312292119.QAA01328@squall.cis.ufl.edu>
% To: sfbay::jimgray
% Subject: Re:  2Q - and dynamic buffer pool sizes.


From shasha Wed Dec 29 16:58:17 1993
Received: by SHASHA.CS.NYU.EDU (5.61/1.34)
	id AA00589; Wed, 29 Dec 93 16:58:00 -0500
Date: Wed, 29 Dec 93 16:58:00 -0500
From: shasha (Dennis Shasha)
Message-Id: <9312292158.AA00589@SHASHA.CS.NYU.EDU>
To: jimgray@sfbay.ENET.dec.com, ted@squall.cis.ufl.edu
Subject: RE: Re:  2Q - and dynamic buffer pool sizes.
Cc: carr_richard@tandem.com, shasha@shasha.cs.nyu.edu
Status: R

Dear Jim and Richard,

On the benchmarks we've run so far, a variant of
second chance that we've tried seems far worse than 2Q.

Would you, Jim, have any interest in my coming to your offices
on January 18 to give a talk on 2Q and seeing the
work your group is doing?

Dennis

-------
Kurt Brown -- 2q in some of their resource experiments.
