From jshmain@gmail.com  Tue Mar 12 15:16:15 2013
Received: from mx.cims.nyu.edu (mx.cims.nyu.edu [128.122.49.99])
	by mail.cims.nyu.edu (8.14.5+Sun/8.14.5) with ESMTP id r2CJGFOP016141
	for <shasha@mail.cims.nyu.edu>; Tue, 12 Mar 2013 15:16:15 -0400 (EDT)
Received: from mail-vb0-f48.google.com (mail-vb0-f48.google.com [209.85.212.48])
	by mx.cims.nyu.edu (8.14.4+Sun/8.14.4) with ESMTP id r2CJG6UE006405
	for <shasha@courant.nyu.edu>; Tue, 12 Mar 2013 15:16:06 -0400 (EDT)
Received: by mail-vb0-f48.google.com with SMTP id fc21so66688vbb.35
        for <shasha@courant.nyu.edu>; Tue, 12 Mar 2013 12:16:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:x-received:in-reply-to:references:date:message-id
         :subject:from:to:content-type;
        bh=2Dm4C57KguSWFiKtqUMwJEFBtX7N6Ikdq2axPGAh5Xc=;
        b=Bm+sgdNBnsd7SuExdw6PgW6FvHxOuKgN5JLK8+ikSiZjD0kRLZ7xmMgL85lzx9Vw9Y
         vAfC6vR+ikgIBq+UfNb6O+5WXg1Nrt+dWbAaeWKpXgw9jBtt+hmat6/9upYH8PTNoQLw
         V7RcYpDZW7otEkql9vAqtzuMGV9R2wbIj6Ohji0dEpFqfmlJ6hZeaeJIdOgi/9jyaOZC
         AcvWGaVoiGFZMzTwHUw+IMq4x7AfsTywpuI1CHUaPKy13e8WrydL5TGrq7/BjPzgLNUg
         Lcmh2FVsg5QwK7dz7dAyZz03clAvvCOnemOswd6yAePCOETCWJCQpz0YOO8N8lEa2juT
         gv9Q==
MIME-Version: 1.0
X-Received: by 10.52.65.73 with SMTP id v9mr6101359vds.30.1363115760693; Tue,
 12 Mar 2013 12:16:00 -0700 (PDT)
Received: by 10.58.255.194 with HTTP; Tue, 12 Mar 2013 12:16:00 -0700 (PDT)
In-Reply-To: <CAG5ZFjh5Rm9pUE0-7m9aKJ-A_by2KktJJ6Bi=XtuST1w5_FJ1A@mail.gmail.com>
References: <201303121529.r2CFT0Ed021997@crunchy12.cims.nyu.edu>
	<CAG5ZFjgTXQOGTWXRZwewRpuTQxXr0rENiEvFdiXMESucMv-j-w@mail.gmail.com>
	<CAG5ZFjh5Rm9pUE0-7m9aKJ-A_by2KktJJ6Bi=XtuST1w5_FJ1A@mail.gmail.com>
Date: Tue, 12 Mar 2013 15:16:00 -0400
Message-ID: <CAG5ZFjgthU8c6cmM3kmYgwCC-R_KC9X4ZjAJ+vbYe19tMTPEJw@mail.gmail.com>
Subject: Re: tickerplant/rdb/hdb
From: Jeffrey Shmain <jshmain@gmail.com>
To: Dennis Shasha <shasha@courant.nyu.edu>
Content-Type: multipart/mixed; boundary=20cf3071cab4b438af04d7bf1e81
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.1 (mx.cims.nyu.edu [128.122.49.99]); Tue, 12 Mar 2013 15:16:06 -0400 (EDT)
X-Virus-Scanned: clamav-milter 0.97.4 at mx
X-Virus-Status: Clean
X-Scanned-By: MIMEDefang 2.73 on 128.122.49.99
Status: R
Content-Length: 28679

--20cf3071cab4b438af04d7bf1e81
Content-Type: multipart/alternative; boundary=20cf3071cab4b438ac04d7bf1e7f

--20cf3071cab4b438ac04d7bf1e7f
Content-Type: text/plain; charset=ISO-8859-1

I have included customu.q file, which is my own implementation of the
tickerplant methods.  If the q process loads this file it becomes and
effective Ticker plant.  The flow is the following:

1.  Publisher opens a handle to the ticker plant and calls .u.pub
[tablename;records]

2.  Internally TickerPlant loops through its client state and resend the
message only to the clients who are subscribed to this table and sym field
from records.
     The structure of that tables is something like this:
         table   handle |  symlist
         -------    ---------  ------------
         `quote 0         | `IBM`HPQ
         `trade  0         | `LUV`IBM
          `quote 5        | `IBM`C`GE`GS

3.  In order for the clients to subscribe the would need to open a handle
to Ticker Plant and call .u.sub [table;symlist] remotely
        The ticker plant will update its internal table and register the
client in a table above.

4.  Upon the receipt of a message Ticker Plant checks the table above and
call upd function on the interested clients remotely.  The clients must
override upd[t;x] function in order to receive the messages;


There is also .u.end function that would be triggered by Ticker Plants on
all clients at the end of the day.  All clients must override this
function.  In reallity most of the times it will only be relevant to RDB to
dump its tables to the disk.

THhanks,

Jeff


On Tue, Mar 12, 2013 at 1:40 PM, Jeffrey Shmain <jshmain@gmail.com> wrote:

> Dennis,
>
> One more suggestion is to use another  level of partitioning (in addition
> to security ID).  This would be a source ID(aka table name). This makes
> sense logically and is very often implemented in practice.  For example
> high volume sources such as level 2 quote information could stored in a
> separate RDB and HDB from last sales.
>
> Jeff
> On Mar 12, 2013 11:41 AM, "Jeffrey Shmain" <jshmain@gmail.com> wrote:
>
>> Dennis,
>>
>> The tick in the zip file looks a bit oversimplified.  For example I dont
>> think there is logging there and log recovery.  Also the end of day message
>> that causes the RDB to flush to HDB.  I have another version that I wrote,
>> which is a bit more functional, but still not as complete as KX version.
>>
>> Jeff
>>
>>
>> On Tue, Mar 12, 2013 at 11:29 AM, Dennis Shasha <shasha@courant.nyu.edu>wrote:
>>
>>>
>>>         **** Dear Jeff,
>>>
>>> Hi Dennis,
>>>
>>> I would be happy to work with you on creating the full course outline and
>>> providing some of the content.  Below are few points that I thought would
>>> be useful:
>>>
>>> 1.  I think the first thing to introduce students to Ticker Plant/RDB
>>> would
>>> be to start with some theoretic definition of publish/subscribe model.
>>> Here is a short description on that (
>>> http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern)  Almost
>>> all
>>> financial firms employ some kind of pub/sub model and this is exactly the
>>> reason why KX has adapted this approach in their Ticker Plant/RDB
>>> implementation.
>>>
>>> *** Thanks.
>>>
>>> 2.  Here is a tutorial from First Derivatives that has some useful
>>> information about Ticker Plants/RDBs/HDBs (*
>>>
>>> http://www.foraysintolife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf)*
>>>
>>> *** Thanks.
>>>
>>> 3.  Another thing is getting u.q file from KX (Simon?).  I think Ticker
>>> Plant is their proprietary software, but I don't see why they would have
>>> any problems sharing it for educational purposes.
>>>
>>> *** I got a bunch of software from Chris Burke.
>>> http://cs.nyu.edu/cs/faculty/shasha/papers/linuxhdb.zip
>>> I think it should have anything.
>>>
>>>
>>> Now as far as the outline, you are correct:  For market data systems, I
>>> have seen the systems scaled(partitioned) by security IDs.  For order
>>> management/risk management, usually, by TraderId.  Ticker Plant uses sym
>>> column to subscribe by a specific field, but the semantic of the sym
>>> column
>>> depends on the business.
>>>
>>> As far as replication, this can be achieved via multiple methods.  The
>>> easiest is just to subscribe another RDB (on a different site) to the
>>> same
>>> Ticker Plant.  However, in that case the failure of the Ticker Plant on
>>> one
>>> site would mean the interruption of service.  We may wish to consider
>>> replicating the full thread (Ticker Plant/RDB) and give the publishers an
>>> option to publish to both.
>>>
>>> *** right I would like to show this.
>>>
>>> I like the idea of segregating queries.  Usually the Technology
>>> departments
>>> just round robin between the sites, as they do not have a good framework
>>> for query segregation.
>>>
>>> *** Cool.
>>>
>>> I have seen the systems combing RDBs with HDBs.  Usually this is achieved
>>> by having a smart gateway that connects to both.  Clients usually do not
>>> have any idea if the data is in HDB or RDB, the gateway itself breaks the
>>> queries up, hits the right place, and then combines the results before
>>> returning to clients.
>>>
>>> Repartition will be tough, but achievable.  In the places I worked
>>> repartition required to bring the system offline.  The complexity is in
>>> creating and API for clients and publishers to receive the new partition
>>> information and to automatically adjust.
>>>
>>> When do you want to start teaching the course and what do you think are
>>> the
>>> next steps?
>>>
>>> *** I'm going to make the course available in the next few days, so
>>> I'm going to have to work this stuff out in the next two weeks.
>>>
>>> Thanks,
>>>
>>> Jeff
>>>
>>>
>>> *** Thanks to you, Dennis
>>>
>>>
>>>
>>>
>>> On Mon, Mar 11, 2013 at 11:18 AM, Dennis Shasha <shasha@courant.nyu.edu
>>> >wrote:
>>>
>>> >
>>> > Dear Jeff,
>>> > So, the setup I'm imagining works in stages:
>>> >
>>> > 1. feeder client --> tickerplant --> partitioned based on securityids
>>> to
>>> > rdbs
>>> >
>>> > 2. feeder client --> tickerplant --> partitioned based on securityids
>>> to
>>> > rdbs
>>> > + replication to a single site
>>> >
>>> > 3. queries to partitioned site or replicated site depending on query
>>> type.
>>> >
>>> > 4. system combining rdb and hdb.
>>> >
>>> > 5. Ability to repartition if demand is high.
>>> >
>>> > Best,
>>> > Dennis
>>> >
>>>
>>> --047d7b6dcf4681c86504d7ba92e7
>>> Content-Type: text/html; charset=ISO-8859-1
>>> Content-Transfer-Encoding: quoted-printable
>>>
>>> <div dir=3D"ltr"><div><div><div><div><div><div><div><div><div><div>Hi
>>> Denni=
>>> s,<br><br></div>I would be happy to work with you on creating the full
>>> cour=
>>> se outline and providing some of the content.=A0 Below are few points
>>> that =
>>> I thought would be useful:<br>
>>> <br></div>1.=A0 I think the first thing to introduce students to Ticker
>>> Pla=
>>> nt/RDB would be to start with some theoretic definition of
>>> publish/subscrib=
>>> e model.=A0 Here is a short description on that (<a href=3D"
>>> http://en.wikip=
>>> edia.org/wiki/Publish%E2%80%93subscribe_pattern">
>>> http://en.wikipedia.org/wi=
>>> ki/Publish%E2%80%93subscribe_pattern<http://en.wikipedia.org/wi=ki/Publish%E2%80%93subscribe_pattern></a>)=A0
>>> Almost all financial firms emp=
>>> loy some kind of pub/sub model and this is exactly the reason why KX has
>>> ad=
>>> apted this approach in their Ticker Plant/RDB implementation.=A0 <br>
>>> <br>2.=A0 Here is a tutorial from First Derivatives that has some useful
>>> in=
>>> formation about Ticker Plants/RDBs/HDBs (<b><a href=3D"
>>> http://www.foraysint=
>>> olife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf">
>>> http://www.foraysin=
>>> tolife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf</a>)</b><br>
>>> </div><br></div>3.=A0 Another thing is getting u.q file from KX
>>> (Simon?).=
>>> =A0 I think Ticker Plant is their proprietary software, but I don&#39;t
>>> see=
>>>  why they would have any problems sharing it for educational
>>> purposes.=A0 <=
>>> br><br>
>>> </div>Now as far as the outline, you are correct:=A0 For market data
>>> system=
>>> s, I have seen the systems scaled(partitioned) by security IDs.=A0 For
>>> orde=
>>> r management/risk management, usually, by TraderId.=A0 Ticker Plant uses
>>> sy=
>>> m column to subscribe by a specific field, but the semantic of the sym
>>> colu=
>>> mn depends on the business.<br>
>>> <br></div>As far as replication, this can be achieved via multiple
>>> methods.=
>>> =A0 The easiest is just to subscribe another RDB (on a different site)
>>> to t=
>>> he same Ticker Plant.=A0 However, in that case the failure of the Ticker
>>> Pl=
>>> ant on one site would mean the interruption of service.=A0 We may wish
>>> to c=
>>> onsider replicating the full thread (Ticker Plant/RDB) and give the
>>> publish=
>>> ers an option to publish to both. =A0=A0 <br>
>>> <br></div>I like the idea of segregating queries.=A0 Usually the
>>> Technology=
>>>  departments just round robin between the sites, as they do not have a
>>> good=
>>>  framework for query segregation. <br><br></div>I have seen the systems
>>> com=
>>> bing RDBs with HDBs.=A0 Usually this is achieved by having a smart
>>> gateway =
>>> that connects to both.=A0 Clients usually do not have any idea if the
>>> data =
>>> is in HDB or RDB, the gateway itself breaks the queries up, hits the
>>> right =
>>> place, and then combines the results before returning to clients.<br>
>>> <br></div>Repartition will be tough, but achievable.=A0 In the places I
>>> wor=
>>> ked repartition required to bring the system offline.=A0 The complexity
>>> is =
>>> in creating and API for clients and publishers to receive the new
>>> partition=
>>>  information and to automatically adjust.<br>
>>> <br></div>When do you want to start teaching the course and what do you
>>> thi=
>>> nk are the next
>>> steps?<br><br>Thanks,<br><br>Jeff<br><div><div><div><div><d=
>>> iv><br><br><div><div><div><div><div><div><div><br><br></div></div></div>
>>> </div></div></div></div></div></div></div></div></div></div><div
>>> class=3D"g=
>>> mail_extra"><br><br><div class=3D"gmail_quote">On Mon, Mar 11, 2013 at
>>> 11:1=
>>> 8 AM, Dennis Shasha <span dir=3D"ltr">&lt;<a href=3D"mailto:
>>> shasha@courant.=
>>> nyu.edu" target=3D"_blank">shasha@courant.nyu.edu</a>&gt;</span>
>>> wrote:<br>
>>> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
>>> .8ex;border-left:1p=
>>> x #ccc solid;padding-left:1ex"><br>
>>> Dear Jeff,<br>
>>> So, the setup I&#39;m imagining works in stages:<br>
>>> <br>
>>> 1. feeder client --&gt; tickerplant --&gt; partitioned based on
>>> securityids=
>>>  to rdbs<br>
>>> <br>
>>> 2. feeder client --&gt; tickerplant --&gt; partitioned based on
>>> securityids=
>>>  to rdbs<br>
>>> + replication to a single site<br>
>>> <br>
>>> 3. queries to partitioned site or replicated site depending on query
>>> type.<=
>>> br>
>>> <br>
>>> 4. system combining rdb and hdb.<br>
>>> <br>
>>> 5. Ability to repartition if demand is high.<br>
>>> <br>
>>> Best,<br>
>>> Dennis<br>
>>> </blockquote></div><br></div>
>>>
>>> --047d7b6dcf4681c86504d7ba92e7--
>>>
>>>
>>

--20cf3071cab4b438ac04d7bf1e7f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>I have included customu.q file, which =
is my own implementation of the tickerplant methods.=A0 If the q process lo=
ads this file it becomes and effective Ticker plant.=A0 The flow is the fol=
lowing:<br>
<br></div>1.=A0 Publisher opens a handle to the ticker plant and calls .u.p=
ub [tablename;records]<br><br></div>2.=A0 Internally TickerPlant loops thro=
ugh its client state and resend the message only to the clients who are sub=
scribed to this table and sym field from records.<br>
</div>=A0=A0=A0=A0 The structure of that tables is something like this:=A0=
=A0 <br></div>=A0=A0=A0=A0=A0=A0=A0=A0 table =A0 handle |=A0 symlist<br><di=
v><div><div><div><div>=A0=A0=A0=A0=A0=A0=A0=A0 ------- =A0=A0 ---------=A0 =
------------<br></div><div>=A0 =A0 =A0 =A0=A0 `quote 0 =A0 =A0 =A0 =A0 | `I=
BM`HPQ<br>
</div><div>=A0 =A0 =A0 =A0=A0 `trade=A0 0 =A0 =A0 =A0 =A0 | `LUV`IBM<br></d=
iv><div>=A0 =A0 =A0 =A0 =A0 `quote 5=A0=A0=A0=A0=A0=A0=A0 | `IBM`C`GE`GS<br=
><br></div><div>3.=A0 In order for the clients to subscribe the would need =
to open a handle to Ticker Plant and call .u.sub [table;symlist] remotely<b=
r>
</div><div>=A0=A0=A0=A0=A0=A0=A0 The ticker plant will update its internal =
table and register the client in a table above.<br><br></div><div>4.=A0 Upo=
n the receipt of a message Ticker Plant checks the table above and call upd=
 function on the interested clients remotely.=A0 The clients must override =
upd[t;x] function in order to receive the messages;<br>
</div><div><br><br></div><div>There is also .u.end function that would be t=
riggered by Ticker Plants on all clients at the end of the day.=A0 All clie=
nts must override this function.=A0 In reallity most of the times it will o=
nly be relevant to RDB to dump its tables to the disk.<br>
<br></div><div>THhanks,<br><br>Jeff<br></div></div></div></div></div></div>=
<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Tue, Mar 1=
2, 2013 at 1:40 PM, Jeffrey Shmain <span dir=3D"ltr">&lt;<a href=3D"mailto:=
jshmain@gmail.com" target=3D"_blank">jshmain@gmail.com</a>&gt;</span> wrote=
:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><p dir=3D"ltr">Dennis,</p>
<p dir=3D"ltr">One more suggestion is to use another=A0 level of partitioni=
ng (in addition to security ID).=A0 This would be a source ID(aka table nam=
e). This makes sense logically and is very often implemented in practice.=
=A0 For example high volume sources such as level 2 quote information could=
 stored in a separate RDB and HDB from last sales.=A0 </p>


<p dir=3D"ltr">Jeff</p><div class=3D"HOEnZb"><div class=3D"h5">
<div class=3D"gmail_quote">On Mar 12, 2013 11:41 AM, &quot;Jeffrey Shmain&q=
uot; &lt;<a href=3D"mailto:jshmain@gmail.com" target=3D"_blank">jshmain@gma=
il.com</a>&gt; wrote:<br type=3D"attribution"><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex=
">

<div dir=3D"ltr"><div>Dennis,<br><br>The tick in the zip file looks a bit o=
versimplified.=A0 For example I dont think there is logging there and log r=
ecovery.=A0 Also the end of day message that causes the RDB to flush to HDB=
.=A0 I have another version that I wrote, which is a bit more functional, b=
ut still not as complete as KX version.=A0 <br>


<br></div>Jeff<br></div><div class=3D"gmail_extra"><br><br><div class=3D"gm=
ail_quote">On Tue, Mar 12, 2013 at 11:29 AM, Dennis Shasha <span dir=3D"ltr=
">&lt;<a href=3D"mailto:shasha@courant.nyu.edu" target=3D"_blank">shasha@co=
urant.nyu.edu</a>&gt;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><br>
=A0 =A0 =A0 =A0 **** Dear Jeff,<br>
<div><br>
Hi Dennis,<br>
<br>
I would be happy to work with you on creating the full course outline and<b=
r>
providing some of the content. =A0Below are few points that I thought would=
<br>
be useful:<br>
<br>
1. =A0I think the first thing to introduce students to Ticker Plant/RDB wou=
ld<br>
be to start with some theoretic definition of publish/subscribe model.<br>
Here is a short description on that (<br>
<a href=3D"http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern" =
target=3D"_blank">http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pa=
ttern</a>) =A0Almost all<br>
financial firms employ some kind of pub/sub model and this is exactly the<b=
r>
reason why KX has adapted this approach in their Ticker Plant/RDB<br>
implementation.<br>
<br>
</div>*** Thanks.<br>
<div><br>
2. =A0Here is a tutorial from First Derivatives that has some useful<br>
</div>information about Ticker Plants/RDBs/HDBs (*<br>
<a href=3D"http://www.foraysintolife.com/wp-content/uploads/2011/08/FDKdbTu=
torial.pdf)*" target=3D"_blank">http://www.foraysintolife.com/wp-content/up=
loads/2011/08/FDKdbTutorial.pdf)*</a><br>
<br>
*** Thanks.<br>
<div><br>
3. =A0Another thing is getting u.q file from KX (Simon?). =A0I think Ticker=
<br>
Plant is their proprietary software, but I don&#39;t see why they would hav=
e<br>
any problems sharing it for educational purposes.<br>
<br>
</div>*** I got a bunch of software from Chris Burke.<br>
<a href=3D"http://cs.nyu.edu/cs/faculty/shasha/papers/linuxhdb.zip" target=
=3D"_blank">http://cs.nyu.edu/cs/faculty/shasha/papers/linuxhdb.zip</a><br>
I think it should have anything.<br>
<div><br>
<br>
Now as far as the outline, you are correct: =A0For market data systems, I<b=
r>
have seen the systems scaled(partitioned) by security IDs. =A0For order<br>
management/risk management, usually, by TraderId. =A0Ticker Plant uses sym<=
br>
column to subscribe by a specific field, but the semantic of the sym column=
<br>
depends on the business.<br>
<br>
As far as replication, this can be achieved via multiple methods. =A0The<br=
>
easiest is just to subscribe another RDB (on a different site) to the same<=
br>
Ticker Plant. =A0However, in that case the failure of the Ticker Plant on o=
ne<br>
site would mean the interruption of service. =A0We may wish to consider<br>
replicating the full thread (Ticker Plant/RDB) and give the publishers an<b=
r>
option to publish to both.<br>
<br>
</div>*** right I would like to show this.<br>
<div><br>
I like the idea of segregating queries. =A0Usually the Technology departmen=
ts<br>
just round robin between the sites, as they do not have a good framework<br=
>
for query segregation.<br>
<br>
</div>*** Cool.<br>
<div><br>
I have seen the systems combing RDBs with HDBs. =A0Usually this is achieved=
<br>
by having a smart gateway that connects to both. =A0Clients usually do not<=
br>
have any idea if the data is in HDB or RDB, the gateway itself breaks the<b=
r>
queries up, hits the right place, and then combines the results before<br>
returning to clients.<br>
<br>
Repartition will be tough, but achievable. =A0In the places I worked<br>
repartition required to bring the system offline. =A0The complexity is in<b=
r>
creating and API for clients and publishers to receive the new partition<br=
>
information and to automatically adjust.<br>
<br>
When do you want to start teaching the course and what do you think are the=
<br>
next steps?<br>
<br>
</div>*** I&#39;m going to make the course available in the next few days, =
so<br>
I&#39;m going to have to work this stuff out in the next two weeks.<br>
<br>
Thanks,<br>
<br>
Jeff<br>
<br>
<br>
*** Thanks to you, Dennis<br>
<div><br>
<br>
<br>
<br>
On Mon, Mar 11, 2013 at 11:18 AM, Dennis Shasha &lt;<a href=3D"mailto:shash=
a@courant.nyu.edu" target=3D"_blank">shasha@courant.nyu.edu</a>&gt;wrote:<b=
r>
<br>
&gt;<br>
&gt; Dear Jeff,<br>
&gt; So, the setup I&#39;m imagining works in stages:<br>
&gt;<br>
&gt; 1. feeder client --&gt; tickerplant --&gt; partitioned based on securi=
tyids to<br>
&gt; rdbs<br>
&gt;<br>
&gt; 2. feeder client --&gt; tickerplant --&gt; partitioned based on securi=
tyids to<br>
&gt; rdbs<br>
&gt; + replication to a single site<br>
&gt;<br>
&gt; 3. queries to partitioned site or replicated site depending on query t=
ype.<br>
&gt;<br>
&gt; 4. system combining rdb and hdb.<br>
&gt;<br>
&gt; 5. Ability to repartition if demand is high.<br>
&gt;<br>
&gt; Best,<br>
&gt; Dennis<br>
&gt;<br>
<br>
</div>--047d7b6dcf4681c86504d7ba92e7<br>
Content-Type: text/html; charset=3DISO-8859-1<br>
Content-Transfer-Encoding: quoted-printable<br>
<br>
&lt;div dir=3D3D&quot;ltr&quot;&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div=
&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;Hi De=
nni=3D<br>
s,&lt;br&gt;&lt;br&gt;&lt;/div&gt;I would be happy to work with you on crea=
ting the full cour=3D<br>
se outline and providing some of the content.=3DA0 Below are few points tha=
t =3D<br>
I thought would be useful:&lt;br&gt;<br>
&lt;br&gt;&lt;/div&gt;1.=3DA0 I think the first thing to introduce students=
 to Ticker Pla=3D<br>
nt/RDB would be to start with some theoretic definition of publish/subscrib=
=3D<br>
e model.=3DA0 Here is a short description on that (&lt;a href=3D3D&quot;<a =
href=3D"http://en.wikip" target=3D"_blank">http://en.wikip</a>=3D<br>
<a href=3D"http://edia.org/wiki/Publish%E2%80%93subscribe_pattern" target=
=3D"_blank">edia.org/wiki/Publish%E2%80%93subscribe_pattern</a>&quot;&gt;<a=
 href=3D"http://en.wikipedia.org/wi=3Dki/Publish%E2%80%93subscribe_pattern"=
 target=3D"_blank">http://en.wikipedia.org/wi=3D<br>


ki/Publish%E2%80%93subscribe_pattern</a>&lt;/a&gt;)=3DA0 Almost all financi=
al firms emp=3D<br>
loy some kind of pub/sub model and this is exactly the reason why KX has ad=
=3D<br>
apted this approach in their Ticker Plant/RDB implementation.=3DA0 &lt;br&g=
t;<br>
&lt;br&gt;2.=3DA0 Here is a tutorial from First Derivatives that has some u=
seful in=3D<br>
formation about Ticker Plants/RDBs/HDBs (&lt;b&gt;&lt;a href=3D3D&quot;<a h=
ref=3D"http://www.foraysint" target=3D"_blank">http://www.foraysint</a>=3D<=
br>
<a href=3D"http://olife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf" t=
arget=3D"_blank">olife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf</a>=
&quot;&gt;<a href=3D"http://www.foraysin" target=3D"_blank">http://www.fora=
ysin</a>=3D<br>



<a href=3D"http://tolife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf" =
target=3D"_blank">tolife.com/wp-content/uploads/2011/08/FDKdbTutorial.pdf</=
a>&lt;/a&gt;)&lt;/b&gt;&lt;br&gt;<br>
&lt;/div&gt;&lt;br&gt;&lt;/div&gt;3.=3DA0 Another thing is getting u.q file=
 from KX (Simon?).=3D<br>
=3DA0 I think Ticker Plant is their proprietary software, but I don&amp;#39=
;t see=3D<br>
=A0why they would have any problems sharing it for educational purposes.=3D=
A0 &lt;=3D<br>
br&gt;&lt;br&gt;<br>
&lt;/div&gt;Now as far as the outline, you are correct:=3DA0 For market dat=
a system=3D<br>
s, I have seen the systems scaled(partitioned) by security IDs.=3DA0 For or=
de=3D<br>
r management/risk management, usually, by TraderId.=3DA0 Ticker Plant uses =
sy=3D<br>
m column to subscribe by a specific field, but the semantic of the sym colu=
=3D<br>
mn depends on the business.&lt;br&gt;<br>
&lt;br&gt;&lt;/div&gt;As far as replication, this can be achieved via multi=
ple methods.=3D<br>
=3DA0 The easiest is just to subscribe another RDB (on a different site) to=
 t=3D<br>
he same Ticker Plant.=3DA0 However, in that case the failure of the Ticker =
Pl=3D<br>
ant on one site would mean the interruption of service.=3DA0 We may wish to=
 c=3D<br>
onsider replicating the full thread (Ticker Plant/RDB) and give the publish=
=3D<br>
ers an option to publish to both. =3DA0=3DA0 &lt;br&gt;<br>
&lt;br&gt;&lt;/div&gt;I like the idea of segregating queries.=3DA0 Usually =
the Technology=3D<br>
=A0departments just round robin between the sites, as they do not have a go=
od=3D<br>
=A0framework for query segregation. &lt;br&gt;&lt;br&gt;&lt;/div&gt;I have =
seen the systems com=3D<br>
bing RDBs with HDBs.=3DA0 Usually this is achieved by having a smart gatewa=
y =3D<br>
that connects to both.=3DA0 Clients usually do not have any idea if the dat=
a =3D<br>
is in HDB or RDB, the gateway itself breaks the queries up, hits the right =
=3D<br>
place, and then combines the results before returning to clients.&lt;br&gt;=
<br>
&lt;br&gt;&lt;/div&gt;Repartition will be tough, but achievable.=3DA0 In th=
e places I wor=3D<br>
ked repartition required to bring the system offline.=3DA0 The complexity i=
s =3D<br>
in creating and API for clients and publishers to receive the new partition=
=3D<br>
=A0information and to automatically adjust.&lt;br&gt;<br>
&lt;br&gt;&lt;/div&gt;When do you want to start teaching the course and wha=
t do you thi=3D<br>
nk are the next steps?&lt;br&gt;&lt;br&gt;Thanks,&lt;br&gt;&lt;br&gt;Jeff&l=
t;br&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;d=3D<br>
iv&gt;&lt;br&gt;&lt;br&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;div&gt;&lt;d=
iv&gt;&lt;div&gt;&lt;div&gt;&lt;br&gt;&lt;br&gt;&lt;/div&gt;&lt;/div&gt;&lt=
;/div&gt;<br>
&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt=
;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=3D3D&quot;g=3D<b=
r>
mail_extra&quot;&gt;&lt;br&gt;&lt;br&gt;&lt;div class=3D3D&quot;gmail_quote=
&quot;&gt;On Mon, Mar 11, 2013 at 11:1=3D<br>
8 AM, Dennis Shasha &lt;span dir=3D3D&quot;ltr&quot;&gt;&amp;lt;&lt;a href=
=3D3D&quot;mailto:<a href=3D"mailto:shasha@courant." target=3D"_blank">shas=
ha@courant.</a>=3D<br>
<a href=3D"http://nyu.edu" target=3D"_blank">nyu.edu</a>&quot; target=3D3D&=
quot;_blank&quot;&gt;<a href=3D"mailto:shasha@courant.nyu.edu" target=3D"_b=
lank">shasha@courant.nyu.edu</a>&lt;/a&gt;&amp;gt;&lt;/span&gt; wrote:&lt;b=
r&gt;<br>


&lt;blockquote class=3D3D&quot;gmail_quote&quot; style=3D3D&quot;margin:0 0=
 0 .8ex;border-left:1p=3D<br>
x #ccc solid;padding-left:1ex&quot;&gt;&lt;br&gt;<br>
Dear Jeff,&lt;br&gt;<br>
So, the setup I&amp;#39;m imagining works in stages:&lt;br&gt;<br>
&lt;br&gt;<br>
1. feeder client --&amp;gt; tickerplant --&amp;gt; partitioned based on sec=
urityids=3D<br>
=A0to rdbs&lt;br&gt;<br>
&lt;br&gt;<br>
2. feeder client --&amp;gt; tickerplant --&amp;gt; partitioned based on sec=
urityids=3D<br>
=A0to rdbs&lt;br&gt;<br>
+ replication to a single site&lt;br&gt;<br>
&lt;br&gt;<br>
3. queries to partitioned site or replicated site depending on query type.&=
lt;=3D<br>
br&gt;<br>
&lt;br&gt;<br>
4. system combining rdb and hdb.&lt;br&gt;<br>
&lt;br&gt;<br>
5. Ability to repartition if demand is high.&lt;br&gt;<br>
&lt;br&gt;<br>
Best,&lt;br&gt;<br>
Dennis&lt;br&gt;<br>
&lt;/blockquote&gt;&lt;/div&gt;&lt;br&gt;&lt;/div&gt;<br>
<br>
--047d7b6dcf4681c86504d7ba92e7--<br>
<br>
</blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>

--20cf3071cab4b438ac04d7bf1e7f--
--20cf3071cab4b438af04d7bf1e81
Content-Type: application/octet-stream; name="customu.q"
Content-Disposition: attachment; filename="customu.q"

XGQgLnUKCnc6KFt0YWJsZTpgc3ltYm9sJCgpOyBoYW5kbGU6YGludCQoKV07IHN5bWxpc3Q6MGgk
KCkpOyAgICAgLy8gIFN1YnNjcmlwdGlvbiBsaXN0OyBrZXlzIGlzIHRhYmxlcyBhbmQgdmFsdWVz
IGFyZSBjbGllbnRzCnQ6KGB0cmFkZWBxdW90ZWBzcGxpdHNgaW5zdXJhbmNlKTsgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvLyBUYWJsZXMgZGVmaW5l
ZCBpbiB0aGlzIHEgcHJvY2VzcwoKLy8gSW5pdGlhbGl6ZSAudS50IGFuZCAudS53CmluaXQ6eyB0
Ojp0YWJsZXNbXTsgaGNsb3NlIGVhY2ggZXhlYyBoYW5kbGUgZnJvbSAudS53OyBkZWxldGUgZnJv
bSBgLnUud30KCi8vIE92ZXJyaWRlIC56LnBjIHRvIGNsZWFuIHVwIC51Lncgb24gY2xpZW50IGRp
c2Nvbm5lY3RzCmRlbDp7W3RibDtobmRsXSBkZWxldGUgZnJvbSBgLnUudyB3aGVyZSBoYW5kbGU9
aG5kbCx0YWJsZT10YmwgfQouei5wYzp7ZGVsWzt4XSBlYWNoIHR9CgovLyBTZWxlY3Rpb24gY3Jp
dGVyaWEgZm9yIG1lc3NhZ2UgcHVibGlzaGluZyAtIHN5bSBjb2x1bW4gbXVzdCBiZSBkZWZpbmVk
IGluIGFsbCBtZXNzYWdlcwpzZWw6e1tyZWNzO3N5bXNdCiAgICAgICAgcmVzOiRbc3ltcyB+IGVu
bGlzdCBgOyByZWNzOyBzZWxlY3QgZnJvbSByZWNzIHdoZXJlIHN5bSBpbiBzeW1zXTsKICAgICAg
ICA6cmVzOwogIH0KCi8vIFB1Ymxpc2ggbWV0aG9kIHRvIGl0ZXJhdGUgdGhyb3VnaCBhbGwgY2xp
ZW50cyBhbmQgY2FsbCB1cGQgb24gdGhlbSByZW1vdGVseQpwdWI6e1t0O3hdCiAgICAgICAge1t0
O3g7aF0KICAgICAgICAgICB4OnNlbFt4OyAudS53W3QsaF1gc3ltbGlzdF07CiAgICAgICAgICAg
aWZbMDxjb3VudCB4OyAobmVnIGgpKGB1cGQ7dDt4KV0gfVt0O3hdIGVhY2ggZXhlYyBkaXN0aW5j
dCBoYW5kbGUgZnJvbSAudS53IHdoZXJlIHRhYmxlPXQ7CiAgfQoKLy8gU3Vic2NyaXB0aW9uIG1l
dGhvZCBmb3IgdGhlIGNsaWVudHMgdG8gY2FsbAovLyBXaWxsIGFkZCB0aGUgY2xpZW50cyB0byAu
dS53CnN1Yjp7W3RibHM7c3ltc10KICAgICAgICAvLyBpZiBgLCBzdWJzY3JpYmUgdGhpcyBoYW5k
bGUgdG8gYWxsIHRhYmxlcwogICAgICAgICAgIGlmW3RibHN+YDsgOnN1Yls7c3ltc10gZWFjaCB0
XTsKCiAgICAgICAgLy8gaWYgdGhlIHRhYmxlIGRvZXMgbm90IGV4aXN0IHJldHVybiB0byB0aGUg
Y2xpZW50IHRoZSBuYW1lIG9mIHRoaXMgdGFibGUKICAgICAgICAgICBpZltub3QgYWxsIHRibHMg
aW5cOnQ7ICd0YmxzXTsKCiAgICAgICAgLy8gQWRkIHRoZSBuZXcgc3Vic2NyaXB0aW9uIGluZm8g
dG8gLnUudyAod2lsbCBvdmVycmlkZSB0aGUgb2xkIG9uZSBmb3IgdGhpcyB0YWJsZS9oYW5kbGUp
CiAgICAgICAgY250OmNvdW50IHRibHM7CiAgICAgICAgaWZbLTExaCA9IHR5cGUgc3ltczsgc3lt
czplbmxpc3Qgc3ltc107CiAgICAgICAgYC51LncgdXBzZXJ0ICBmbGlwICAodGJscztjbnQjLnou
dztjbnQjZW5saXN0IHN5bXMpOwogICAgICAgIC8vIFJldHVybiB0aGUgc2NoZW1hcyBmcm9tIC51
LnQgdG8gdGhlIGNsaWVudAogICAgICAgIDp7KHg7IGdldCB4KX0gZWFjaCB0YmxzOwogfQoKCi8v
IHdpbGwgaXRlcmF0ZSB0aHJvdWdoIGNsaWVudCBsaXN0IGFuZCBpbnZva2UgLnUuZW5kIG9uIHRo
ZW0gcmVtb3RlbHkKZW5kOnt7KG5lZyB4KSAiLnUuZW5kW10iICB9IGVhY2ggZXhlYyBkaXN0aW5j
dCBoYW5kbGUgZnJvbSAudS53fQoKZGVsdGljazp7dmFsdWUgImRlbGV0ZSBmcm9tIGAiLChzdHJp
bmcgeCk7fQoKCgoKXGQgLgoK
--20cf3071cab4b438af04d7bf1e81--

