Jennifer Spencer jennifer.spencer at stanford.edu
Fri Aug 29 13:12:17 PDT 2008
Thanks everyone for your responses and your help.  I needed the clarifications and thought 
provocations you provided before presenting the situation to our management.

After a long meeting and a lot of time at the white board, we've decided to try the following:
essentially, make clusters act as sets.

> Now, here's an incredibly ugly kludge I just thought of and haven't
> tested that would make this possible.  You create a bunch more targets
> on node 2, each of which subscribes to the target set. 
I considered going this route for a few minutes, until I began to draw it out on paper.  My problem 
with that is that some of the subscribers I have want some of the same (large, frequently updated) 
data tables, so we'd up with extra duplicates.  Bad.

Instead, we will figure out the most efficient division of frequently-desired data tables, and make 
each division into its own cluster (with one set).  We will run multiple clusters between nodes 1 & 2, 
having two slons going for each cluster, one with archiving on.  Each archive job will write to a 
different directory.  Different customers can pick up from whichever of those directories they want, 
being careful to manage their own directory structure so the log filenames don't overlap.  If you see 
any obvious problems with this, please feel free to let me know!

It was either that or write a parsing (filtering) software application for our log-shipping logs (in 
my copious spare time).

I am glad you guys were here to bounce these ideas off.  I kept thinking "maybe there's something I am 
missing, or overthinking...".
-Jennifer


Christopher Browne wrote:
> Andrew Sullivan <ajs at crankycanuck.ca> writes:
>> That said, this makes my teeth hurt just to suggest it.  But the log
>> shipping today doesn't really do what you want.  I know there was some
>> intended work to allow filtering in 2.x.  I don't know if it got off
>> the ground.  Chris?
> 
> Jan's looking at implementing an admin tool to be rather friendlier
> than what we have now; the notion of doing filtering seems to be a
> good couple of steps further down the road.
> 
> That's smelling like "3.x", at this point.  We have loosely talked
> about having "smarter triggers" that store more of the metadata in
> more intelligent form, so that you could indeed pick and choose and
> even transform on the way through.
> 
> It has gotten steps further, conceptually; Jan, Simon and I had a chat
> at PGCon about a "wildly more intelligent COPY" that would get us the
> kinds of efficiencies that COPY does for I/U/D updates.  A key thing
> about that would be making it generally useful as a "Data Load Tool"
> (e.g. - presumably with the potential to be popular for data
> warehouse-like applications) so that it wouldn't hang as a
> Slony-I-only "bag on the side."  But implementation hasn't started,
> and *that* would be a logical prerequisite to filtering.



More information about the Slony1-general mailing list