>
> This is do-able, but it feels mighty clumsy, and even more fragile.
>
> It seems really tempting to think about having a "virtual  
> replication set" for this, and putting that into Slony-I proper.
>
> The notion of the "virtual replication set" still seems pretty clumsy.

Not too sure about hiding it. It is very fragile if it breaks mid-way  
through. If it is hidden it will be difficult to fix, unless you  
think you can make it continue when it finds a virtual set partially  
subscribed. Good transaction boundaries leave the number of possible  
broken states at a small enough count.

> Here's an outline of a very different approach that Jan and I have  
> been talking about that we call "COPY pipelining." It ought to help  
> by parallelizing the data load. It's in the TODO...

Fine for small to mid sized databases. If you're using table  
partitioning because of the size of the structures, and have lots of  
partitions, this isn't going to do much to eliminate the large  
transaction problem.

Granted, it would be very nice if subscribes went a little quicker :)

> ======================================================
> COPY pipelining
>
> - the notion here is to try to parallelize the data load at
> SUBSCRIBE time. Suppose we decide we can process 4 tables at a
> time, we set up 4 threads. We then iterate thus:
>
> For each table
> - acquire a thread (waiting as needed)
> - submit COPY TO stdout to the provider, and feed to
> COPY FROM stdin on the subscriber
> - Submit the REINDEX request on the subscriber
>
> Even with a fairly small number of threads, we should be able to
> process the whole subscription in as long as it takes to process
> the single largest table.
>
> This introduces a risk of locking problems not true at present
> (alas) in that, at present, the subscription process is able to
> demand exclusive locks on all tables up front; that is no longer
> possible if the subscriptions are split across multiple tables.
> In addition, the updates will COMMIT across some period of time on
> the subscriber rather than appearing at one instant in time.
>
> The timing improvement is probably still worthwhile.
>
> http://lists.slony.info/pipermail/slony1-hackers/2007-April/ 
> 000000.html
> ======================================================
>
> _______________________________________________
> Slony1-general mailing list
> Slony1-general at lists.slony.info
> http://lists.slony.info/mailman/listinfo/slony1-general