On Sat, Sep 10, 2016 at 7:57 PM, Steve Singer <steve at ssinger.info> wrote:

> On Sat, 10 Sep 2016, Tory M Blue wrote:
>
>
>> Running into an issue, working with a cluster on another network, so I
>> added node 11 (insert secondary in site2) from
>> node 2 (which is the insert secondary of site 1). Set 2 kept failing, so
>> I went in and dropped node 11 from the
>> cluster config, but node 2, which is the seecondary insert server
>> continues with this error.. I've not inserted
>> anything into node 2, nor can you, I'm really confused how my primary
>> cluster got in this state..
>>
>>
>> Any assistance to help clean it up? Dropping node 2 and re-adding is a 24
>> hour ordeal.
>>
>> 2016-09-10 17:41:50 PDT ERROR  remoteWorkerThread_11: "insert into
>> "_cls".sl_event     (ev_origin, ev_seqno,
>> ev_timestamp,      ev_snapshot, ev_type     ) values ('11', '5000005698',
>> '2016-09-10 08:34:48.691123-07',
>> '6422065:6530984:6422065', 'SYNC'); insert into "_cls".sl_confirm
>> (con_origin, con_received, con_seqno,
>> con_timestamp)    values (11, 2, '5000005698', now()); select
>> "_cls".logApplySaveStats('_cls', 11, '0.047
>> s'::interval); commit transaction;" PGRES_FATAL_ERROR ERROR:  duplicate
>> key value violates unique constraint
>> "sl_event-pkey"
>>
>> DETAIL:  Key (ev_origin, ev_seqno)=(11, 5000005698) already exists.
>>
>>
> So when slon 2 (I assume this is slon 2) is stopped, is/was there a row in
> sl_event with ev_origin=11 and ev_seqno=5000005698
>
> ?
>
> You also didn't you say when things were failing before dropping the node.
>
>
>
>> thanks
>>
>> Tory
>>
>>
>>
>> Sorry Steve

Slon stopped functioning on node 2, and thus that error. Got the normal,
will retry in 10 seconds. Meaning slon was in a bad spot, and typically I
see that when it's not initialized, however that wasn't the case and I got
the error I included 100s of times with the following, will retry in 10
seconds.

What I did was manually removed all node 11 information , since the
dropnode that I ran earlier seemed to make slon think that node 11 was not
configured, yet all the tables , sl_listen:subscribe, path, node etc was
still there on all 5 nodes in the primary cluster.

I'm fully replicated again, and did not have to drop/add node 2 (whew).
Just took some manual tweaking inside the slony schema.

will try to add the node again later to see what happens...

What I'm attempting is

Site 1:
Nodes 1-5
Nodes 1-2 are insert  (obviously primary/secondary)
Nodesl 3-5 are query only

Site 2
Nodes 11-15
Nodes 11-12 are insert (initially will be secondary/secondary, until we
switch from site 1 to site 2)
Nodes 13-15 are query only.

I'm attempting to (which we have done in the past), is keep the load off
node 1 (primary insert DB) and have node 2 handle replication (provide) to
Node 11 (only this step at first).

I will then add Node 12 from node 11 and nodes 13-15 to grab their data
from node 12.

I somehow hacked my way out of the issue :) YAY! But am worried that I
can't replicate to node 11 yet, and while it's wide area network, the delay
is a whopping 2ms and I've dropped all the big indexes so that slon
/network is not waiting for those to complete..

Will update later, but it was odd this host got in this state, weird that
the dropnode failed (it's possible because it was still trying to complete
the initial addnode and thus was busy with loading all the tables, but I
figured that a drop notice would stop all types of replication events, but
it does not. Stopping slon even with the drop waiting on node 11, node 11
came back and instantly started truncating tables again and attempting to
replicate, ignoring the recent Drop instructions.

Thanks
Tory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.slony.info/pipermail/slony1-general/attachments/20160910/c5a58a76/attachment-0001.htm