<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 10, 2016 at 7:57 PM, Steve Singer <span dir="ltr"><<a href="mailto:steve@ssinger.info" target="_blank">steve@ssinger.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sat, 10 Sep 2016, Tory M Blue wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Running into an issue, working with a cluster on another network, so I added node 11 (insert secondary in site2) from<br>
node 2 (which is the insert secondary of site 1). Set 2 kept failing, so I went in and dropped node 11 from the<br>
cluster config, but node 2, which is the seecondary insert server continues with this error.. I've not inserted<br>
anything into node 2, nor can you, I'm really confused how my primary cluster got in this state..<br>
<br>
<br>
Any assistance to help clean it up? Dropping node 2 and re-adding is a 24 hour ordeal.<br>
<br>
2016-09-10 17:41:50 PDT ERROR remoteWorkerThread_11: "insert into "_cls".sl_event (ev_origin, ev_seqno,<br>
ev_timestamp, ev_snapshot, ev_type ) values ('11', '5000005698', '2016-09-10 08:34:48.691123-07',<br>
'6422065:6530984:6422065', 'SYNC'); insert into "_cls".sl_confirm (con_origin, con_received, con_seqno,<br>
con_timestamp) values (11, 2, '5000005698', now()); select "_cls".logApplySaveStats('_cls<wbr>', 11, '0.047<br>
s'::interval); commit transaction;" PGRES_FATAL_ERROR ERROR: duplicate key value violates unique constraint<br>
"sl_event-pkey"<br>
<br>
DETAIL: Key (ev_origin, ev_seqno)=(11, 5000005698) already exists.<br>
<br>
</blockquote>
<br>
So when slon 2 (I assume this is slon 2) is stopped, is/was there a row in sl_event with ev_origin=11 and ev_seqno=5000005698<br>
<br>
?<br>
<br>
You also didn't you say when things were failing before dropping the node.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
thanks<br>
<br>
Tory<br>
<br>
<br>
<br>
</blockquote>
</blockquote></div>Sorry Steve</div><div class="gmail_extra"><br></div><div class="gmail_extra">Slon stopped functioning on node 2, and thus that error. Got the normal, will retry in 10 seconds. Meaning slon was in a bad spot, and typically I see that when it's not initialized, however that wasn't the case and I got the error I included 100s of times with the following, will retry in 10 seconds.</div><div class="gmail_extra"><br></div><div class="gmail_extra">What I did was manually removed all node 11 information , since the dropnode that I ran earlier seemed to make slon think that node 11 was not configured, yet all the tables , sl_listen:subscribe, path, node etc was still there on all 5 nodes in the primary cluster.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I'm fully replicated again, and did not have to drop/add node 2 (whew). Just took some manual tweaking inside the slony schema.</div><div class="gmail_extra"><br></div><div class="gmail_extra">will try to add the node again later to see what happens... </div><div class="gmail_extra"><br></div><div class="gmail_extra">What I'm attempting is</div><div class="gmail_extra"><br></div><div class="gmail_extra">Site 1:</div><div class="gmail_extra">Nodes 1-5</div><div class="gmail_extra">Nodes 1-2 are insert (obviously primary/secondary)</div><div class="gmail_extra">Nodesl 3-5 are query only </div><div class="gmail_extra"><br></div><div class="gmail_extra">Site 2</div><div class="gmail_extra">Nodes 11-15</div><div class="gmail_extra">Nodes 11-12 are insert (initially will be secondary/secondary, until we switch from site 1 to site 2)</div><div class="gmail_extra">Nodes 13-15 are query only.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I'm attempting to (which we have done in the past), is keep the load off node 1 (primary insert DB) and have node 2 handle replication (provide) to Node 11 (only this step at first).</div><div class="gmail_extra"><br></div><div class="gmail_extra">I will then add Node 12 from node 11 and nodes 13-15 to grab their data from node 12.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I somehow hacked my way out of the issue :) YAY! But am worried that I can't replicate to node 11 yet, and while it's wide area network, the delay is a whopping 2ms and I've dropped all the big indexes so that slon /network is not waiting for those to complete..</div><div class="gmail_extra"><br></div><div class="gmail_extra">Will update later, but it was odd this host got in this state, weird that the dropnode failed (it's possible because it was still trying to complete the initial addnode and thus was busy with loading all the tables, but I figured that a drop notice would stop all types of replication events, but it does not. Stopping slon even with the drop waiting on node 11, node 11 came back and instantly started truncating tables again and attempting to replicate, ignoring the recent Drop instructions.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks</div><div class="gmail_extra">Tory</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div></div>