<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 10, 2016 at 5:50 PM, Tory M Blue <span dir="ltr"><<a href="mailto:tmblue@gmail.com" target="_blank">tmblue@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 10, 2016 at 5:44 PM, Tory M Blue <span dir="ltr"><<a href="mailto:tmblue@gmail.com" target="_blank">tmblue@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra">
<p><span>Running into an issue, working with a cluster on another network, so I added node 11 (insert secondary in site2) from node 2 (which is the insert secondary of site 1). Set 2 kept failing, so I went in and dropped node 11 from the cluster config, but node 2, which is the seecondary insert server continues with this error.. I've not inserted anything into node 2, nor can you, I'm really confused how my primary cluster got in this state..</span></p><p><span><br></span></p><p><span>Any assistance to help clean it up? Dropping node 2 and re-adding is a 24 hour ordeal.</span></p><p><span>2016-09-10 17:41:50 PDT ERROR remoteWorkerThread_11: "insert into "_cls".sl_event (ev_origin, ev_seqno, ev_timestamp, ev_snapshot, ev_type ) values ('11', '5000005698', '2016-09-10 08:34:48.691123-07', '6422065:6530984:6422065', 'SYNC'); insert into "_cls".sl_confirm <span> </span>(con_origin, con_received, con_seqno, con_timestamp) values (11, 2, '5000005698', now()); select "_cls".logApplySaveStats('_cls<wbr>', 11, '0.047 s'::interval); commit transaction;" PGRES_FATAL_ERROR ERROR: duplicate key value violates unique constraint "sl_event-pkey"</span></p>
<p><span>DETAIL: Key (ev_origin, ev_seqno)=(11, 5000005698) already exists.</span></p><p><span></span></p></div></div></blockquote></div><br></div><div class="gmail_extra">Also interesting node 2 seems to think node 11 is still legit</div><div class="gmail_extra"><br></div><div class="gmail_extra">
<p><span><a href="http://idb01.prod.ca" target="_blank">idb01.prod.ca</a> is node 11:</span></p><p><span><a href="http://idb01.prod.ca.domain.net" target="_blank">idb01.prod.ca.domain.net</a> is not active. Make sure that it is already part of the slon configuration currently running</span></p><p><span>
</span></p><p><span>select * from _cls.sl_node;</span></p><p><span> no_id | no_active | no_comment | no_failed </span></p><p><span>-------+-----------+----------<wbr>--+-----------</span></p><p><span> 3 | t | Node 3 | f</span></p><p><span> 4 | t | Node 4 | f</span></p><p><span> 5 | t | Node 5 | f</span></p><p><span> 1 | t | Node 1 | f</span></p><p><span> 2 | t | Node 2 | f</span></p><p>
</p><p><span> <b> 11 | t | Node 11 | f</b></span></p><p><span><br></span></p><p><span>Paths are there as well..</span></p><span class="HOEnZb"><font color="#888888"><p><br></p></font></span></div></div></blockquote><div>After manually removing node 11 from all the servers in site 1, node 2 slon started up on it's own and caught up. So not sure what is happening and why node 11 got into that state</div><div><br></div><div>Will look around and make sure that all hosts, know about all hosts</div><div><br></div><div>Thanks</div><div>Tory </div></div></div></div>