<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello slony group,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> I’m testing now with slony1-2.2.4. I have just recently produced an error which effectively stops slon processing on some node A due to some node B being dropped. The event reproduces only
infrequently. As some will know, a slon daemon for a given node which becomes aware its node has been dropped will respond by dropping its cluster schema. There appears to be a race condition between the node B schema drop and the (surviving) node A receipt
of the disableNode (drop node) event. If the former occurs before the latter, all the remote worker threads on node A enter an error state. See the log samples below. I resolved this the first time by deleting all the recent non-SYNC events from the sl_event
tables, and more recently with a simple node A slon restart.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Please advise if there is any ticket I should provide this info to, or if I should create a new one. Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- node 1 log ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:31 UTC [30382] INFO remoteWorkerThread_999999: SYNC 5000000008 done in 0.002 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:33 UTC [30382] INFO remoteWorkerThread_999999: SYNC 5000000009 done in 0.002 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:33 UTC [30382] INFO remoteWorkerThread_2: SYNC 5000017869 done in 0.002 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:33 UTC [30382] INFO remoteWorkerThread_3: SYNC 5000018148 done in 0.004 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] CONFIG remoteWorkerThread_2: update provider configuration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] ERROR remoteWorkerThread_3: "select last_value from "_ams_cluster".sl_log_status" PGRES_FATAL_ERROR ERROR: schema "_ams_clu\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ster" does not exist<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LINE 1: select last_value from "_ams_cluster".sl_log_status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] ERROR remoteWorkerThread_3: SYNC aborted<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] CONFIG version for "dbname=ams<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> host=198.18.102.45<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> user=ams_slony<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sslmode=verify-ca<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sslcert=/usr/local/akamai/.ams_certs/complete-ams_slony.crt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sslkey=/usr/local/akamai/.ams_certs/ams_slony.private_key<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sslrootcert=/usr/local/akamai/etc/ssl_ca/canonical_ca_roots.pem" is 90119<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] ERROR remoteWorkerThread_2: "select last_value from "_ams_cluster".sl_log_status" PGRES_FATAL_ERROR ERROR: schema "_ams_clu\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ster" does not exist<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LINE 1: select last_value from "_ams_cluster".sl_log_status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] ERROR remoteWorkerThread_2: SYNC aborted<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:45 UTC [30382] ERROR remoteListenThread_999999: "select ev_origin, ev_seqno, ev_timestamp, ev_snapshot, "pg_catalog".txid_sna\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">pshot_xmin(ev_snapshot), "pg_catalog".txid_snapshot_xmax(ev_snapshot), ev_type, ev_data1, ev_data2, ev_data3, ev_data4, ev\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">_data5, ev_data6, ev_data7, ev_data8 from "_ams_cluster".sl_event e where (e.ev_origin = '999999' and e.ev_seqno > '5000000009') or (e.ev_origin = '2'\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">and e.ev_seqno > '5000017870') or (e.ev_origin = '3' and e.ev_seqno > '5000018151') order by e.ev_origin, e.ev_seqno limit 40" - ERROR: schema "_ams_cluste\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">r" does not exist<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">LINE 1: ...v_data5, ev_data6, ev_data7, ev_data8 from "_ams_clus...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ^<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:55 UTC [30382] ERROR remoteWorkerThread_3: "start transaction; set enable_seqscan = off; set enable_indexscan = on; " PGRES_FATAL_ERROR ERR\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OR: current transaction is aborted, commands ignored until end of transaction block<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:55 UTC [30382] ERROR remoteWorkerThread_3: SYNC aborted<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:55 UTC [30382] ERROR remoteWorkerThread_2: "start transaction; set enable_seqscan = off; set enable_indexscan = on; " PGRES_FATAL_ERROR ERR\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OR: current transaction is aborted, commands ignored until end of transaction block<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:55 UTC [30382] ERROR remoteWorkerThread_2: SYNC aborted<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">----</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- node 999999 log ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:44 UTC [558] INFO remoteWorkerThread_1: SYNC 5000081216 done in 0.004 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:44 UTC [558] INFO remoteWorkerThread_2: SYNC 5000017870 done in 0.004 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:44 UTC [558] INFO remoteWorkerThread_3: SYNC 5000018150 done in 0.004 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:44 UTC [558] INFO remoteWorkerThread_1: SYNC 5000081217 done in 0.003 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-08 18:06:44 UTC [558] WARN remoteWorkerThread_3: got DROP NODE for local node ID<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NOTICE: Slony-I: Please drop schema "_ams_cluster"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NOTICE: drop cascades to 171 other objects<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">DETAIL: drop cascades to table _ams_cluster.sl_node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_nodelock<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_set<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_setsync<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_table<o:p></o:p></span></p>
<p class="MsoNormal">----</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Tom <span style="font-family:Wingdings">J</span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>