<div dir="ltr"><div><div>Jan,<br><br></div>You are right. Tweaking the tcp keep alive parameters helped.<br><br></div>Now slon.conf contains:<br><div><pre class="">tcp_keepalive=<span class="">true</span>
tcp_keepalive_interval=45
tcp_keepalive_count=20
tcp_keepalive_idle=30
cleanup_interval=30<br><br></pre><pre class="">Thanks a lot for the timely response.<br><br>- Sridevi<br></pre></div><div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 13, 2013 at 11:46 PM, Jan Wieck <span dir="ltr"><<a href="mailto:JanWieck@yahoo.com" target="_blank">JanWieck@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="im">On 06/13/13 06:25, Sridevi R wrote:<br>
> Hello Jan,<br>
><br>
> The Master and Slave DBs talk through a firewall.<br>
> VIP IPs and SNAT IPs are used in pg_hba.conf.<br>
><br>
> The corresponding messages in the postgres server log:<br>
><br>
> 2013-06-13 09:46:21.224 GMT,,,6630,"<a href="http://10.4.2.2:42031" target="_blank">10.4.2.2:42031</a><br>
</div>> <<a href="http://10.4.2.2:42031" target="_blank">http://10.4.2.2:42031</a>>",51b994ed.19e6,1,"",2013-06-13 09:46:21<br>
<div><div class="h5">> GMT,,0,LOG,08P01,"incomplete startup packet",,,,,,,,,""<br>
> 2013-06-13 09:57:38.596 GMT,"postgres","db01",6634,"<ip address printed<br>
> here>:53924",51b994f7.19ea,1,"idle",2013-06-13 09:46:31<br>
> GMT,28/0,0,LOG,08006,"could not receive data from client: Connection<br>
> reset by peer",,,,,,,,,"slon.node_1_listen"<br>
> 2013-06-13 09:57:38.596 GMT,"postgres","db01",6634,"<ip address printed<br>
> here>:53924",51b994f7.19ea,2,"idle",2013-06-13 09:46:31<br>
> GMT,28/0,0,LOG,08P01,"unexpected EOF on client<br>
> connection",,,,,,,,,"slon.node_1_listen"<br>
> 2013-06-13 09:57:38.607 GMT,"postgres","db01",6637,"<ip address printed<br>
> here>:53926",51b994f9.19ed,1,"idle",2013-06-13 09:46:33<br>
> GMT,32/0,0,LOG,08006,"could not receive data from client: Connection<br>
> reset by peer",,,,,,,,,"slon.subscriber_1_provider_1"<br>
> 2013-06-13 09:57:38.607 GMT,"postgres","db01",6637,"<ip address printed<br>
> here>:53926",51b994f9.19ed,2,"idle",2013-06-13 09:46:33<br>
> GMT,32/0,0,LOG,08P01,"unexpected EOF on client<br>
> connection",,,,,,,,,"slon.subscriber_1_provider_1"<br>
> 2013-06-13 09:57:38.608 GMT,"postgres","db01",6635,"<ip address printed<br>
> here>:53925",51b994f7.19eb,1,"idle",2013-06-13 09:46:31<br>
> GMT,31/0,0,LOG,08006,"could not receive data from client: Connection<br>
> reset by peer",,,,,,,,,"slon.node_1_listen"<br>
> 2013-06-13 09:57:38.608 GMT,"postgres","db01",6635,"<ip address printed<br>
> here>:53925",51b994f7.19eb,2,"idle",2013-06-13 09:46:31<br>
> GMT,31/0,0,LOG,08P01,"unexpected EOF on client<br>
> connection",,,,,,,,,"slon.node_1_listen"<br>
><br>
> The client slon log contains:<br>
> 2013-06-13 09:57:38 GMT FATAL cleanupThread: "begin;lock table<br>
> "_xx_cluster".sl_config_lock;select "_xx_cluster".cleanupEvent('10<br>
> minutes'::interval);commit;" - server closed the connection unexpectedly<br>
> This probably means the server terminated abnormally<br>
> before or while processing the request.<br>
<br>
</div></div>This all can very well be a slightly too eager firewall dropping idle<br>
connections. Have you tried to enable TCP keep alive options that kick<br>
in after something like 30 seconds? If not, enable them on both, the PG<br>
server and the Slony side. That usually prevents those firewall issues.<br>
<br>
<br>
Jan<br>
<div class="im"><br>
<br>
><br>
><br>
> Thanks,<br>
> Sridevi<br>
><br>
><br>
><br>
><br>
><br>
> On Thu, Jun 13, 2013 at 12:02 AM, Jan Wieck <<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a>>> wrote:<br>
><br>
> On 06/12/13 10:17, Sridevi R wrote:<br>
> > Jan,<br>
> ><br>
> > Thanks for the reply.<br>
> ><br>
> > The only errors in the slon log are failure of cleanupThread.<br>
> > child process is restarting right after the cleanupThread Failure.<br>
> > This occurs approximately every 10 minutes since cleanup_interval<br>
> is set<br>
> > to 10 minutes.<br>
> ><br>
> > Here is a sample from the log again:<br>
> ><br>
> > 2013-06-06 14:23:27 GMT FATAL cleanupThread: "begin;lock table<br>
> > "_xx_cluster".sl_config_lock;select "_xx_cluster".cleanupEvent('10<br>
> > minutes'::interval);commit;" - server closed the connection<br>
> unexpectedly<br>
> > This probably means the server terminated abnormally<br>
> > before or while processing the request.<br>
> > 2013-06-06 14:23:27 GMT CONFIG slon: child terminated signal: 9; pid:<br>
> > 16135, current worker pid: 16135<br>
> > 2013-06-06 14:23:27 GMT CONFIG slon: restart of worker in 10 seconds<br>
><br>
> "server closed the connection unexpectedly" ...<br>
><br>
> Is this connection by any chance through some firewall or NAT gateway<br>
> that will drop idle connections?<br>
><br>
> What are the corresponding postmaster server log entries? Since slony<br>
> reports an unexpected connection drop from the server, the server must<br>
> have some message in its log too, because the client never sent the 'X'<br>
> libpq protocol message.<br>
><br>
><br>
> Jan<br>
><br>
><br>
> ><br>
> > Thanks ,<br>
> > Sridevi<br>
> ><br>
> ><br>
> > On Wed, Jun 12, 2013 at 7:33 PM, Jan Wieck <<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a><br>
> <mailto:<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a>><br>
</div></div><div><div class="h5">> > <mailto:<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a> <mailto:<a href="mailto:JanWieck@yahoo.com">JanWieck@yahoo.com</a>>>> wrote:<br>
> ><br>
> > On 06/12/13 07:14, Sridevi R wrote:<br>
> > > Hello,<br>
> > ><br>
> > > The slony logs are consistently posting this error:<br>
> > ><br>
> > > 2013-06-12 10:01:05 GMT FATAL cleanupThread: "begin;lock table<br>
> > > "_xx_cluster".sl_config_lock;select<br>
> "_xx_cluster".cleanupEvent('10<br>
> > > minutes'::interval);commit;" - server closed the connection<br>
> > unexpectedly<br>
> > > 2013-06-12 10:12:24 GMT FATAL cleanupThread: "begin;lock table<br>
> > > "_xx_cluster".sl_config_lock;select<br>
> "_xx_cluster".cleanupEvent('10<br>
> > > minutes'::interval);commit;" - server closed the connection<br>
> > unexpectedly<br>
> > ><br>
> > > checked and found that sl_confirm table is not cleaned up.<br>
> cleanup<br>
> > event<br>
> > > never succeeds.<br>
> > > Additionally, the child processes terminates and restarts<br>
> after each<br>
> > > such cleanup failure.<br>
> > ><br>
> > > 2013-06-11 11:20:04 GMT CONFIG slon: child terminated<br>
> signal: 9; pid:<br>
> > > 20172, current worker pid: 20172<br>
> > > 2013-06-11 11:20:04 GMT CONFIG slon: restart of worker in 10<br>
> seconds<br>
> > ><br>
> > > When cleanup is run manually, on the psql prompt it runs to<br>
> completion<br>
> > > without any issues and cleans up sl_event and sl_confirm tables<br>
> > > "begin;lock table "_xx_cluster".sl_config_lock;select<br>
> > > "_xx_cluster".cleanupEvent('10 minutes'::interval);commit;"<br>
> > ><br>
> > > Soln version: 2.1.2<br>
> > ><br>
> > > Any help/insight would be greatly appreciated.<br>
> ><br>
> > Slon kills its worker(s) with signal 9 (SIGKILL) when it needs to<br>
> > restart, like when there are errors in event processing or if it<br>
> > receives certain signals. Are there any other errors in the<br>
> slon log or<br>
> > is something on the machine sending signals to slon?<br>
> ><br>
> ><br>
> > Jan<br>
> ><br>
> > ><br>
> > > Thanks,<br>
> > > Sridevi<br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Slony1-general mailing list<br>
> > > <a href="mailto:Slony1-general@lists.slony.info">Slony1-general@lists.slony.info</a><br>
> <mailto:<a href="mailto:Slony1-general@lists.slony.info">Slony1-general@lists.slony.info</a>><br>
</div></div>> > <mailto:<a href="mailto:Slony1-general@lists.slony.info">Slony1-general@lists.slony.info</a><br>
<div class=""><div class="h5">> <mailto:<a href="mailto:Slony1-general@lists.slony.info">Slony1-general@lists.slony.info</a>>><br>
> > > <a href="http://lists.slony.info/mailman/listinfo/slony1-general" target="_blank">http://lists.slony.info/mailman/listinfo/slony1-general</a><br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Anyone who trades liberty for security deserves neither<br>
> > liberty nor security. -- Benjamin Franklin<br>
> ><br>
> ><br>
><br>
><br>
> --<br>
> Anyone who trades liberty for security deserves neither<br>
> liberty nor security. -- Benjamin Franklin<br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Slony1-general mailing list<br>
> <a href="mailto:Slony1-general@lists.slony.info">Slony1-general@lists.slony.info</a><br>
> <a href="http://lists.slony.info/mailman/listinfo/slony1-general" target="_blank">http://lists.slony.info/mailman/listinfo/slony1-general</a><br>
><br>
<br>
<br>
--<br>
Anyone who trades liberty for security deserves neither<br>
liberty nor security. -- Benjamin Franklin<br>
</div></div></blockquote></div><br></div></div></div></div></div>