Steve, I've also noticed something even more bizarre.<div><br></div><div>Fortunately, I have log_shipping set up on a third subscriber, and grepping through the files, I find the following:</div><div><br></div><div><div>
[root@asp-dbr1 log_staging]# for i in `ls`; do grep -Hn cron_lock $i; done </div><div>slony1_log_3_00000000000011179582.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div>slony1_log_3_00000000000011180903.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div>slony1_log_3_00000000000011181730.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div>slony1_log_3_00000000000011182005.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div>slony1_log_3_00000000000011182466.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div>slony1_log_3_00000000000011183270.sql:13:insert into "public"."cron_lock" ("id","lock_until_time") values ('anonymous_marking','2011-05-11 11:40:02.456091');</div>
<div><br></div><div><div>[root@asp-dbr1 log_staging]# diff slony1_log_3_00000000000011179582.sql slony1_log_3_00000000000011180903.sql</div><div>7c7</div><div>< select "_sac_uk".archiveTracking_offline('11179582', '2011-05-11 11:33:39.97637');</div>
<div>---</div><div>> select "_sac_uk".archiveTracking_offline('11180903', '2011-05-11 11:46:11.544073');</div></div><div><br></div>It seems that something has gone horribly wrong with replication such that it's beginning to even replay events.</div>
<div><br></div><div>Does this look like an issue you may already be aware of?</div><div>--Richard</div><div><br></div><div><br></div><div><br><div class="gmail_quote">On Wed, May 11, 2011 at 2:43 PM, Richard Yen <span dir="ltr"><<a href="mailto:dba@richyen.com">dba@richyen.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br><br><div class="gmail_quote"><div class="im">On Wed, May 11, 2011 at 2:26 PM, Steve Singer <span dir="ltr"><<a href="mailto:ssinger@ca.afilias.info" target="_blank">ssinger@ca.afilias.info</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 11-05-11 04:29 PM, Richard Yen wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks Steve. I've put the dumps for the master and my two slaves at<br>
<a href="http://richyen.com/slony/" target="_blank">http://richyen.com/slony/</a><br>
</blockquote>
<br></div>
As Jan suspected there isn't really anything still left in the logs about events from before the failed insert.<br>
<br>
When should the row have been deleted from cron_lock? immediately before or sometime before?<br></blockquote></div><div>The cron runs every hour, so there should have been a DELETE at 10:40AM, just milliseconds before the INSERT.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Did anything else happen around this time? (server restarts, moving masters etc?)<br></blockquote></div><div>No, we didn't do anything during this time. The previous action I performed on this cluster was at 10:04AM, which was to INSERT a row into another table in the database. The sync_interval I have is 500ms, so I'm fairly confident that this would have been processed long before the offending INSERT to cron_lock arrived at 10:40AM. The oldest entry in sl_even right now is '2011-05-11 10:40:02.354653'</div>
<div><br></div><div>If we can't get any more clues, I'll go ahead an process the DELETE. Thanks again for your help in trying to get to the bottom of this.</div><div>--Richard</div></div>
</blockquote></div><br></div>