<div class="gmail_quote">On Wed, May 2, 2012 at 2:39 PM, Steve Singer <span dir="ltr"><<a href="mailto:ssinger@ca.afilias.info" target="_blank">ssinger@ca.afilias.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">Are any of the above possible:</div>
<br>
1. You had multiple slon daemons writing to the same log archive directory (maybe for different clusters?)<br></blockquote><div>We have several clusters writing to a directory, but there's a separate directory for each cluster. For example:</div>
<div><br></div><div>/home/log_ship</div><div>/home/log_ship/cluster1/new_logfiles</div><div>/home/log_ship/cluster2/new_logfiles</div><div>/home/log_ship/cluster3/new_logfiles</div><div>...etc... </div><div> </div><div>We don't have two daemons writing to the same directory.</div>
<div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. The mechanism you used for copying the .sql files could have caused processes to try to write to the same file on the destination machine<br></blockquote><div>I'm fairly certain this is not the case. The files that I sent you were directly from the origin machine, not from the destination machine. Our scheme is like this:</div>
<div><br></div><div>Node1 is origin</div><div>Node2 is subscriber, with -a mode, writing files to /home/log_ship/cluster1/new_logfiles</div><div>Cronjob moves files from /home/log_ship/cluster1/new_logfiles to /home/log_ship/cluster1/log_staging (we filter out the *.sql.tmp files so that we can let them finish writing before we move them)</div>
<div>RemoteNode makes rsync connection to Node2 and copies the files from Node2/home/log_ship/cluster1/log_staging to its local directory</div><div>Log files are replayed</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If the answer to both of those is no then maybe there is a bug in how archive file numbers are assigned in remote_worker.c:archive_open.<br>
We don't YET see any obvious faults with this logic but if this logic somehow assigned 2 slon worker threads the same id then you could get a file like you sent us.<br></blockquote><div><br></div><div>As I look at the files you sent me, I only see differences between the third (Node X, Event XXXXXX) and seventh (archiveTracking_offline(xxx,'xxxx-xx-xx xx:xx:xx')) lines. I noticed that Node number can vary per file, but only one daemon has the -a option enabled. Not sure why the node number changes--shouldn't it always correspond to the node number of the daemon with the -a option turned on?</div>
<div><br></div><div>Aside from that, I tried poking around the sl_* tables that I had dumped, but didn't really find anything. One thing is certain, though--a given DML statement shows up in sl_log_x only once, even though it shows up several times in the various logship files that are generated. I can't seem to find the corresponding sl_event row, so I'm not sure if there might be anything in that direction, in terms of duplicated events.</div>
<div><br></div><div>--Richard</div></div>