<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 13, 2019 at 7:59 AM Christopher Browne <<a href="mailto:cbbrowne@afilias.info">cbbrowne@afilias.info</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Jun 12, 2019 at 11:58 PM Tory M Blue <<a href="mailto:tmblue@gmail.com" target="_blank">tmblue@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>First postgres 9.x</div><div>CentOS 7.2</div><div>Replication flows:</div><div>primary ->secondary-> 3 query slaves</div><div><br></div><div>Okay here it is, </div><div><br></div><div>My dump started:</div><div><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971gmail-m_-1330120594842900310gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971gmail-m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971gmail-m_-1330120594842900310gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971gmail-m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">Start export of schema clsdb for user postgres at Wed Jun 12 19:00:48 PDT 2019</span></p></div><div><br></div><div>So if I'm reading this correct it's 20:45 but something is showing that the last _received_event_ts was at 19:38? over an hour ago?</div><div>sl_lag_time 1 hour?</div><div><br></div><div>Primary:</div><div>
<p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">select * from _cls.sl_status limit 3;</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_origin | st_received | st_last_event | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_event_ts<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_last_receive</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">d |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_ts<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_event_ts <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_lag_num_events |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">st_lag_time<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">-----------+-------------+---------------+-------------------------------+----------------</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">--+-------------------------------+-------------------------------+-------------------+---</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">--------------</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>14 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:11.065976-07 | 2019-06-12 19:38:03.311267-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>15 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:12.202539-07 | 2019-06-12 19:38:03.311267-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:08.342637-07 | 2019-06-12 19:38:03.311267-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
</p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">(3 rows)</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">I have 2.4 million rows in the sl_log_1 table, which looks like it started to hockey stick right around 19:30.</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p></div></div></div></blockquote><div><br></div><div>You're reading it pretty well right.</div><div><br></div><div>It is mighty curious that it waited 37-ish minutes for replication to fall off.</div><div><br></div><div>It would be interesting (and it's too late for today) to capture the lag between those times.</div><div><br></div><div>Thus, something like...</div><div><br></div><div>#!/bin/sh</div><div>PGDATABASE=postgresql://cbbrowne@localhost:7096/shared-sample </div><div>psql -d $PGDATABASE -c "create table if not exists lag_by_time (as_at timestamp default now(), replication_lag interval);"</div><div>for i in `seq 180`; do</div><div> psql -d $PGDATABASE -c "insert into lag_by_time (replication_lag) select min(st_lag_time) from _cls.sl_status;"</div><div> sleep 60<br></div><div>done</div><div><br></div><div>and launch that at about 19:00. At whatever point replication_lag heads towards the Stanley Cup, that's the time at which Something Happened.</div><div><br></div><div>It's probably interesting to look at pg_stat_activity to see what long-running queries there might be.</div><div><br></div><div>I'm somewhat suspicious that, despite what you were intending, perhaps the pg_dump is nevertheless capturing the schema _cls; it would be "sorta normal" if, at the point at which pg_dump starts dumping _cls material, replication starts lagging.</div><div><br></div><div>In particular, anything in pg_stat_activity that started at 19:38 ought to be looked at with particular suspicion as a candidate for being the cause.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">On the secondary (which replicates to my query db's (where the dump is run from (not run from the primary), it includes these lines</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">Secondary:</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>select * from _cls.sl_status limit 3;</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_origin | st_received | st_last_event | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_event_ts<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_la</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">st_received |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_ts<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_event_ts <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>| st</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">_lag_num_events | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>st_lag_time<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">-----------+-------------+---------------+-------------------------------+------</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">------------+-------------------------------+-------------------------------+---</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">----------------+-----------------</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650000 | 2019-06-12 19:38:03.815508-07 | 2019-06-12 19:38:03.338486-07 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>405 | 01:07:35.106539</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>13 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650403 | 2019-06-12 20:45:17.701455-07 | 2019-06-12 20:45:15.355233-07 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>2 | 00:00:23.089792</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>14 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650404 | 2019-06-12 20:45:26.642094-07 | 2019-06-12 20:45:25.360167-07 |<span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-Apple-converted-space"> </span>1 | 00:00:13.084858</span></p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)">
</p><p class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-m_-4005058105477411665gmail-m_-5652145762641767971m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">(3 rows)</span></p></div></div></div></blockquote><div><br></div><div>May I presume that the time you ran this query was around 20:45 or 20:46? That looks unsurprising/consistent with the first query; events going through node 11, which appears to be the master, are held back to 19:38. And the few events that go through coming from other nodes are lightly laggy, consistent with those nodes not generating SYNC events very often.<br></div></div></div></blockquote><div><br></div><div>You assume correctly. i ran the queries back to back, and yes node 11 is the primary/master and node 12 is the standby, with nodes 13-15 just query nodes and only a subset of the data, they are not available for failover/switchover, only nodes 11-12 are.</div><div><br></div><div>I saw nothing in the locks when I looked. We did notice that in our second dump, while we are using a -n table, we are not including a -N _slonschema,. Now that should work!! but we are going to add the -N _slonschema as well as the -n table, just to see. Will poke it some more this eve!</div><div><br></div><div>Thanks again</div><div>Tory </div></div></div>