Andrew Sullivan <ajs at crankycanuck.ca> writes:
> On Sat, Jan 29, 2005 at 10:35:17AM +0000, John Sidney-Woollett wrote:
>> 
>> I think that the slon_watchdog2 script queries the database to see 
>> whether the node is working properly - is this the best approach?
>
> It's a good approach in that it checks not only whether the daemon
> is listening, but also whether it's actually working.  The latter
> question is not a trivial one -- if something is wrong, you want to
> know it sooner rather than later.  The watchdog also logs in a way
> designed to be convenient for Nagios.  Chris can provide more
> details.

It's not all quite up to what managerial types might want to term
"best practices."  (I at some point should go through the
documentation to see about making sure that places where the term
might apply, it gets used, so as to make life easier for those Gentle
Readers who have PHBs that get all excited about that...)

The watchdog _isn't_ particularly logging for Nagios; what is logging
for Nagios is the separate "replication test" script,
test_slony_replication.pl.

But yes, the way slon_watchdog2 queries the DB to see if the node is
working is certainly pointing towards "best practices."  

- It isn't sufficient to simply see if the slon process is there, as
  some threads may shut down without the whole thing dying

- It isn't sufficient to simply see if events are propagating, as if a
  big COPY_SET is under way, that represents One Big Event that might 
  run for 28 hours if it's a big replication set.

Thus, whether a particular slon is "doing OK" seems best measured by
the combination of:

1. If it's not running ===> Problem
2. If it's running, and events are making it through ===> OK
3. If it's running, and a COPY_SET is in progress ====> Treat as OK

4. What's left over:
      -> It's running (#1 wasn't the case)
      -> It's not propagating events (#2 wasn't the case)
      -> There's no COPY_SET, so there's no good excuse for not
         propagating events

   Which implies something's broken.  A specific scenario that we have
   run into is where a node is running on a remote site via a WAN, and
   the connectivity is a little flakey such that DB connections
   sometimes silently die.

It may be that there are further problem scenarios worth pointing out,
although the above logic has been pretty successful thus far.  Not so
much for telling Nagios anything, but rather for indicating policy
where we tell slon processes to restart themselves.
  
There are a set of queries in <tools/test_slony_state.pl> (see CVS
HEAD) which rummage thru the state of each node.  It seems to me that
THAT is the way to get to further Nagios tests.  

 - Some of the tests check to see if sl_listen configuration seems
   OK/is broken

 - Other tests look for growth of tables like sl_log_1, pg_listener

 - Others look to see if some nodes aren't confirming results

 - Still others look at whether there seem to be long running
   transactions on some nodes.

These all represent sorts of problems that are at least somewhat
orthogonal to whether slon is running.

What is unfortunate is that Nagios has rather limited capabilities to
cope with the possibility that multiple things are broken.  But given
that that is a possible problem...

It seems to me that the "Best Practices" would be to add the tests
from the watchdog as well as some tests based on test_slony_state.pl
in to improve the replication test.  

At present, the replication test requires adding in a custom table,
and does quite a bit of work surrounding pushing thru some largely
useless updates thru that table.  If sl_event/sl_confirm were more
extensively tested, that would probably make the use of the table
"slony_test" unnecessary.
-- 
"cbbrowne","@","ca.afilias.info"
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 673-4124 (land)