Bill Moran wrote:
> In response to Jerry Sievers <jerry at jerrysievers.com>:
>
>   
>> Bill Moran <wmoran at collaborativefusion.com> writes:
>>
>>     
>>> In response to Jerry Sievers <jerry at jerrysievers.com>:
>>>
>>>       
>>>> Slony 1.1.5
>>>>
>>>> Assuming a cascaded cluster of 3 nodes arranged as follows;
>>>>
>>>> Master
>>>>         Slave1
>>>>                 Slave2
>>>>
>>>> If the middle tier Slave1 is totally lost, is there a way to get
>>>> Slave2 pointing to Master without data loss, or at all?
>>>>
>>>> I am unable to do it with Slonik but had thought this was workable. 
>>>>         
>>> What, exactly, did you try to do?
>>>
>>> You should be able to do it by simply issuing a new subscribe set ()
>>> command with the correct parameters to subscribe it to Master.
>>>       
>> That was my assumption and I guess also believe that Slonik would feed
>> the new subscription info directly to the now cut=-off slave. .  This
>> does not seem to be the case.
>>
>> original subscription looked like 
>>
>> set 1,
>> provider 2,
>> receiver 3
>>
>> New subscription;
>>
>> set 1,
>> provider 1, 
>> receiver 3
>>
>> But with the node that was in the middle feeding slave 2 down, I think
>> there's no way slave 2 ever hears of the new subscription and it keeps
>> trying to connect to the slave that used to be feeding it.
>>
>> I've queried the sl_* tables on the  node that I want to reposition
>> and see that nothing is  changing.
>>
>> Any ideas?
>>     
>
> This may come across harsh.  I hope not, but I fear that there's no way
> to describe this without sounding harsh.
>
> If you tell us _what_you_did_ instead of _describing_what_you_think_
> you_did_, it's much more likely that people on the list will be able
> to help.
>
> This is a common problem.  Lots of people say, "I entered the correct
> commands", but refuse (for some unknown reason) to simply cut/paste
> the commands into their email.
>
> Unfortunately, this strategy comes up against the fact that 99% of the
> problems that people have are operator error, and usually boil down to
> a very specific misuse of a command.  Without knowing exactly what those
> commands are, we'll never find the problem.
>
>   
Well put.
> That being said, I'm not sure that this is _supposed_ to work.  It' may
> be necessary to "DROP NODE" of node 2 in order for Slony to understand
> that it's no longer participating.  Haven't tried this personally, but
> Slony should automatically reshape paths when node 2 is officially removed.
>
> Of course, someone who knows better may correct me on this.
>   
Dropping out node #2 will actually NOT work, because node #2 is still 
regarded as data provider for node #3 in some places.

I would expect it to be useful to submit STORE PATH entries for the 
paths that are still functioning, as that would "bump" the remaining 
nodes, telling them all to regenerate listen path (sl_listen) entries.

I think the first thing I'd try to do is to drop whatever PATH 
references to node #2 that I could, and add all the valid ones that I can.

Here are all the possible ones:

  DROP PATH ( SERVER = 1, CLIENT = 2 );
  DROP PATH ( SERVER = 2, CLIENT = 1 );
  DROP PATH ( SERVER = 3, CLIENT = 2 );
  DROP PATH ( SERVER = 2, CLIENT = 3 );

By default, those will be injected at node #1; submitting them to node 
#3 would be a useful alternative:

  DROP PATH ( SERVER = 1, CLIENT = 2, EVENT NODE = 3 );
  DROP PATH ( SERVER = 2, CLIENT = 1, EVENT NODE = 3 );
  DROP PATH ( SERVER = 3, CLIENT = 2, EVENT NODE = 3 );
  DROP PATH ( SERVER = 2, CLIENT = 3, EVENT NODE = 3 );

I expect that some of those would fail due to appearing to cut off 
communications; all the same, the more of them that *do* go away, the 
likelier that useful communications will be re-established between nodes 
1 and 3.

Supposing I have a preamble (e.g. - CLUSTER NAME and ADMIN CONNINFO 
statements) in the file /tmp/preamble.slonik, then the following shell 
script could do the trick:

#!/bin/sh
PREAMBLE=`cat /tmp/preamble.slonik`
for eventnode in 1 3; do
    for server in 1 2 3; do
    for client in 1 2 3; do
        echo "${PREAMBLE}" > /tmp/slonik-script.$$
        echo "drop path (server = ${server}, client = ${client}, event 
node = ${eventnode});" | grep 2 >> /tmp/slonik-script.$$
        slonik /tmp/slonik-script.$$
     done
     done
done

That'll remove all the paths it can, initially, which will trigger 
rebuilding of listen paths.

Submit STORE PATH requests for all the non-related-to-node-2 paths and 
that may also do good.

Restarting the slons may also help.

This is all a bit messy, but the situation is, if anything, that...