During the initial copy, the master database had a problem and died.  Fair enough, usually Slony will try again.  But this time, it has gotten itself into an invalid state -- see the attached log dump.

I have already copied a LOT of data, and the connection between master and slave isn't reliable, so I don't want to start over.  I REALLY need to get this system replicated SOON.  How can I repair this problem and get Slony to continue?  For the moment, I've killed the slon(1) daemons, but if I restart them, the "sequence ID 51 has already been assigned" message just keeps getting repeated.

Thanks,
Craig


NOTICE:  truncate of "emol_warehouse_1"."sample" succeeded
2008-04-04 00:29:30 PDT DEBUG1 cleanupThread:    0.002 seconds for cleanupEvent()
2008-04-04 01:05:33 PDT ERROR  remoteWorkerThread_1: "select SL.seql_seqid, SL.seql_last_value,     "_emol_warehouse_1_cluster".slon_quote_brute(PGN.nspname) || '.' ||     "\
_emol_warehouse_1_cluster".slon_quote_brute(PGC.relname) as tab_fqname     from "_emol_warehouse_1_cluster".sl_sequence SQ, "_emol_warehouse_1_cluster".sl_seqlog SL,        \
      "pg_catalog".pg_class PGC,              "pg_catalog".pg_namespace PGN   where SQ.seq_set = 5            and SL.seql_seqid = SQ.seq_id           and SL.seql_ev_seqno = \
'43'             and PGC.oid = SQ.seq_reloid             and PGN.oid = PGC.relnamespace; " server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
2008-04-04 01:05:33 PDT WARN   remoteWorkerThread_1: data copy for set 5 failed - sleep 15 seconds
2008-04-04 01:05:34 PDT DEBUG1 cleanupThread: 2163.722 seconds for delete logs
2008-04-04 01:05:49 PDT DEBUG1 copy_set 5
2008-04-04 01:05:49 PDT DEBUG1 remoteWorkerThread_1: connected to provider DB
NOTICE:  truncate of "emol_warehouse_1"."sample" succeeded
2008-04-04 01:16:52 PDT DEBUG1 cleanupThread:    0.016 seconds for cleanupEvent()
2008-04-04 01:16:52 PDT DEBUG1 cleanupThread:    0.032 seconds for delete logs
2008-04-04 01:28:43 PDT DEBUG1 cleanupThread:    0.001 seconds for cleanupEvent()
2008-04-04 01:28:43 PDT DEBUG1 cleanupThread:    0.002 seconds for delete logs
2008-04-04 01:40:45 PDT DEBUG1 cleanupThread:    0.002 seconds for cleanupEvent()
2008-04-04 01:40:45 PDT DEBUG1 cleanupThread:    0.014 seconds for delete logs
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrup\
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2008-04-04 01:43:59 PDT ERROR  remoteListenThread_1: "select ev_origin, ev_seqno, ev_timestamp,        ev_minxid, ev_maxxid, ev_xip,        ev_type,        ev_data1, ev_data2,     \
   ev_data3, ev_data4,        ev_data5, ev_data6,        ev_data7, ev_data8 from "_emol_warehouse_1_cluster".sl_event e where (e.ev_origin = '1' and e.ev_seqno > '1012') order by e\
.ev_origin, e.ev_seqno" - server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
2008-04-04 01:44:09 PDT DEBUG1 remoteListenThread_1: connected to 'dbname=emol_warehouse_1 host=aurum user=postgres port=5435'
NOTICE:  Slony-I: cleanup stale sl_nodelock entry for pid=20890
CONTEXT:  SQL statement "SELECT  "_emol_warehouse_1_cluster".cleanupNodelock()"
PL/pgSQL function "cleanupevent" line 77 at PERFORM
2008-04-04 01:51:32 PDT DEBUG1 cleanupThread:    0.001 seconds for cleanupEvent()
2008-04-04 01:51:32 PDT DEBUG1 cleanupThread:    0.002 seconds for delete logs
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted sha\
red memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2008-04-04 01:52:35 PDT ERROR  remoteWorkerThread_1: "select SL.seql_seqid, SL.seql_last_value,     "_emol_warehouse_1_cluster".slon_quote_brute(PGN.nspname) || '.' ||     "_emol_w\
arehouse_1_cluster".slon_quote_brute(PGC.relname) as tab_fqname     from "_emol_warehouse_1_cluster".sl_sequence SQ, "_emol_warehouse_1_cluster".sl_seqlog SL,              "pg_cata\
log".pg_class PGC,              "pg_catalog".pg_namespace PGN   where SQ.seq_set = 5            and SL.seql_seqid = SQ.seq_id           and SL.seql_ev_seqno = '43'             and \
PGC.oid = SQ.seq_reloid             and PGN.oid = PGC.relnamespace; " server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
2008-04-04 01:52:35 PDT WARN   remoteWorkerThread_1: data copy for set 5 failed - sleep 30 seconds
NOTICE:  there is no transaction in progress
2008-04-04 01:53:05 PDT DEBUG1 copy_set 5
2008-04-04 01:53:05 PDT DEBUG1 remoteWorkerThread_1: connected to provider DB
2008-04-04 01:53:06 PDT ERROR  remoteWorkerThread_1: "select "_emol_warehouse_1_cluster".setAddSequence_int(5, 51, '"emol_warehouse_1"."sample_id_seq"', 'emol_warehouse_1.sample_id\
_seq')" PGRES_FATAL_ERROR ERROR:  Slony-I: setAddSequence_int(): sequence ID 51 has already been assigned
2008-04-04 01:53:06 PDT WARN   remoteWorkerThread_1: data copy for set 5 failed - sleep 60 seconds
NOTICE:  there is no transaction in progress
2008-04-04 01:54:06 PDT DEBUG1 copy_set 5
2008-04-04 01:54:06 PDT DEBUG1 remoteWorkerThread_1: connected to provider DB
2008-04-04 01:54:07 PDT ERROR  remoteWorkerThread_1: "select "_emol_warehouse_1_cluster".setAddSequence_int(5, 51, '"emol_warehouse_1"."sample_id_seq"', 'emol_warehouse_1.sample_id\
_seq')" PGRES_FATAL_ERROR ERROR:  Slony-I: setAddSequence_int(): sequence ID 51 has already been assigned
2008-04-04 01:54:07 PDT WARN   remoteWorkerThread_1: data copy for set 5 failed - sleep 60 seconds
NOTICE:  there is no transaction in progress
2008-04-04 01:55:07 PDT DEBUG1 copy_set 5
2008-04-04 01:55:07 PDT DEBUG1 remoteWorkerThread_1: connected to provider DB
2008-04-04 01:55:07 PDT ERROR  remoteWorkerThread_1: "select "_emol_warehouse_1_cluster".setAddSequence_int(5, 51, '"emol_warehouse_1"."sample_id_seq"', 'emol_warehouse_1.sample_id\
_seq')" PGRES_FATAL_ERROR ERROR:  Slony-I: setAddSequence_int(): sequence ID 51 has already been assigned
2008-04-04 01:55:07 PDT WARN   remoteWorkerThread_1: data copy for set 5 failed - sleep 60 seconds
NOTICE:  there is no transaction in progress