Mon May 9 16:19:32 PDT 2005
- Previous message: [Slony1-commit] By cbbrowne: More on the "saga of the duplicate key violation."
- Next message: [Slony1-commit] By cbbrowne: Changes to RPM spec file per Devrim Gunduz to address
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Log Message:
-----------
Added support for a new drop_indices option. When (as happens by
default) this is set to TRUE, the COPY_SET event will drop indices
from the destination node before purging table data and COPYing
in fresh data, and then recreates the indices after the COPY.
This is expected to provide a substantial performance improvement
for several reasons that add together:
1. It means the DELETE statement doesn't need to do any work on indices
2. It means the COPY statement doesn't touch the indices
3. Even if there are a lot of dead tuples in the table, only the live
ones wind up expressed in the indices that get regenerated.
The fact of recreating indices immediately after the COPY will have the
further salutory effect of making good use of the data already in the
cache, as the data copied in via COPY is likely to be in cache, so
no I/O will be necessary to get at it.
Modified Files:
--------------
slony1-engine/doc/adminguide:
bookindex.sgml (r1.7 -> r1.8)
slonconf.sgml (r1.4 -> r1.5)
slony1-engine/src/slon:
confoptions.h (r1.19 -> r1.20)
remote_worker.c (r1.81 -> r1.82)
-------------- next part --------------
Index: slonconf.sgml
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/doc/adminguide/slonconf.sgml,v
retrieving revision 1.4
retrieving revision 1.5
diff -Ldoc/adminguide/slonconf.sgml -Ldoc/adminguide/slonconf.sgml -u -w -r1.4 -r1.5
--- doc/adminguide/slonconf.sgml
+++ doc/adminguide/slonconf.sgml
@@ -114,6 +114,9 @@
</listitem>
</varlistentry>
+
+
+
<varlistentry id="slon-config-logging-log-timestamp-format" xreflabel="slon_conf_log_timestamp_format">
<term><varname>log_timestamp_format</varname> (<type>string</type>)</term>
<indexterm>
@@ -283,6 +286,33 @@
</para>
</listitem>
</varlistentry>
+ <varlistentry id="slon-drop-index" xreflabel="slon_drop_indices_for_copy">
+ <term><varname>drop_indices_for_copy</varname> (<type>boolean</type>)</term>
+ <indexterm>
+ <primary><varname>drop_indices_for_copy</varname> configuration parameter</primary>
+ </indexterm>
+ <listitem>
+ <para>Determines whether or not the COPY_SET event should drop
+ indices for each table while copying each table. It is much
+ faster to regenerate indices <quote>en masse</quote>
+ <emphasis>after</emphasis> loading the data than it is to have
+ them rebuilt incrementally as the <command>copy</command>
+ loads the data in.
+ </para>
+
+ <para> The default value is <quote>true</quote> at the time of
+ writing, in order that we get maximum testing out of it. If
+ you run into problems, set it to <quote>false</quote>.</para>
+
+ <note> <para> If this option is activated, the old <quote>rule
+ of thumb</quote> that you ought to drop all unneeded indices
+ on subscriber nodes in order to improve
+ <command>COPY_SET</command> performance becomes wrong. The
+ improved locality of reference to cached data reverses that
+ rule. </para></note>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect1>
</article>
Index: bookindex.sgml
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/doc/adminguide/bookindex.sgml,v
retrieving revision 1.7
retrieving revision 1.8
diff -Ldoc/adminguide/bookindex.sgml -Ldoc/adminguide/bookindex.sgml -u -w -r1.7 -r1.8
--- doc/adminguide/bookindex.sgml
+++ doc/adminguide/bookindex.sgml
@@ -25,13 +25,13 @@
<primaryie> DDL changes
</primaryie>
<secondaryie> changing the database schema ,
- <ulink url="ddlchanges.html" role="AEN1653">Database Schema Changes (DDL)</ulink>
+ <ulink url="ddlchanges.html" role="AEN1665">Database Schema Changes (DDL)</ulink>
</secondaryie>
</indexentry>
<indexentry>
<primaryie> Frequently Asked Questions about Slony-I,
- <ulink url="faq.html" role="AEN2415"></ulink>
+ <ulink url="faq.html" role="AEN2427"></ulink>
</primaryie>
</indexentry>
@@ -40,14 +40,14 @@
<indexentry>
<primaryie>cluster,
- <ulink url="concepts.html#AEN399" role="AEN401">Cluster</ulink>,
- <ulink url="cluster.html" role="AEN483">Defining Slony-I Clusters</ulink>
+ <ulink url="concepts.html#AEN398" role="AEN400">Cluster</ulink>,
+ <ulink url="cluster.html" role="AEN481">Defining Slony-I Clusters</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>cluster_name configuration parameter,
- <ulink url="slon-config-connection.html" role="AEN3529">Connection settings</ulink>
+ <ulink url="slon-config-connection.html" role="AEN3542">Connection settings</ulink>
</primaryie>
</indexentry>
@@ -55,19 +55,19 @@
<primaryie>configuration
</primaryie>
<secondaryie>of the slon daemon,
- <ulink url="runtime-config.html" role="AEN3427"></ulink>
+ <ulink url="runtime-config.html" role="AEN3440"></ulink>
</secondaryie>
</indexentry>
<indexentry>
<primaryie>conn_info configuration parameter,
- <ulink url="slon-config-connection.html" role="AEN3539">Connection settings</ulink>
+ <ulink url="slon-config-connection.html" role="AEN3552">Connection settings</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>Controlled switchover,
- <ulink url="failover.html#AEN1007" role="AEN1009"> Controlled Switchover</ulink>
+ <ulink url="failover.html#AEN1005" role="AEN1007"> Controlled Switchover</ulink>
</primaryie>
</indexentry>
@@ -76,7 +76,7 @@
<indexentry>
<primaryie>desired_sync_time configuration parameter,
- <ulink url="slon-config-interval.html" role="AEN3609">Event Tuning</ulink>
+ <ulink url="slon-config-interval.html" role="AEN3622">Event Tuning</ulink>
</primaryie>
</indexentry>
@@ -85,7 +85,7 @@
<indexentry>
<primaryie>failover upon system failure,
- <ulink url="failover.html#AEN1034" role="AEN1036"> Failover</ulink>
+ <ulink url="failover.html#AEN1032" role="AEN1034"> Failover</ulink>
</primaryie>
</indexentry>
@@ -94,37 +94,37 @@
<indexentry>
<primaryie>listen paths,
- <ulink url="listenpaths.html" role="AEN1081">Slony-I listen paths</ulink>
+ <ulink url="listenpaths.html" role="AEN1093">Slony-I listen paths</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>locking issues,
- <ulink url="locking.html" role="AEN1256">Locking Issues</ulink>
+ <ulink url="locking.html" role="AEN1268">Locking Issues</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>log_level configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3478">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3491">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>log_pid configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3487">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3500">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>log_timestamp configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3496">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3509">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>log_timestamp_format configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3505">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3518">Logging</ulink>
</primaryie>
</indexentry>
@@ -133,7 +133,7 @@
<indexentry>
<primaryie>node,
- <ulink url="concepts.html#AEN413" role="AEN415">Node</ulink>
+ <ulink url="concepts.html#AEN412" role="AEN414">Node</ulink>
</primaryie>
</indexentry>
@@ -142,7 +142,7 @@
<indexentry>
<primaryie>origin node,
- <ulink url="concepts.html#AEN441" role="AEN443"> Origin, Providers and Subscribers</ulink>
+ <ulink url="concepts.html#AEN440" role="AEN442"> Origin, Providers and Subscribers</ulink>
</primaryie>
</indexentry>
@@ -151,13 +151,13 @@
<indexentry>
<primaryie>pid_file configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3517">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3530">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>provider node,
- <ulink url="concepts.html#AEN441" role="AEN445"> Origin, Providers and Subscribers</ulink>
+ <ulink url="concepts.html#AEN440" role="AEN444"> Origin, Providers and Subscribers</ulink>
</primaryie>
</indexentry>
@@ -166,13 +166,13 @@
<indexentry>
<primaryie>replicating a first database,
- <ulink url="slonyadmin.html#FIRSTDB" role="AEN595">Replicating Your First Database</ulink>
+ <ulink url="slonyadmin.html#FIRSTDB" role="AEN593">Replicating Your First Database</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>replication set,
- <ulink url="concepts.html#AEN433" role="AEN435"> Replication Set</ulink>
+ <ulink url="concepts.html#AEN432" role="AEN434"> Replication Set</ulink>
</primaryie>
</indexentry>
@@ -193,61 +193,61 @@
<indexentry>
<primaryie>Slony-I for PostgreSQL version upgrades,
- <ulink url="versionupgrade.html" role="AEN2085">Using Slony-I for PostgreSQL Upgrades</ulink>
+ <ulink url="versionupgrade.html" role="AEN2097">Using Slony-I for PostgreSQL Upgrades</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>sql_on_connection configuration parameter,
- <ulink url="slon-config-connection.html" role="AEN3549">Connection settings</ulink>
+ <ulink url="slon-config-connection.html" role="AEN3562">Connection settings</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>subscriber node,
- <ulink url="concepts.html#AEN441" role="AEN450"> Origin, Providers and Subscribers</ulink>
+ <ulink url="concepts.html#AEN440" role="AEN449"> Origin, Providers and Subscribers</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>subscribing nodes,
- <ulink url="subscribenodes.html" role="AEN759">Subscribing Nodes</ulink>
+ <ulink url="subscribenodes.html" role="AEN757">Subscribing Nodes</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>sync_group_maxsize configuration parameter,
- <ulink url="slon-config-interval.html" role="AEN3585">Event Tuning</ulink>
+ <ulink url="slon-config-interval.html" role="AEN3598">Event Tuning</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>sync_interval configuration parameter,
- <ulink url="slon-config-interval.html" role="AEN3562">Event Tuning</ulink>
+ <ulink url="slon-config-interval.html" role="AEN3575">Event Tuning</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>sync_interval_timeout configuration parameter,
- <ulink url="slon-config-interval.html" role="AEN3571">Event Tuning</ulink>
+ <ulink url="slon-config-interval.html" role="AEN3584">Event Tuning</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>syslog configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3450">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3463">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>syslog_facility configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3459">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3472">Logging</ulink>
</primaryie>
</indexentry>
<indexentry>
<primaryie>syslog_ident configuration parameter,
- <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3469">Logging</ulink>
+ <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3482">Logging</ulink>
</primaryie>
</indexentry>
@@ -256,7 +256,7 @@
<indexentry>
<primaryie>vac_frequency configuration parameter,
- <ulink url="slon-config-interval.html" role="AEN3599">Event Tuning</ulink>
+ <ulink url="slon-config-interval.html" role="AEN3612">Event Tuning</ulink>
</primaryie>
</indexentry>
Index: confoptions.h
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/src/slon/confoptions.h,v
retrieving revision 1.19
retrieving revision 1.20
diff -Lsrc/slon/confoptions.h -Lsrc/slon/confoptions.h -u -w -r1.19 -r1.20
--- src/slon/confoptions.h
+++ src/slon/confoptions.h
@@ -31,6 +31,7 @@
bool logpid;
bool logtimestamp;
+bool drop_indices;
char *log_timestamp_format;
char *sql_on_connection;
@@ -212,6 +213,25 @@
&logtimestamp,
true
},
+
+ {
+ {
+ (const char *)"drop_indices_for_copy",
+ gettext_noop("Indicates that indices should be dropped "
+ "on client while invoking COPY_SET event. "
+ "This provides a big performance boost as it "
+ "is cheaper to regenerate indices after loading "
+ "data than it is to incrementally generate "
+ "indexes as the load generates. Default is false."
+ ),
+ NULL,
+ SLON_C_BOOL
+ },
+ &drop_indices,
+ true
+ },
+
+
NULL
};
@@ -297,6 +317,9 @@
&sql_on_connection,
NULL
},
+
+
+
#ifdef HAVE_SYSLOG
{
{
Index: remote_worker.c
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/src/slon/remote_worker.c,v
retrieving revision 1.81
retrieving revision 1.82
diff -Lsrc/slon/remote_worker.c -Lsrc/slon/remote_worker.c -u -w -r1.81 -r1.82
--- src/slon/remote_worker.c
+++ src/slon/remote_worker.c
@@ -888,19 +888,16 @@
}
else if (strcmp(event->ev_type, "ACCEPT_SET") == 0)
{
- int set_id, old_origin, new_origin;
- PGresult *res;
-
slon_log(SLON_DEBUG2, "start processing ACCEPT_SET\n");
- set_id = (int) strtol(event->ev_data1, NULL, 10);
-
+ int set_id = (int) strtol(event->ev_data1, NULL, 10);
slon_log(SLON_DEBUG2, "ACCEPT: set=%d\n", set_id);
- old_origin = (int) strtol(event->ev_data2, NULL, 10);
-
+ int old_origin = (int) strtol(event->ev_data2, NULL, 10);
slon_log(SLON_DEBUG2, "ACCEPT: old origin=%d\n", old_origin);
- new_origin = (int) strtol(event->ev_data3, NULL, 10);
-
+ int new_origin = (int) strtol(event->ev_data3, NULL, 10);
slon_log(SLON_DEBUG2, "ACCEPT: new origin=%d\n", new_origin);
+ int event_no = event->ev_seqno;
+ slon_log(SLON_DEBUG2, "ACCEPT: move set seq=%d\n", event_no);
+ PGresult *res;
slon_log(SLON_DEBUG2, "got parms ACCEPT_SET\n");
/* If we're a remote node, and haven't yet
@@ -2174,6 +2171,8 @@
char conn_symname[64];
SlonDString query1;
SlonDString query2;
+ SlonDString query3;
+ SlonDString indexregenquery;
int ntuples1;
int ntuples2;
int tupno1;
@@ -2267,6 +2266,8 @@
pro_dbconn = pro_conn->dbconn;
loc_dbconn = local_conn->dbconn;
dstring_init(&query1);
+ dstring_init(&query3);
+ dstring_init(&indexregenquery);
sprintf(seqbuf, INT64_FORMAT, event->ev_seqno);
@@ -2727,6 +2728,64 @@
}
PQclear(res2);
+ if (drop_indices) {
+ /*
+ * Drop indices from table on the local database
+ */
+ slon_mkquery(&query1,
+ "select 'alter table %s drop constraint ' || \"pg_catalog\".quote_ident(co.conname), "
+ " 'alter table %s add ' || "
+ " pg_get_constraintdef(co.oid) "
+ "from pg_class c, pg_constraint co, pg_namespace ns "
+ "where "
+ " '%s' = \"pg_catalog\".quote_ident(ns.nspname) || '.' || \"pg_catalog\".quote_ident(c.relname) and "
+ " co.connamespace = ns.oid and "
+ " co.contype in ('p', 'u') and "
+ " c.oid = co.conrelid and "
+ " ns.oid = c.relnamespace "
+ "UNION ALL "
+ "select 'drop index ' || \"pg_catalog\".quote_ident(ns.nspname) || '.' || \"pg_catalog\".quote_ident(ci.relname) || ';' , "
+ " pg_get_indexdef(ci.oid) "
+ "from "
+ " pg_class c, pg_namespace ns, pg_class ci, pg_index i "
+ "where "
+ " '%s' = \"pg_catalog\".quote_ident(ns.nspname) || '.' || \"pg_catalog\".quote_ident(c.relname) and "
+ " i.indrelid = c.oid and "
+ " i.indexrelid = ci.oid and "
+ " not exists (select * from pg_constraint co where connamespace = ns.oid and "
+ " conrelid = c.oid and contype in ('p', 'u') and co.conname = ci.relname);",
+
+ tab_fqname, tab_fqname, tab_fqname, tab_fqname);
+ res2 = PQexec(loc_dbconn, dstring_data(&query1));
+ ntuples2 = PQntuples(res2);
+ slon_log(SLON_DEBUG2, "remoteWorkerThread_%d: Found %d indices for %s - %s\n",
+ node->no_id, ntuples2, tab_fqname, dstring_data(&query1));
+
+ slon_mkquery(&indexregenquery, " ");
+
+ slon_log(SLON_DEBUG2, "remoteWorkerThread_%d: start dropping %d indices\n", node->no_id, ntuples2);
+ for (tupno2 = 0; tupno2 < ntuples2; tupno2++)
+ {
+ slon_mkquery(&query3,
+ "%s;",
+ PQgetvalue(res2, tupno2, 0));
+ if (query_execute(node, loc_dbconn, &query3) < 0)
+ {
+ slon_log(SLON_FATAL, "remoteWorkerThread_%d: drop index during copy failed: %d\n",
+ node->no_id, dstring_data(&query3));
+ PQclear(res2);
+ slon_abort();
+ } else {
+ slon_log(SLON_DEBUG2, "remoteWorkerThread_%d: Dropped index %s\n",
+ node->no_id, dstring_data(&query3));
+ }
+
+ slon_appendquery(&indexregenquery,
+ "%s;",
+ PQgetvalue(res2, tupno2, 1));
+ }
+ }
+
/*
* Begin a COPY from stdin for the table on the local DB
*/
@@ -3002,6 +3061,20 @@
INT64_FORMAT " bytes copied for table %s\n",
node->no_id, copysize, tab_fqname);
+ if (drop_indices) {
+ slon_log(SLON_DEBUG2, "now, regenerate indices...\n");
+ slon_log(SLON_DEBUG2,
+ "remoteWorkerThread_%d: "
+ "regenerate indices: %s",
+ node->no_id, dstring_data(&indexregenquery));
+ if (query_execute(node, loc_dbconn, &indexregenquery) < 0)
+ {
+ PQclear(res2);
+ slon_disconnectdb(pro_conn);
+ dstring_free(&indexregenquery);
+ return -1;
+ }
+ }
/*
* Analyze the table to update statistics
*/
- Previous message: [Slony1-commit] By cbbrowne: More on the "saga of the duplicate key violation."
- Next message: [Slony1-commit] By cbbrowne: Changes to RPM spec file per Devrim Gunduz to address
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Slony1-commit mailing list