Bill Moran wmoran at potentialtech.com
Wed Sep 30 06:02:18 PDT 2009
In response to roctaiwan <nettreeinc at gmail.com>:
> 
> first, I don't think I had turn on WAL. The #archive_mode parameter has not
> been turned on. Unless it's on naturally by this is how postgresql should
> behave.
> second, how much space is big enough to handle 10M inserts? I have 6.4GB
> available right now. 

6.4G is almost certainly not enough to be replicating databases in the 10s
of millions of rows via Slony.

However, without details of your database, it's hard to say.  If each row
is a single INT field, the answer is much different than if each row is
made up of a dozen TEXT fields, each of which has War and Peace stored in
it.

When Slony replicates your 10M row transaction, it has to copy all the rows
into the table.  If your slave is configured with forwarding, it also copies
it into the sl_log table.  Each of those actions has to be WAL logged as it's
occurring.  This results in your 10M rows existing 4 times on the target
server as the action is occurring.

It's quite possible that the space was freed back up before your GUI window
could show the out of space condition.  Also, you seem to have put your
DB on the / partition, which means an out of space condition is going to
hose things royally.  Assuming your system is one big partition, you're
going to be competing against things like /tmp and /var/log, which are also
frequent culprits to fill up disks.

There's a lot of information we don't have, but I suggest you take a look at
it for your own purposes:
* How much disk space does your DB occupy?  (pg_database_size() is your
  friend here)
* How much do the larger transactions you expect to have require?  You'll
  need to set up some tests to figure it out.
* You need to get some experience watching PostgreSQL growing and shrinking
  the space it uses on disk.  Tools like MRTG can be useful to gaining that
  experience.
* 6G is not lot of disk space.

I'm frequently frustrated by people who argue "I've got plenty of space free,
10G" when it's on a 300G partition.  3% is not a lot of space, no matter how
many G it turns out to be.  If you've got 290G of data, that could trivially
expand to 310G during normal operation.  In any event, a database storing
10s of millions of rows on a partition with only 6G free is a problem, whether
that 6G is 50% free or 5% free.  (and the .4G isn't even worth mentioning)

> melvin6925 wrote:
> > 
> >>Again, mentioned it at very beginning. If I load whole 10M records all at
> >>once, records won't show even for 3 days. 
> > 
> > 
> > And again, as previously mentioned by myself and others. and your own
> > finding in the error log, you do not have enough space for the postgresql
> > wal file to handle 10 million inserts. 
> > 
> >  
> > Melvin Davidson 
> > 
> 
> -- 
> View this message in context: http://www.nabble.com/copying-large-file-won%27t-get-replicate-tp25609853p25675248.html
> Sent from the Slony-I -- General mailing list archive at Nabble.com.
> 
> _______________________________________________
> Slony1-general mailing list
> Slony1-general at lists.slony.info
> http://lists.slony.info/mailman/listinfo/slony1-general


-- 
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/


More information about the Slony1-general mailing list