Josh Berkus wrote:

>Darcy, Chris, Jan,
>
>  
>
>>FWIW, I'm looking at their system now and am seeing persistent lock
>>contention on pg_listener:
>>
>> relation | database  | transaction |  pid  |     mode      | granted
>>----------+-----------+-------------+-------+---------------+---------
>>    16414 | 318624738 |             | 18883 | ExclusiveLock | f
>>    16414 | 318624738 |             | 19028 | ExclusiveLock | f
>>    16414 | 318624738 |             | 19205 | ExclusiveLock | t
>>
>>
>>This lock contention seems constant.       Why would pg_listener be having
>>blocking exclusive locks?
>>    
>>
>
>To follow this up: the above lock contention is preventing vacuum from working 
>on pg_listener:
>
>INFO:  vacuuming "pg_catalog.pg_listener"
>INFO:  "pg_listener": found 0 removable, 500018 nonremovable row versions in 
>6175 pages
>DETAIL:  500006 dead row versions cannot be removed yet.
>There were 0 unused item pointers.
>0 pages are entirely empty.
>CPU 0.01s/0.12u sec elapsed 0.14 sec.
>INFO:  analyzing "pg_catalog.pg_listener"
>INFO:  "pg_listener": 6175 pages, 12 rows sampled, 12 estimated total rows
>VACUUM
>
>This *severe* table attenuation is the cause of pg_listener queries taking > 1 
>second and starting to block.    How do we clear it up without bringing the 
>system down?
>
>  
>
THAT would be the problem.

Is there some old transaction lingering around perhaps <IDLE in 
transaction>?

That table will presumably need a VACUUM FULL when you get a chance.  
(That will slow replication while it's running, but shouldn't have much 
impact on users; it doesn't block inserts into replicated tables...)