Andrew Sullivan <ajs at crankycanuck.ca> writes:
> On Sun, Oct 02, 2005 at 07:39:44PM -0700, David Fetter wrote:
>> 1.  Is the Afilias QA team on this list?  If not, which lists are they
>> on?  We all need to get in better touch here.
>
> Not as far as I know; but to be more pointed, Afilias _does not_ do
> the QA of these releases before they happen.  The Afilias QA folks do
> testing of proposed releases to production systems for Afilias
> purposes, but they don't test this software before it's released. 
> (This state of affairs is admittedly partly because, when I was
> supervising Chris[1], I could alter his job description to include
> maintenance of Slony-I; but it's been well over a year since I had
> any supervisory duties of the QA folks, so I couldn't assign them the
> work.  But in truth, the QA folks at Afilias are already
> overcommitted.)  Slony-I was always intended to be a community
> project: Afilias seeded it, but it's not an Afilias project released
> to the community, and was never intended to be.
>
>> 2.  What exactly is the release process, and where is a broad, obvious
>> place to post it?
>
> It sounds like one needs to be made.  Volunteers?

It's pretty unclear, at this point, as far as major releases are
concerned.

We have tried to have as many success reports on as many platforms as
possible, and have solicited assistance in that.

There are a set of steps for releasing a build; it is somewhat error
prone at this point due to there being quite a few places where files
reference CVS tags and/or release versioning information.

If someone can suggest ways of grabbing those tags/version numbers
automatically, that ought to be a help.

>> 3.  What regression tests are running, and how do we know they've
>> passed on all platforms?  I know this is a tough thing even just in a
>> combinatorial sense, but it's one we have to do.
>
> My suspicion is that this will fall out of (2).  In particular, I
> don't think that there are currently assurances that people have run
> the tests prior to release.  While on the one hand, that seems like a
> pretty bad thing, on the other hand, it's more or less how PostgreSQL
> itself was released for ages.  That said. . .

One of the "new in 1.2" things is supposed to be the new, scripted
test framework that Darcy has had under review for [probably too long
now].  It's not in CVS yet; search the mailing list for the URL to the
patch...

The problem up until now has been that the "test bed" (e.g. - in
src/ducttape) has too many undesirable components.  It depends on X,
for instance, and doesn't permit any way of automatically running
through a bunch of tests.

I'd hope that the New Test Framework would permit us a way of setting
up some failover tests in an automated way...

>> 4.  The subject of a "build farm" has come up several times in
>> different venues.  Can we get this thing spec'ed and put together in
>> time for 1.2?  What design pieces are still missing?
>
> . . .this'd no doubt help.  But I don't know the answer to your
> questions.

We need to get Darcy's tests added to CVS, and to then construct a
more complete set of tests based on that test harness.

Vis-a-vis the 1.1.1 situation, I daresay I was in something of a "rock
and hard place" situation...

 1.  I needed certain changes to deploy 1.1.x by September 30th.

 2.  There hasn't been time to get the testing done to figure out
     what's up with FAIL OVER.

     I was very much Otherwise Occupied until September 20th, and my
     schedule didn't permit enough time to dig deep enough to be able
     to address the issue.

When I'm back from vacation, I'll be very keen to get better
procedures in place for future releases.
-- 
select 'cbbrowne' || '@' || 'ca.afilias.info';
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 673-4124 (land)