Robert 'Callahan wrote:
Darin Fisher wrote:
>I think that we should try to reach a point where we can require
>regression/unit tests for every patch that fixes a bug or introduces a
>new API.

Sounds good to me. Sounds even better if this leverages QA
staff/community volunteers with a minimum burden on developers :-)

I've never seen that work. The only thing I've seen work is when the
developers are responsible for writing the bulk of the automated tests.

Developers are not as likely to write testable code if they are not the
ones experiencing the pain of writing tests for that code.

QA staff and community volunteers should be writing additional tests,
and tests of a different type than those written by developers. But
developers need to step up and write tests.

Robert 'Callahan wrote:
Darin Fisher wrote:
>I think that we should try to reach a point where we can require
>regression/unit tests for every patch that fixes a bug or introduces a
>new API.

Sounds good to me. Sounds even better if this leverages QA
staff/community volunteers with a minimum burden on developers :-)

I've never seen that work. The only thing I've seen work is when the
developers are responsible for writing the bulk of the automated tests.

Developers are not as likely to write testable code if they are not the
ones experiencing the pain of writing tests for that code.

QA staff and community volunteers should be writing additional tests,
and tests of a different type than those written by developers. But
developers need to step up and write tests.

Friday 2006-06-23 12:07 +1200, Robert 'Callahan wrote:
When you say "tests" I'm not sure if you mean test cases or test
automation. Developers must support test automation. But for layout and
rendering code, at least, a single framework can test all kinds of
content, including new kinds of content, so we don't necessarily need to
do anything to make it "testable".

If you mean specifically test cases, then I have to say that I've met
several Mozilla QA volunteers and staff who are far better at writing
test cases than I am. Not to mention the fact that most of the bugs that
I fix come have decent test cases attached before I look at them.

FWIW (sort of related): thing that occurred to me today is that one
way to write layout and CSS regression tests that *don't* require a
baseline run without the developer's changes is to have tests where the
reference rendering is also in HTML. Many of Hixie's tests could have a
simple reference rendering written in HTML that would be independent of
the operating system, the user's fonts, and even of many changes in our
default behavior (e.g., changing the ua.css margin on BDY or other such
trivial changes that would require a completely new baseline).

Tests of this type would be easier to hook up to something like
tinderbox that can go red if the tests fail, although they are harder to
write.
-David

Friday 2006-06-23 12:07 +1200, Robert 'Callahan wrote:
When you say "tests" I'm not sure if you mean test cases or test
automation. Developers must support test automation. But for layout and
rendering code, at least, a single framework can test all kinds of
content, including new kinds of content, so we don't necessarily need to
do anything to make it "testable".

If you mean specifically test cases, then I have to say that I've met
several Mozilla QA volunteers and staff who are far better at writing
test cases than I am. Not to mention the fact that most of the bugs that
I fix come have decent test cases attached before I look at them.

FWIW (sort of related): thing that occurred to me today is that one
way to write layout and CSS regression tests that *don't* require a
baseline run without the developer's changes is to have tests where the
reference rendering is also in HTML. Many of Hixie's tests could have a
simple reference rendering written in HTML that would be independent of
the operating system, the user's fonts, and even of many changes in our
default behavior (e.g., changing the ua.css margin on BDY or other such
trivial changes that would require a completely new baseline).

Tests of this type would be easier to hook up to something like
tinderbox that can go red if the tests fail, although they are harder to
write.
-David