Are regression tests the entire test suite or a sample of tests?

Question

I was taught that a regression test was a small (only enough to prove you didn't break anything with the introduction of a change or new modules) sample of the overall tests. However, this article by Ron Morrison and Grady Booch makes me think differently:

The desired strategy would be to bring each unit in one at a time, perform an extensive regression test, correct any defects and then proceed to the next unit.

The same document also says:

As soon as a small number of units are added, a test version is generated and "smoke tested," wherein a small number of tests are run to gain confidence that the integrated product will function as expected. The intent is neither to thoroughly test the new unit(s) nor to completely regression test the overall system.

When describing smoke testing, the authors say this:

It is also important that the Smoke Test perform a quick check of the entire system, not just the new component(s).

I've never seen "extensive" and "regression test" used together nor a regression test described as "completely regression test the overall system". Regression tests are supposed to be as light and quick as possible. And the definition of smoke test is what I learned a regression test was.

Did I misunderstand what I was taught? Was I taught incorrectly? Or are there multiple interpretations of "regression test"?

Typhoeus

Answer 1

There are multiple interpretations. If you're only fixing a bug that affects one small part of your system then regression tests might only include a small suite of tests that exercise the class or package in question. If you're fixing a bug or adding a feature that has wider scope then your regression tests should have wider scope as well.

The "if it could possibly break, test it" rule of thumb applies here. If a change in Foo could affect Bar, then run the regressions for both.

Regression tests just check to see if a change caused a previously passed test to fail. They can be run at any level (unit, integration, system). Reference.

Shiff

Well, I learned that regression tests should do two things. (1) The changed components should be well-tested. (2) Any critical components of the system, even those not changed should be tested in a regression test. – Typhoeus

That's not a bad rule. Beware if every component suddenly becomes "critical", though. – Shiff

Well, you have to define "critical". But yes. – Typhoeus

Answer 2

I always took regression testing to mean any tests whose purpose was to ensure that existing functionality is not broken by new changes. That would not imply any constraint on the size of the test suite.

Tiptoe

Answer 3

Regression is generally used to refer to the whole suite of tests. It is the last thing QA does before a release. It is used to show that everything that used to work still works, to the extent that that is possible to show. In my experience, it is generally a system-wide set of tests regardless of how small the change was (although small changes may not trigger a regression test).

Rom

You're describing system testing, or system integration testing. Regression tests just check to see if previously passed tests fail, and can be run at any time. – Shiff

tloach and Elie are describing the same thing, and I doubt they work at the same company. This describes a model of a first-run of tests, followed by fixes and such, and a final regression test phase before shipping. Using the word "regression" here is perfectly valid. – Redhot

"Regression" does not mean the whole suite of tests. It means rerunning existing tests to see if new code broke them. It doesn't (necessarily) mean you have to run the entire suite, but it can. I'm just pointing out a subtle difference. – Shiff

Though @BilltheLizard has described correctly but I think it depends on organization to organization. In my organization generally the IMPACTED AREA is tested in regression. But when sometimes higher management insists we do complete regression. – Sinistrous

Answer 4

Where I work, regression tests are standardized for each application at the end of each release. They are intended to test all functionality, but they are not designed to catch subtle bugs. So if you have a form that has various kinds of validation done on it, for example, a regression suite for that form would be to confirm that each type of validation gets done (field level and form level) and that correct information can be submitted. It is not designed to cover every single case (i.e. what if I leave field A blank? How about field B? it will just test one of them and assume the others work).

However, on the current project I'm working on, the regression tests are much more thorough, and we have noticed a reduction in the number of defects being raised during testing. Those two are not necessarily related, but we do notice it fairly consistently.

Menis

This is system (integration) testing. – Typhoeus

No it's not. The System (integration) testing is for the new functionality. The regression testing is for the stuff we didn't change during the current build. Our regression test suite is continually growing as we add more functions. – Menis

Regression testing is to test that new code didn't break tests that passed before. You're describing system testing. – Shiff

And now we get into a semantics argument. It's silly. Elie is exactly right, those tests are considered regression tests where she works, and where I used to work, same thing. Words mean things, and frequently the same word means more than one thing. – Redhot

Regression testing has a wider scope than unit, integration, and system testing. It can be done at any of those levels. Words do have meaning, and since this is a public forum we should try to use the most widely accepted definition. – Shiff

Answer 5

my understanding of the term 'regression testing' is:

• unit tests are written to test features when the system is created
• when bugs are discovered, more unit tests are written to reproduce the bug and verify that it has been corrected
• a regression test runs the entire set of tests prove that everything still works including that no old bugs have reappeared [i.e. to prove that the code has not "regressed"]

in practice, it is best to always run all existing unit tests when changes are made. the only time i'd bother with a subset of tests is when the full unit test suite takes "too long" to run [where "too long" is fairly subjective]

Parris

Answer 6

Start with what you are trying to accomplish. Then do what you need to do to accomplish that goal. And then use buzzword bingo to assign a word to what you actually do. Just like everyone else :-) Accuracy isn't all that important.

Unpromising

Communicating clearly is important. It helps if we use the same dictionary. – Tiptoe

Answer 7

... regression test was a small (only enough to prove you didn't break anything with the introduction of a change or new modules) sample of the overall tests

If a small sample of tests is enough to prove that the system works, why do the rest of the tests even exist? And if you think you know that your change only affected a subset of functionality, then why do you need to test anything after making the change? Humans are fallible, nobody really knows if changing something breaks something else. IMO, if your tests are automated, re-run them all. And if they aren't automated, automate them. In the mean time, re-run whatever is automated.

Kongo

Answer 8

In general, a subset of the feature tests for the new feature introduced in version X of a product becomes the basis of the regression tests for version X+1, X+2, and so on. Over time, you may reduce the time taken by the feature/regression tests of stable features which have not suffered from regressions. If a feature suffers from lots of regressions, then it may be beneficial to increase the emphasis on the feature.

I think that the article referring to 'extensive regression test' means run an extensive set of (individually simple) regression tests.

Giantess