Testing Times!

[jeshyr]

I'd really like to work on more testing bugs ... but I'm not sure where to start. I really enjoyed looking at all the *.t test files when I did Bug 5116 and boilerplated all the test files.

Currently in Bugzilla we only have two bugs that are listed under Testing:

Bug 4439 - fix test suite so it can be run safely against the db.
Bug 3525 - Investigate separating tests files from main files

The second of those is from 2011 and I think it would be obsoleted by Bug 4439 anyway, because the rationale for separating is that we don't accidentally screw up prod's database?

In any case they look too big and scary for me to start with, and I know that uncanny_kate and karelia and others have discussed working on tests too, so I want to make sure I'm not doing stuff that's already been done or obsoleted.

Is there anywhere else I could start with doing testing things??

Comments

[alierak]

So, one strategy would be to try running some of the tests, see what breaks, and decide whether those are bugs in the code, bugs in the tests, or just issues with the environment.

For example, when I try to prove t/00-compile.t in production, it dies like this:

t/00-compile.t .. 23/1214
# Failed test 'STDERR of DW/RenameToken.pm'
# at t/00-compile.t line 51.
# got: 'UNIVERSAL->import is deprecated and will be removed in a future perl at /usr/share/perl5/Google/Checkout/General/Util.pm line 103
# '
# expected: ''
Bailout called. Further testing stopped: Test failed. BAIL OUT!.
FAILED--Further testing stopped: Test failed. BAIL OUT!.

I'm not sure exactly what category this goes in -- we're not actually using Google Checkout in production, but still have the module installed. Maybe there's an upgrade available for it that would make the warning go away, but maybe it would be possible to change something in DW::Shop::Engine::GoogleCheckout to avoid the warning instead. So this might be considered a problem with the production environment, or it might be a small code fix.

So, if you hit one of those and have no idea what to do with it, you can also try to take the approach of just disabling that test in order to get past it and see what else would fail next. There's a mechanism in 00-compile.t for silently skipping files that we know have no hope of working in a compile test. There's a suggestion in bug 4439 that we create a subdirectory like t/bad and move entire .t files into it. Some test frameworks have ways to indicate that a test should be skipped, etc.

The other big idea from bug 4439 is that it's not safe to run all of the tests if you care about the integrity of your database. Apparently the full test suite will create temp users, post test entries, etc., without cleaning up after itself, and Dre has taken a stab at listing the .t files to avoid. As far as I know Kareila's recently picked up that bug and is planning on fixing it (though whether by disabling tests or improving the temp user feature I'm not sure). That said, in a dreamhack, you might not be too worried about messing up the db if you (or you plus help) can wipe it out and set it up again afterwards.

Then there's the strategy of examining the test coverage and finding an area you think needs more tests (e.g., where bugfixes have recently been committed without adding tests, or areas are constantly having bugs reported). You can also look for things in the test suite that say TODO, FIXME, SKIP, etc., where somebody might have meant to come back and improve a test.

But as bug 3525 mentions, not all of the tests (or components needed for testing) are in t/ -- for example, views/dev/ has stuff used for testing css and js, written with Template Toolkit and placed where it needs to be so that you can load it in a browser as if it were part of the site. This is a bit different from the perl command-line sort of pass/fail tests under t/, and maybe there are other categories of tests we could be using too.

On 3525 itself, I disagree with the suggestion to use multiple repositories and multicvs.conf (since that was eliminated in bug 4436), but I could certainly see having all of the "dev stuff" packed away in a "dev" or "t" area, and then have a script that puts them in place when you actually want them in a dev environment.

Sorry if this is long and handwavy, let me know if you want something more concrete...