purplecat (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
purplecat) wrote in ![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png)
dw_dev_training2012-03-25 12:24 pm
purplecat) wrote in dw_dev_training2012-03-25 12:24 pm
Entry tags:
Test Plan Walkthrough: Bug 2154
As well as uploading a patch file for bug 2154, I uploaded the notes I had made for the test plan for the patch in case they were of use in the review process. ![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
denise asked if I could write a post explaining how I came up with the tests for this comm, since it was something people often asked about.
A little bit of background first. I've worked with a number of large code bases in my career but normally as either the only user of the final system, or with a very small number of users in an informal setting where bugs cause minimal disruption and can be quickly fixed. Testing is usually based on an automated set of ad hoc tests that are run before a commit and added to as I feel necessary. However, in the long tradition of universities getting people to teach stuff they've never used, I had actually lectured on testing for four years as part of a software engineering module. So basically, this was my first opportunity to put the theory I had inflicted onto approx. 350 students into practice.
I wrote up a test plan in advance of starting testing. I wanted to separate out the process of thinking up tests from actually doing them. This had a couple of advantages. If I were to get bored and fed up halfway through the test process, it would be clear to me what I had tested and what I hadn't and would prevent me fooling myself that I had actually done all the testing I had thought necessary. Secondly, writing the plan in advance allowed me to organise the process into a logical order minimising, for instance, the number of times I had to log out of one user account and log into another.
Bug 2154 relates to community settings. At the moment you can provide an entry containing posting guidelines and configure your community to display a link to this entry to new members when they join up. Bug 2154 was to extend this feature to allow the link to go to the community profile instead of an entry, if desired. Currently there is a text box for a url in the community settings page which, if filled in, automatically sets the url to be the posting guidelines. This was replaced by a three way choice of a) no posting guidelines, b) use the community profile for posting guidelines and c) use an entry for the posting guidelines. Under choice c) the user got to fill in the URL for the entry they were interested in. This links to screenshot of the new settings page since that may make it easier to see the layout and options.
So clearly I needed, at the very least, tests for communities with options a), b) and c) set. I also knew, because I'd found a bug related to it during development, that I needed to check what was happening if "junk" information were filled into the text box. Specifically I wanted to check that the "junk" url* didn't get stored or displayed in any of the three cases. In the database the URL for the posting guidelines, and the preference for whether the guidelines were "none", "profile" or "entry" were stored separately - so there was all sorts of potential for bugs where the community administrator had selected "none" or "profile" and yet the URL for an entry would nevertheless get displayed somewhere. I decided to test both if "junk" were put in the box and either options a) or b) selected (when the junk should just be ignored) and what happened if junk were put in the box with "entry" selected (which would flag up an error) and then the user either switched to either a) or b) or corrected the URL. That gave me 8 different community settings to investigate. I made notes in the test regime when I was expecting to see error messages, to save me having to recall what the test case was supposed to demonstrate.
I also wanted to check that the patch behaved correctly both when a communities settings were entered for the first time and when they were changed. So those 8 community settings gave me 8 separate communities to create (and then I'd write tests for what would happen when I modified them).
I actually wrote my test regime 24 hours before I did the testing. That happened by chance but did have benefits. Over-night it occurred to me that I wanted to check not just what happened when junk was entered into the text box and a) or b) was selected, but also what happened when a valid URL got put in the box. Basically the URL might get filtered out by error detection if it was garbage, but might nevertheless get stored and cause trouble if it were valid. So that put two more communities on my list.
It should be noted that testing combinations of the radio buttons and the information entered into the text box was very much driven by my knowledge that those two pieces of information were stored separately and yet were related, so I needed to check that the right information was retrieved no matter what random set of inputs had been entered. This is in contrast to my decision that I didn't need to test against combinations of the other options on the settings page.
So the first part of my test regime was to create these 10 communities, input the desired settings and then see if any error messages were displayed properly, and whether the post set-up screen displayed the correct link to the posting guidelines. Running these tests actually did pick up a bug in the implementation - when option b) was selected a link to the posting guidelines wasn't displayed to the community administrator. This was detected on the second test. I fixed the bug but didn't go back and recheck the first two test cases, making the judgment call that any problems caused by the fix would be revealed in the later test cases. If you like the first three test cases were really just a pre-filter to catch obvious bugs quickly before looking for more subtle ones.
As a note there are a lot of options on the settings page for communities and they are all processed by the same code, so to test really, really thoroughly you would need to check that combinations of these options didn't interact in unexpected ways. I didn't do the maths, but I know enough combinatorics to know that testing all possible combinations of every button on that page would involve a lot of testing. I made a judgement call that the code I'd added or altered was sufficiently self-contained that it wouldn't interact with all the other settings on the page, even though it was contained in the same file as the code managing those settings and would be run at the same time.
Having set up the 10 communities I then needed to test that the right links would be displayed to anyone joining the community. I think if I had been a bit more familiar with the DW codebase I might have decided that checking that subscribers saw the right thing would have been sufficient, but I'm not, so I decided to test that the right links appeared both when someone joined the community and when someone just subscribed to the community.
Then I wanted to see what happened if the community settings were changed. I could have added yet more communities at this point so, for instance I could see what happened to a community originally set up to have no posting guidelines which was then switched to guidelines in the community profile and what happened to a community with no posting guidelines that was then switched to guidelines held in an entry... and so on for all the combinations of junk URLs and so forth. However I made a judgment call from the way I'd coded things up that the various options involving junk URLs, wouldn't make any difference at the modification stage, if they hadn't at the creation stage. I decided what I wanted to check here was what happened if
As a result I divided my 10 communities into three groups - those which currently had option a) selected, those with option b) selected and those with option c) selected and made sure that within each group I checked the switch to both the other options. In those cases where some kind of URL might have sneakily been stored in the database, I made a note in the test document that I should check that the URL wasn't pre-filled in the settings form.
Then I tested what happened when someone subscribed to all those communities, and what happened when someone who had subscribed under the previous settings, left the community and then rejoined it under the new settings (this was to check there was no odd persistence of the community settings in the user's preferences).
This was a final test that occurred to me over-night, and not when I original wrote the tests down.
In the cases where a community had at one point linked to some posting guidelines in an entry, but then changed to either option a) or option b), I wanted to check that the URL for the entry did remain in the database, so if the community wanted to switch back to using the entry it could easily - the old URL would turn up pre-filled into the text box on the settings page. I also needed to check what happened if they switched back to using an entry, but wanted to link to a different post.
And then I needed to check that someone subscribing to those communities would be linked to the right URL.
The final document can be seen here. I've made a couple of notes, for instance that I was running into community creation limits and you can finddenise's explanation for avoiding those in the comments to Bug 2154.
It should be noted that there is a lot of post-hoc rationalisation in the above. One of the things I've always found useful about going through any semi-formal design process it that it forces you to actually think about options. The simple fact of sitting down to write out my tests, made me think of things that needed testing. The document was not created in exactly the order given above - there was more back and forth as I thought of things - and that's not to mention the tests that occurred to me once I'd slept on the idea, after having written my initial plan out.
*I didn't test the correct detection of junk urls since I was using a pre-existing method for that. So I basically just typed "garbage" into the text box for these cases.
denise asked if I could write a post explaining how I came up with the tests for this comm, since it was something people often asked about.A little bit of background first. I've worked with a number of large code bases in my career but normally as either the only user of the final system, or with a very small number of users in an informal setting where bugs cause minimal disruption and can be quickly fixed. Testing is usually based on an automated set of ad hoc tests that are run before a commit and added to as I feel necessary. However, in the long tradition of universities getting people to teach stuff they've never used, I had actually lectured on testing for four years as part of a software engineering module. So basically, this was my first opportunity to put the theory I had inflicted onto approx. 350 students into practice.
I wrote up a test plan in advance of starting testing. I wanted to separate out the process of thinking up tests from actually doing them. This had a couple of advantages. If I were to get bored and fed up halfway through the test process, it would be clear to me what I had tested and what I hadn't and would prevent me fooling myself that I had actually done all the testing I had thought necessary. Secondly, writing the plan in advance allowed me to organise the process into a logical order minimising, for instance, the number of times I had to log out of one user account and log into another.
Bug 2154
Bug 2154 relates to community settings. At the moment you can provide an entry containing posting guidelines and configure your community to display a link to this entry to new members when they join up. Bug 2154 was to extend this feature to allow the link to go to the community profile instead of an entry, if desired. Currently there is a text box for a url in the community settings page which, if filled in, automatically sets the url to be the posting guidelines. This was replaced by a three way choice of a) no posting guidelines, b) use the community profile for posting guidelines and c) use an entry for the posting guidelines. Under choice c) the user got to fill in the URL for the entry they were interested in. This links to screenshot of the new settings page since that may make it easier to see the layout and options.
So clearly I needed, at the very least, tests for communities with options a), b) and c) set. I also knew, because I'd found a bug related to it during development, that I needed to check what was happening if "junk" information were filled into the text box. Specifically I wanted to check that the "junk" url* didn't get stored or displayed in any of the three cases. In the database the URL for the posting guidelines, and the preference for whether the guidelines were "none", "profile" or "entry" were stored separately - so there was all sorts of potential for bugs where the community administrator had selected "none" or "profile" and yet the URL for an entry would nevertheless get displayed somewhere. I decided to test both if "junk" were put in the box and either options a) or b) selected (when the junk should just be ignored) and what happened if junk were put in the box with "entry" selected (which would flag up an error) and then the user either switched to either a) or b) or corrected the URL. That gave me 8 different community settings to investigate. I made notes in the test regime when I was expecting to see error messages, to save me having to recall what the test case was supposed to demonstrate.
I also wanted to check that the patch behaved correctly both when a communities settings were entered for the first time and when they were changed. So those 8 community settings gave me 8 separate communities to create (and then I'd write tests for what would happen when I modified them).
I actually wrote my test regime 24 hours before I did the testing. That happened by chance but did have benefits. Over-night it occurred to me that I wanted to check not just what happened when junk was entered into the text box and a) or b) was selected, but also what happened when a valid URL got put in the box. Basically the URL might get filtered out by error detection if it was garbage, but might nevertheless get stored and cause trouble if it were valid. So that put two more communities on my list.
It should be noted that testing combinations of the radio buttons and the information entered into the text box was very much driven by my knowledge that those two pieces of information were stored separately and yet were related, so I needed to check that the right information was retrieved no matter what random set of inputs had been entered. This is in contrast to my decision that I didn't need to test against combinations of the other options on the settings page.
Testing initial settings worked properly
So the first part of my test regime was to create these 10 communities, input the desired settings and then see if any error messages were displayed properly, and whether the post set-up screen displayed the correct link to the posting guidelines. Running these tests actually did pick up a bug in the implementation - when option b) was selected a link to the posting guidelines wasn't displayed to the community administrator. This was detected on the second test. I fixed the bug but didn't go back and recheck the first two test cases, making the judgment call that any problems caused by the fix would be revealed in the later test cases. If you like the first three test cases were really just a pre-filter to catch obvious bugs quickly before looking for more subtle ones.
As a note there are a lot of options on the settings page for communities and they are all processed by the same code, so to test really, really thoroughly you would need to check that combinations of these options didn't interact in unexpected ways. I didn't do the maths, but I know enough combinatorics to know that testing all possible combinations of every button on that page would involve a lot of testing. I made a judgement call that the code I'd added or altered was sufficiently self-contained that it wouldn't interact with all the other settings on the page, even though it was contained in the same file as the code managing those settings and would be run at the same time.
Testing the behaviour for users joining communities
Having set up the 10 communities I then needed to test that the right links would be displayed to anyone joining the community. I think if I had been a bit more familiar with the DW codebase I might have decided that checking that subscribers saw the right thing would have been sufficient, but I'm not, so I decided to test that the right links appeared both when someone joined the community and when someone just subscribed to the community.
Testing that modification of settings worked correctly
Then I wanted to see what happened if the community settings were changed. I could have added yet more communities at this point so, for instance I could see what happened to a community originally set up to have no posting guidelines which was then switched to guidelines in the community profile and what happened to a community with no posting guidelines that was then switched to guidelines held in an entry... and so on for all the combinations of junk URLs and so forth. However I made a judgment call from the way I'd coded things up that the various options involving junk URLs, wouldn't make any difference at the modification stage, if they hadn't at the creation stage. I decided what I wanted to check here was what happened if
- A community with one setting was switched to either of the other settings.
- A community which had been set up with either option a) or b) but where some information had also been put in the text box, didn't behave strangely when modified.
As a result I divided my 10 communities into three groups - those which currently had option a) selected, those with option b) selected and those with option c) selected and made sure that within each group I checked the switch to both the other options. In those cases where some kind of URL might have sneakily been stored in the database, I made a note in the test document that I should check that the URL wasn't pre-filled in the settings form.
Testing behaviour for users joining modified communities
Then I tested what happened when someone subscribed to all those communities, and what happened when someone who had subscribed under the previous settings, left the community and then rejoined it under the new settings (this was to check there was no odd persistence of the community settings in the user's preferences).
Testing persistence of entry guidelines that had been abandoned
This was a final test that occurred to me over-night, and not when I original wrote the tests down.
In the cases where a community had at one point linked to some posting guidelines in an entry, but then changed to either option a) or option b), I wanted to check that the URL for the entry did remain in the database, so if the community wanted to switch back to using the entry it could easily - the old URL would turn up pre-filled into the text box on the settings page. I also needed to check what happened if they switched back to using an entry, but wanted to link to a different post.
And then I needed to check that someone subscribing to those communities would be linked to the right URL.
Final Notes
The final document can be seen here. I've made a couple of notes, for instance that I was running into community creation limits and you can find
denise's explanation for avoiding those in the comments to Bug 2154.It should be noted that there is a lot of post-hoc rationalisation in the above. One of the things I've always found useful about going through any semi-formal design process it that it forces you to actually think about options. The simple fact of sitting down to write out my tests, made me think of things that needed testing. The document was not created in exactly the order given above - there was more back and forth as I thought of things - and that's not to mention the tests that occurred to me once I'd slept on the idea, after having written my initial plan out.
*I didn't test the correct detection of junk urls since I was using a pre-existing method for that. So I basically just typed "garbage" into the text box for these cases.