Teaching programmers to test
Would you trust a programmer to test your application? It’s like asking a Fox to guard the chicken-house, right?
Well, sometimes you don’t have a choice, or you do but that choice is to release the application untested…
As part of an Agile testing consulting engagement I am doing at a friend’s company, yesterday I started providing short training sessions for programmers who need to learn how to test better. It’s not that this company doesn’t have testers, they have good testers! But as many other agile teams they have much more testing tasks than available testers, and they want programmers to take part in at least some of their testing tasks.
Programmers are not good at testing!
A while I wrote a post explaining why I think programmers make poor testers, and I still think that in a sense it is like asking a dog to fly (take a look at the cool picture in the post 😉). On the other hand I also believe that with some good intentions, hard work, and perseverance you can teach a dog to skip far enough and that way to jump over large obstacles.
In the same way, you will not be able to magically transform programmers into a good testers overnight, but you can start by working with them on their weaknesses and then teaching them some simple and effective testing techniques. With good intentions, a little practice, some hand-holding, and constant feedback you will be able to get some good testing help in your project.
Step 1 – understand the limitation and weaknesses of a programmer
I started my session yesterday by going over the points I listed in my earlier blog that make programmers “less-than-perfect-testers”:
- Parental feeling towards their own code.
- Placing the focus mainly on positive scenarios, instead of actively looking for the bugs.
- Tendency to look at a “complex problem” as a collection of “smaller, simpler, and isolated cases”.
- Less End-to-End or User-Perspective oriented.
- Less experience and knowledge of the common bugs and application pitfalls.
It made a big difference during the session to not only list the weaknesses but talk about why each of them is naturally present in programmers due to the nature of their work, their knowledge and trainin
Step 2 – how to plan tests
I’ve seen that many (most?) programmers have the perception that testing requires little or no planning.
Maybe we testers are guilty of this misconception since we don’t get them involved in our test planning process as much as we should, but the truth of the matter is that when we ask them to test they they automatically grab a mouse and start pressing on the buttons without much consideration or thought.
Test Planning is a cardinal aspect of good tested, so I gave them a couple of ideas and principles planing:
- DON’T TEST YOUR OWN CODE!
When as a team they are asked to test, they should make sure to divide the tasks so that each of them is testing the code developed by other programmers as much as possible. - Work with your testing team to create test sets.
This specific team is using PractiTest (the hosted QA and Test Management platform my company is developing), but it can be any other tool or format (even word and excel!). They should sit with their testers and define what test cases need to be run, reusing the testing scripts already available in their repositories. - Expand your scenarios by making “Testing Lists”
- Features that were created or modified (directly or indirectly)
- User profiles and scenarios to be verified.
- Different environments / configurations / datasets to be tested
They help you get a better idea of what you want to test while you are running your manual test scripts, and they also serve as a verification list to consult towards the end of the tests when you are looking for additional ideas ow when you want to make sure you are not missing anything of importance. - Testing Heuristics - SFDEPOT (read San Francisco Depot)
The use of (good) heuristics greatly improve the quality of your testing. I provided the programmers with the heuristic I learned fist and still helps me up to this day. I read about SFDEPO(T) from James Bach some years ago – you can check one of the sources for it from James' Site.
SFDEPOT stands for:
Structure (what the product is)
Function (what the produce does)
Data (what it processes)
Platform (what it depends upon)
Operations (how will it be used)
Time (when will it be used)There are other heuristics and Mnemonics you can take from the Internet…
Step 3 – what to do when running tests
We talked a lot about tips to help perform good testing sessions.
- Have a notebook handy.
In it you can take notes such as testing ideas you will want to test later, bugs you ran into and want to report later in order to “not to cut your testing thoughts”, etc. - Work with extreme data.
Big files vs. Small files Equivalent input classes [ -10 ; 0 ; 1 ; 10,000,000 ; 0.5 ; not-a-number ] Dates: Yesterday, now, 10 years from now, etc. - Think about negative scenarios.
- How would Mr. Bean use your software?
- How would a hacker try to exploit the system?
- What would happen if…? (blackout, run out of space, exceptions, etc)
- What if your 2 year old would hijack the keyboard in the middle of an operation? - Focus & Defocus.
I used this technique to explain to them that during their testing process they need to make a conscious effort to always look at the bigger picture (application, system, process) and not only focus on the specific function they were testing. - Fight Inattentional Blindness.
I used the following film of the kids passing the balls to explain the concept of Inattentional Blindness and it worked great! We were 8 people in the room, and only I had seen the video before. Out of the 7 participants one is currently a tester, another one is a former tester turned programmer, the rest are “regular programmers”. The cool thing is that only the 2 testers saw the Gorilla the first time… talk about making a point!
Step 4 – what to do when (you think) you are done testing
We talked about how even when you think you are done testing you should always make sure there is nothing else that should be tested, and what techniques they can use in order to find these additional places:
- Walking the dog
Taking a break from your tests, doing another tasks for 30 – 60 minutes, and then returning to review the tests you did and what you found. This break usually helps to refresh the mind and to come up with more ideas. - Doing a walk-through session to review the tests you did.
Most of the time you will be able to get more ideas as you explain to your peers about tests you just did. The funny part is that many of these ideas will come from yourself and the things you think about when you are explaining your tests to others out loud. - Ask for ideas from other developers or testers.
Simple as it sounds, come to others in your team and ask them to pitch you ideas of stuff they think you could test. 90% of the stuff you will already have tested, but the other 10% might prove useful too!
In the end is a questions of mindset and motivation
One of the things I like most of agile teams (and many non-agile but yes smart development teams) is that they define Quality to be the responsibility of the whole team and not only of the testing guys running the tests tasks at the end of the process.
My final piece of advice to the group was that everything starts from them, and their understanding that testing is not a trivial task and definitely not a reprimand for doing a bad job or for finishing their tasks ahead of time. What’s more, I am sure that once these guys start testing better and gaining a testing perspective of their application they will also start developing better software too.
Now I have another 3 development teams I need to train. Let’s see how it goes with them
