How I setup our test documentation

What are you going to test for this story?

That was a question that I ask myself a lot. This is a question that needs to be answered for every user story that I test.

We do have a lot of automated tests. Tests that are written in python. It are tests on API level, so without interaction of the user interface.  I also have to test manually. Not every part of our huge system can be automated. For manual testing, I created some test documentation. That documentation contains what will be tested. I do not just write in detail what I look for, but in general. There should be room for doing crazy stuff while testing too.

How should the system react? What should the software do? These are some of the questions that are answered in the testing documentation.


The first version of this test documentation was a document written in word. It worked, because I was the only one who edited the document. When the company grew, this was also not suitable anymore. I and my colleagues can not work on that document at the same time. That is the reason that I looked at a new format for my test documentation.

What did I want?

  • It must be simple to edit.
  • No code language like html or so. Preferable in markdown or another simple format.
  • Different users should be able to edit the same document at the same time.
  • It should be placed in version control, so that we can merge it, like other code.

The new documentation

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

Our testing code was already in python. Python has some nice documentation. Why not use the same documentation framework as the python community uses? That could be a good alternative. I found the Sphinx project. This project seems to be what I wanted. On the main page the first sentence was promising. It talks about easy to create and beautiful documentation.

The first thing I did was installing Sphinx. So I entered:

That was an easy one. The next one was in an empty directory, I typed the command

That is also explained in the First Steps With Sphinx Tutorial.
This sphinx-quickstart program asked me a lot of questions. I answered them one by one. One of the last question was if it should generate a make file. I answered yes. Now I do not need to remember some large command like:

Now the program has generated some files, that can be placed in version control. Building the documentation is very simple:

Test documentation itself needs to be in reStructuredText format. It is also very easy to learn. It is nearly the same as markdown. More information on reStructuredText can be found on their webpage.

Because now the complete test documentation is in text files, it is possible to store it in version control. We use git in our company, so that infrastructure can be re-used. I also make use of our jenkins build server. This build server generates the documentation for us after each check-in. And that documentation is afterwards copied to a web server. Now everybody can have a look at what is tested at any time.

It is a system that is easy to use now, and it is transparent for everybody. Isn’t that what I wanted?

Bugs should be stories too!

The application we are building is not so fast anymore. So I entered an issue in our bug tracking database. After a while the bug is prioritised by our product owner and later in time the bug is placed on scrum board. Good practice you say. Nobody likes known bugs, they should be solved as fast as possible.
The fix for this bug was not that difficult. At least for the programming side. It just was to make some calls asynchronous and the complete program behaves faster. There was also some little impact in testing, because only a few interface functions where changed.
While fixing the tests, we discovered that everything was impacted. Our system behaved completely different as before. It was not wrong, but our test system needed to update.
Why is fixing the test code taking so long? It is not that difficult? No?
No, it is not difficult to fix the tests, but it is a lot of work. So the planning for this task is not correct.
What do we do if a bug gets on our board. We never refine bugs in a refinement meeting like the other stories. We only put them on our board. Look if there seem to be a lot of fixing time. Then we put 2 story points on it. If there is nearly no fixing time, it gets 1 story point.
I am not a fan of this approach. I do want to see the bugs, if it are not quick fixes, in a refinement meeting. They can have a lot of impact. Maybe the impact is only in the implementation. Maybe the effort is in our regression tests.
In my point of view, a bug is just like a user story. So we should threat them as stories too.