Re-enabling right click

Recently, the right click on our front-end is disabled. Why did we decide to disable the right click?

Our application is a web application that runs on a machine. It has a touch screen. When a user presses a very long time on the screen, it means that he performs a right click on that specific spot where he clicks.

We do not want that a user sees a context menu that is always shown when right clicking. So we decided to disable that right click in our application.

But this has also a drawback. A developer or tester sometimes needs that right click. I need it when I create the selenium scripts. I need to see the path of an element. Which classes do it have and so on.

Fortunately there is a plugin in chrome that enables the right click again. The plugin is called Enable Right Click and can be found in the Chrome Web Store.


My first little pytest plugin

I test our backend via an API interface. It is a json rpc api that talks over a websocket. The test framework that I created is written in python. The testrunner is py.test. py.test already has a lot of plugins that are very handy.

All went well, until last week. The latest tests I wrote should not run as is. Some of them where tests that did a calibration. Others where tests that should run before the calibration was done.  I did find a pytest plugin that could order the tests and run them in that order. But I was not satisfied with the plugin itself. The markers where not chosen well for our project.

So I decided to have our own simple plugin that fits our needs. I wanted something like this:

This is very easy to implement. I created a file in a plugin folder with the following content:

This function is called by pytest when the collection of all tests is completed. By altering the items list, we can manipulate the tests that are running, also the order of the tests.

Now I can use the marker run_when_not_calibrated, just what I wanted.

I only needed to put the following line in the file to enable the plugin. This is needed because I did not create a setup for the plugin yet.

So with a little bit of extra code, I easily can manipulate the order of the tests.

Testers can learn from the Hunger Games

Recently I read the first book of The Hunger Games Trilogy written by Suzanne Collins. The book is also very interesting if you think a little bit deeper about it. What can we as software testers learn from the Hunger Games?

1. Explore the environment.

When Katniss, the main character in the book enters the virtual world, she did not know how it looked like. Every year the games take place in a new virtual world. What does it mean for her? She needs to know the world. Where is the water supply, where can she find food. And so on. How does she find it? By exploring that new world.
In the world of software testers, this is the same. Every new product, or even every new build, some new features can be found. The tester needs to explore that new build. How does it work? What are the mistakes? If I do this, what happens then next in the system?
Like Katniss, the tester also needs to remember what the system did on certain places. Sometimes buttons can appear and disappear, depending on the state of the program. Search for states, search for little changes that may not happen.

2. Keep living (testing)

To win the game, Katniss should listen to one advice. Just keep living. Do not let other persons kill you.

For us, software testers, this is also the case. We have to win. But when do we win? When our customer(s) is (are) satisfied with the quality of our software product we provide.

For us, this means that we need to test the product until quality is good enough. Good enough to win the trust of our customers in the product. Customers who are not happy will not return in the future.

3. Challenge the rules.

The rules of the Hunger Games are very simple. Kill each other and the last one that survives will win the game. In the very end, only two players where alive. Two from the same district. Katniss had the poison berries and gave a few berries to Peeta. They wanted to eat them together. If they did, then there should be no winner that year. At the very last moment, suddenly there was a voice yelling that this year, there are two winners. It means they defeated the rules.

Is this not like we always do? We challenge the system to see if the rules are fulfilled. Rules are in our case, the requirements. We try to get around requirements to see if there are mistakes in the requirements itself. The system can maybe put in such a state, that the system violates some requirements.

4. Rules can change.

Suddenly there was a voice. The rules of the game changes. Now two persons can win the game if they are from the same district. Katniss was happy about that. Now she does not have to kill Peeta. So this means that the strategy of Katniss is now completely different. She can look after Peeta and together they can try to survive. But in the end the rules of the game changes again. This means again a strategy change for the players.

This looks a lot like we work with our agile team. The requirements can change very fast. This mean that also the testing strategy can change and also the testing itself. What yesterday looked like a bug is now no bug anymore. Or maybe a new one appears.

We as testers in agile teams need to adapt to changes. Try to do it, then you will survive.

How I setup our test documentation

What are you going to test for this story?

That was a question that I ask myself a lot. This is a question that needs to be answered for every user story that I test.

We do have a lot of automated tests. Tests that are written in python. It are tests on API level, so without interaction of the user interface.  I also have to test manually. Not every part of our huge system can be automated. For manual testing, I created some test documentation. That documentation contains what will be tested. I do not just write in detail what I look for, but in general. There should be room for doing crazy stuff while testing too.

How should the system react? What should the software do? These are some of the questions that are answered in the testing documentation.


The first version of this test documentation was a document written in word. It worked, because I was the only one who edited the document. When the company grew, this was also not suitable anymore. I and my colleagues can not work on that document at the same time. That is the reason that I looked at a new format for my test documentation.

What did I want?

  • It must be simple to edit.
  • No code language like html or so. Preferable in markdown or another simple format.
  • Different users should be able to edit the same document at the same time.
  • It should be placed in version control, so that we can merge it, like other code.

The new documentation

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

Our testing code was already in python. Python has some nice documentation. Why not use the same documentation framework as the python community uses? That could be a good alternative. I found the Sphinx project. This project seems to be what I wanted. On the main page the first sentence was promising. It talks about easy to create and beautiful documentation.

The first thing I did was installing Sphinx. So I entered:

That was an easy one. The next one was in an empty directory, I typed the command

That is also explained in the First Steps With Sphinx Tutorial.
This sphinx-quickstart program asked me a lot of questions. I answered them one by one. One of the last question was if it should generate a make file. I answered yes. Now I do not need to remember some large command like:

Now the program has generated some files, that can be placed in version control. Building the documentation is very simple:

Test documentation itself needs to be in reStructuredText format. It is also very easy to learn. It is nearly the same as markdown. More information on reStructuredText can be found on their webpage.

Because now the complete test documentation is in text files, it is possible to store it in version control. We use git in our company, so that infrastructure can be re-used. I also make use of our jenkins build server. This build server generates the documentation for us after each check-in. And that documentation is afterwards copied to a web server. Now everybody can have a look at what is tested at any time.

It is a system that is easy to use now, and it is transparent for everybody. Isn’t that what I wanted?

Bugs should be stories too!

The application we are building is not so fast anymore. So I entered an issue in our bug tracking database. After a while the bug is prioritised by our product owner and later in time the bug is placed on scrum board. Good practice you say. Nobody likes known bugs, they should be solved as fast as possible.
The fix for this bug was not that difficult. At least for the programming side. It just was to make some calls asynchronous and the complete program behaves faster. There was also some little impact in testing, because only a few interface functions where changed.
While fixing the tests, we discovered that everything was impacted. Our system behaved completely different as before. It was not wrong, but our test system needed to update.
Why is fixing the test code taking so long? It is not that difficult? No?
No, it is not difficult to fix the tests, but it is a lot of work. So the planning for this task is not correct.
What do we do if a bug gets on our board. We never refine bugs in a refinement meeting like the other stories. We only put them on our board. Look if there seem to be a lot of fixing time. Then we put 2 story points on it. If there is nearly no fixing time, it gets 1 story point.
I am not a fan of this approach. I do want to see the bugs, if it are not quick fixes, in a refinement meeting. They can have a lot of impact. Maybe the impact is only in the implementation. Maybe the effort is in our regression tests.
In my point of view, a bug is just like a user story. So we should threat them as stories too.

Demonstrating incomplete work sucks

The current sprint is ended. Our team hold the sprint review. At the sprint review, we review the work that was completed and the planned work that was not completed to the stakeholders.

The last weeks, our reviews are a little bit hectic. This time, things where not different. I tested yesterday some features and they seem to work, with some minor bugs in it. If I was a stakeholder, I could live with those minor issues. So for me it was simple. We could close some stories and enter extra bugs in our database and go on.

This is for me a better scenario than what we encounter in our review meeting. The demonstration was not done on the version I took while testing, but a next version. “Because some more bugs are fixed”. That is fine for me. I really like that more bugs are fixed.

Unfortunately the new changes introduced some other things that where broken. We did not automate everything in our build pipeline. For example, we do not have automatic user interface tests. If something breaks the user interface, it is not detected until it is seen with manual testing. There are other parts of our system that needs manual testing too. The main reason they are not yet automated is a lack of time.

You could now ask to me: “Why did you allow that the new version is taken for demo purposes?” That is because I am one tester in a team. But we do have 3 teams that are working on the same product. All three teams also do have a review on the same moment on the same machine. So I do also have to trust the other teams that they tested the changes they made. I do have to trust them that they did not break anything.

But they did break some things. At this moment I am a little bit afraid. Afraid that changes that a team makes, brakes some features of another team. We have 3 teams. Each team has one tester that helps the team. But as tester, we help the team to test stories. We do not have sufficient time to check everything on lets call it epic level. At this moment, I am thinking that the testers do not belong in the team. At this moment, I think that the testers should test more on epic level.

We have a scaling problem. How do we scale scrum to more teams? All interesting questions we should solve one by one. I already read about scrum of scrums, but this does not say how testing the integration points. And more, how can the testers help?

How do other teams tackle this kind of problem?

Dear team, we have a problem.

I noticed a few days that nobody of our team is watching to the outcome of the automated integration tests. Our tests run on an integration server. Since the beginning of our sprint, one test is failing. The result is that the build was not a single day in our sprint green.

failingThe first day of our sprint, one of our tests broke. The test was already running for several months without failing. One of the stories had impact on code that could break this test. In normal circumstances, the fix is there within a few hours. That is fine. Green tests give confidence.
This time, things are different. The test is failing until the story that has impact on it, is implemented. But still today, the day before the review meeting, the test is still failing. And indeed, the story is not finished yet.

This is a problem for me. More teams are working on the same repository. The other teams also rely on the integration tests. Their safety net is gone already for at least 2 weeks. Of course they can have a look at the status of the testing job and see that only that test is failing. But because it is taking such a long time now, nobody looks at it at this moment.

How can we solve this problem?

  • Disable the test?
  • Ignore the test result until it is fixed?
  • Mark the test as unstable until it is fixed?

What option is chosen is not important. Only, ignoring the test results as it is today is a risk. If something else fails, nobody will notice it. And that is just not what we want.

So next retrospective this problem will be on the table.

Forgotten tests

Time is passing by. So are our software versions. The developers in my team are creating a lot of code. That is good. We want a good complete product that works, don’t we? The team is an agile team. The lonely tester in that team is me. Agile you say? This means that the team tests every story?

I can not answer the question with a simple yes or no. It is a little bit complicated. The team is trying to test all stories. Still some gaps are missing. I try to explain what I mean.

In the beginning of the project we were a small team with 5 developers and 1 tester. We started from nothing. That means that we created a continuous integration server. We also created a test framework. In that framework we can test our running back-end. The back-end communicates with hardware. For this out team created a simulator, to simulate the hardware. There is also a user interface that interacts via an API with that back-end. The tests in this framework are using that API. We call the tests our integration tests. The name does not matter.

Then a few months later, the team expanded. Then we where with more than 12 team members. The conclusion is simple at first. Create 2 teams out of one big team. I already saw a problem with the current situation. There is only one tester. The other team had no tester. We saw it not as a huge problem because the teams are responsible for quality and not only 1 person.

That is the theory at least. In practice it is more difficult. One of the problems we are now dealing with is the fact that some stories have impact on the other team. Our team is for the moment blocked on the other team. Not blocked on creating new code, but blocked on testing. Why are we blocked? Because we focus on stories and not on testing the complete system. It should be, but it is at this moment becoming a huge work to test all. We have to make choices and test only subsets. I think that this problem also occurs if we are in one team, so it was only a matter of time when this occurs. How should we deal with system testing? The testing of the complete system?

In waterfall that part is easy. We create releases and test on that release. Now we do have a release after each story. We just can not test everything every story. This means that we forget those “release” tests in the pile of work we have. That is our problem at the moment. We still have no real releases.

How do you deal with this problem?


Brainstorming, it is a very common thing that is used in agile teams. I never felt very comfortable in that process. I think I know why now.

Brainstorming, it was an idea of Alex Osborn. He found that his employees did have fantastic ideas at that moment, but they where not creative enough. And if they had fantastic and creative ideas, they did not tell it to the colleagues. Why? Because they where afraid for criticism. That is why he kind of invented the brainstorming process. He was looking for rules that could help give people freedom of mind to reveal new ideas. The four rules are:

  • No criticism of ideas
  • Wilder ideas are better, try to exaggerate
  • Go for larger quantities of ideas, not for quality
  • Build on each others ideas

His idea was that if a group of people follows these rules, they create a lot of great ideas, more and better than if you do it separately. This process became after a lot of years more or less common practice and also in agile brainstorming is used a lot. Rooms filled with pictures on whiteboards or paper, who does not have seen that yet?

But is it like this? Is brainstorming better? In 1963, Marvin Dunnette published a scientific paper that proved the opposite.

He let 24 groups of 4 people brainstorm on several problems. The groups generated a lot of ideas. Afterwards, the participants had to brainstorm on similar problems on their one. To simulate that they also form a group, the ideas of the original group members where added together. So he could compare it. The results where amazing. The individuals produced more ideas than they did if they worked in group. And they did not only generate more ideas, the quality was similar. From 24 groups, 23 groups produced a larger number of different ideas if they worked alone.

Later in history, this experiment was confirmed by other experiments. So it is very curious that brainstorming is still that popular. Brainstorming does not improve creativity. It does not generate the fantastic ideas. It has however his good parts. It improves the social cohesion in the group, so do not throw brainstorming away, it can still be useful. The experiment of Marvin Dunnette did also mention that if first a brainstorming in group is done, and afterwards another individually, that the outcome is much higher.

Brainstorming can also be creative if it is done in another way. If it is done online, for example via a chat group or via mail. So next time if we do want some great ideas, try to do it in a slack chat room. But it is up to your team if you use such chat rooms or not.

Valuable testing oracles you need to know as software tester

Yesterday, there was a discussion about our testing framework. The discussion ended soon in discussing testing oracles. I create this post to help with that discussion.

What is a test oracle?

Everybody knows about the Sphinx, the oracle in an ancient times. Oedipus went to Thebes, and the Sphinx asked him her riddle. He was able to solve it.  But that is not the oracle I am talking about. In the testing world of bits and bytes, we can state the following. A testing oracle is a mechanism by which we determine if the software under test is behaving reasonably.

An oracle is a mechanism for determining whether the program has passed or failed a test.
Cem Kamer

Cem Kamer has another definition, but it is in fact still the same definition.  The discussion yesterday was more or less about one type of testing oracle. There are many oracles. I will explain a few of them in this post.

No Oracle

The first oracle that I found is a simple one. Use no oracle. Just run the program you are testing, do not check for correctness but only check if it crashes. You can do this by hand . Just take random keyboard inputs. Call functions via an API and do not check the return values.

This oracle is easy to create and runs fast. Only running this oracle is not enough. This oracle only detects spectacular things like crashes. Be sure, this oracle is not the only oracle you use.

Independent Implementation

Create a separate program that generates the expected results. The test checks if the outcome of the oracle and the system are the same. This oracle is completely customizable for the system under test, so the results can be exact. The disadvantage is that it can be expensive to create.

Consistency oracle

This is an oracle that is also not difficult to create. The oracle compares the results of the system with a “golden master” or a reference. It detects that results change. The oracle can check without knowing what the results should be. Like every oracle, also here are some disadvantages. One of them is that if the system has a legacy bug. It is not discovered, and it can give a false feeling of confidence. This kind of oracle is good in conjunction with unit tests that are verifying an algorithm.

In our case, we have an algorithm in our software that calculates values. The software sends these values to hardware. Unit tests should verify if the algorithm is correct. This means that the integration tests should not verify the correctness of the algorithm. The integration test checks if software sends a value to hardware. The test stores that value to check for it in the next run.

Only when the algorithm changes the test result changes too. This oracle is good for regression testing.

Constraint based oracle

Sometimes a test validates simple valid or invalid combinations. This oracle is the one for such tests. One example is for example that the test can verify if a user’s birth date is before the date he started working.

These are easy checks. Implementing these checks has a low cost. The test code can become invasive. That is one drawback.


Test oracles are useful while testing. There are many of them. I only did mentioned a few of them. Using a few of them will improve testing.  Only using one of them is not a good idea, every oracle has some disadvantages. One oracle to rule them all does not apply here.