Brainstorming, it is a very common thing that is used in agile teams. I never felt very comfortable in that process. I think I know why now.

Brainstorming, it was an idea of Alex Osborn. He found that his employees did have fantastic ideas at that moment, but they where not creative enough. And if they had fantastic and creative ideas, they did not tell it to the colleagues. Why? Because they where afraid for criticism. That is why he kind of invented the brainstorming process. He was looking for rules that could help give people freedom of mind to reveal new ideas. The four rules are:

  • No criticism of ideas
  • Wilder ideas are better, try to exaggerate
  • Go for larger quantities of ideas, not for quality
  • Build on each others ideas

His idea was that if a group of people follows these rules, they create a lot of great ideas, more and better than if you do it separately. This process became after a lot of years more or less common practice and also in agile brainstorming is used a lot. Rooms filled with pictures on whiteboards or paper, who does not have seen that yet?

But is it like this? Is brainstorming better? In 1963, Marvin Dunnette published a scientific paper that proved the opposite.

He let 24 groups of 4 people brainstorm on several problems. The groups generated a lot of ideas. Afterwards, the participants had to brainstorm on similar problems on their one. To simulate that they also form a group, the ideas of the original group members where added together. So he could compare it. The results where amazing. The individuals produced more ideas than they did if they worked in group. And they did not only generate more ideas, the quality was similar. From 24 groups, 23 groups produced a larger number of different ideas if they worked alone.

Later in history, this experiment was confirmed by other experiments. So it is very curious that brainstorming is still that popular. Brainstorming does not improve creativity. It does not generate the fantastic ideas. It has however his good parts. It improves the social cohesion in the group, so do not throw brainstorming away, it can still be useful. The experiment of Marvin Dunnette did also mention that if first a brainstorming in group is done, and afterwards another individually, that the outcome is much higher.

Brainstorming can also be creative if it is done in another way. If it is done online, for example via a chat group or via mail. So next time if we do want some great ideas, try to do it in a slack chat room. But it is up to your team if you use such chat rooms or not.

Mutable Default Arguments in python

Our team has some simulators and integration tests written in python. For the most part, Python is a simple language that avoid surprises. That is why we took this language as the language used for testing. There are some few situations that can be confusing though. One of the most common surprise are the default arguments. More specific, mutable default arguments.

Suppose you have a function like the following:

Of course, you can call this function with or without any parameters. What happens in the following situation?

You expect that a new list is created each time the function is called, because the second parameter is not provided. So you expect the following output:

But that is not the case. What you see on your console is the following:

A new list is created only once. It is only created on definition time of the function and not on calling time like in other languages. This means that if you mutated the object, you also have mutated it for future usages as well.
We should create an object each time the function is called. This can be done by adding another default argument that signals us, that there is no argument provided.

And now the output is the one we did expect.

Valuable testing oracles you need to know as software tester

Yesterday, there was a discussion about our testing framework. The discussion ended soon in discussing testing oracles. I create this post to help with that discussion.

What is a test oracle?

Everybody knows about the Sphinx, the oracle in an ancient times. Oedipus went to Thebes, and the Sphinx asked him her riddle. He was able to solve it.  But that is not the oracle I am talking about. In the testing world of bits and bytes, we can state the following. A testing oracle is a mechanism by which we determine if the software under test is behaving reasonably.

An oracle is a mechanism for determining whether the program has passed or failed a test.
Cem Kamer

Cem Kamer has another definition, but it is in fact still the same definition.  The discussion yesterday was more or less about one type of testing oracle. There are many oracles. I will explain a few of them in this post.

No Oracle

The first oracle that I found is a simple one. Use no oracle. Just run the program you are testing, do not check for correctness but only check if it crashes. You can do this by hand . Just take random keyboard inputs. Call functions via an API and do not check the return values.

This oracle is easy to create and runs fast. Only running this oracle is not enough. This oracle only detects spectacular things like crashes. Be sure, this oracle is not the only oracle you use.

Independent Implementation

Create a separate program that generates the expected results. The test checks if the outcome of the oracle and the system are the same. This oracle is completely customizable for the system under test, so the results can be exact. The disadvantage is that it can be expensive to create.

Consistency oracle

This is an oracle that is also not difficult to create. The oracle compares the results of the system with a “golden master” or a reference. It detects that results change. The oracle can check without knowing what the results should be. Like every oracle, also here are some disadvantages. One of them is that if the system has a legacy bug. It is not discovered, and it can give a false feeling of confidence. This kind of oracle is good in conjunction with unit tests that are verifying an algorithm.

In our case, we have an algorithm in our software that calculates values. The software sends these values to hardware. Unit tests should verify if the algorithm is correct. This means that the integration tests should not verify the correctness of the algorithm. The integration test checks if software sends a value to hardware. The test stores that value to check for it in the next run.

Only when the algorithm changes the test result changes too. This oracle is good for regression testing.

Constraint based oracle

Sometimes a test validates simple valid or invalid combinations. This oracle is the one for such tests. One example is for example that the test can verify if a user’s birth date is before the date he started working.

These are easy checks. Implementing these checks has a low cost. The test code can become invasive. That is one drawback.


Test oracles are useful while testing. There are many of them. I only did mentioned a few of them. Using a few of them will improve testing.  Only using one of them is not a good idea, every oracle has some disadvantages. One oracle to rule them all does not apply here.

How I clone very large repositories with git

We have problems lately to clone a repository from our git server. It seems that the one of our repositories is too big to clone from scratch. We are suffering with network timeouts. We use git as version control, so it is not a problem. There are alternatives.

  1. You can copy the closed source directory from a colleague.
  2. You can clone it from a bundle.

This post is about the second option. I am a fan of bundling a repository to share it. How does that work?

Person A has already cloned a repository of the project on his system. What should he do?

Git created a file called closed-source.bundle in the current directory. This file is a copy of the complete repository now. It is a single file. Person A can share this file via a network drive. Person B can then copy that file and just take a clone of it.

Person B has now a checked out version of the project with a remote that points to the bundle. Remove that remote and set to the correct location.

Now person B can work like he just cloned it directly from the remote git server.

Found an awesome tool to create wonderful screen captures

One of the next challenges in the 30 days of testing challenge is to find and use a new tool. I already had some tests with a new tool like I explained in another blog post. I also told yesterday that I make for bugs some screen movies to explain what happens to the developers. A picture says more than thousand words.

I use a tool to create nice screen captures. It is LICEcap. That tool records some animated gif. I always use the animated gif and not the lcf format.

ScreenShot00219When opening the application, you can see a window with a transparent layer. LICEcap records what occurs in this part of the screen.

ScreenShot00218There is a button on the bottom right with the text Record that you click when starting a new screen capture. Then you see a dialog, where you can put some options and a name for the output file. Just pressing save in the dialog and the tool starts with recording what you are doing.

A nice tool that makes my life better. Developers understand the bug reports better when they see a nice animated gif.

How to write a fantastic bug report

Today I found a question in my mailbox.

Bart, I am solving a bug report. According to developer X (who created the bug report) you said that ZZZ happen. How can I reproduce this?

A normal question, no? In this particular case it is not.

Last week I got some feedback from one of our stakeholders. I was testing a feature that was failing. We call the feature that was failing feature simulation in our application. We also have criterion simulation and criterion trigger. The stakeholder told me that criterion trigger did not work. I still needed to investigate it. I believe in facts, and not what some stakeholder tells me that happens.
Later that day, I told our scrum master what I was doing and what I discovered so far. I told him that I discovered that the simulation was not working anymore. I also told him that one of our stakeholder had an issue with triggering.

Then it was time to leave for me. The weekend passes. On Monday I looked at our electronic scrum board. I saw that someone added a new story was to our scrum board! Our sprint ends in two days. We did not finish all our planned stories yet. How is this possible? What has happened?

The new story is a bug report. The bug report is a strange one. It does not contain any description. It only contains a title.

What do I write in a bug report?

  • A good title:
    • The title summarizes the problem as best as possible. Because it it often put in bug lists, it is important to be as specific as possible. But also not too long.
  • A description:
    • The description describes the problem as it happens. I always place steps to reproduce the problem in a description. When reading those steps, the developers should be able to reproduce the problem. This is why I never create a new report on rumors. I also attach a picture or a little screencast to the bug report. Our bug reports have some meta data. There are fields for severity or the status of that bug. What is lacking is a field to fill in a software version. The software version is also present in the description of the bug report

The product owner decides afterwards what the priority of this new bug will be. He talks to know the impact to several people about this new bug.

Remember that a bug report is to help the developers to fix a problem. Developers can fix problems faster if the description of that problem is better described.

Sharing a quote

I found a quote that  can be applied both in my personal life and in my life as software tester.

When nothing is sure,
everything is possible.

It applies in my personal life. I do long distance trails. This weekend I took for a walk of 50 km. In august, it will be 100 km.

But it also applies to software development and testing. Sometimes I do not feel confident about the quality of a software version. Suddenly, it gets better and better. It is also possible that it get worse.

Automatic testing framework

We have a build chain. After each code change in our version control (git) there is a build on our continuous integration server. This build creates each time an installable executable and runs some integration tests.

The integration tests will interact with our backend. We have a frontend and a backend. That backend talks to another service, plc’s and other hardware. For the moment there is not yet communication with PLC’s implemented, but it will be soon. For the other hardware, I wrote a hardware simulator in python. I did prefer erlang, but my colleagues objected about it, because the erlang is too difficult. I find that a stupid reason, but not worth arguing about. So I wrote the simulator in python. It was not that hard, because we have some commands that just interrogates some memory in hardware. If a register in memory is set to a value, the real hardware interacts with it. My simulator does not interact with all registers, only with the registers that have state in it.

After I wrote the simulator, I wrote some tests. Integration tests that calls the backend. Why not the frontend? Because it is easier to script calls to the backend, and they are less error prone. At first I choose for the robot framework. This is a keyword driven framework written in python. All tests can be put in version control, so this means that we also can do code reviews and other stuff that the developers do.

I did write the tests in robot framework, but the implementation of the keywords did have an implementation in a library. And now I am happy that I did it this way. Because at a retrospective in the past, we decided that our developers should also write integration tests. The objection was, 2 languages for testing? Can it not be only in python? So I searched a little bit and found py.test. This unit test framework can also run other tests that run longer. It is a nice alternative and the impact to convert from robot to py.test was not that hard. Only the tests itself must be rewritten, because I did implement the driver code already in python. A few sprints later, we had a decent testing framework. And I am also happy, because every team member can now write integration tests, and they are, so I am a happy tester now.

Challenge 11 and 12 ready

A few days that I had no time at all to do anything for the daily challenge. However, I also can say that I performed 2 tasks. Be it, that I posted it on twitter. The Challenge of yesterday was easy.

11: Take A picture of your team.

So I took a picture of our team while we where in our planning and design session. That was an easy task and I posted it on twitter. This is the picture that I took.

teamIt was not our complete team, because we are in holiday season this and next month in Belgium, but half a team is also good. In a few weeks, our team is also divided into two, because we are too big to be efficient. Now we are with 12 people. Way to big to be successful. We are hiring an extra software tester and then we can move on with two complete teams. For the moment we are lacking a software tester.

The next challenge is also not that difficult. It is:

12: Doodle a problem.

We have a problem. Part of our team is in another location and therefore it is not easy to have communication. Therefore I doodled a problem that I see at this moment and also in the near future when we are divided in two or maybe three teams. One team will be on a different location as the other, so I see a problem there with inter-team communication.


And it is true, we are already working with slack, having skype meetings and so. But still, it is not so easy to talk to each other if you are in a different location. I always go to the other location on Fridays. That is helping, but not enough.

Found accessibility bug with a new tool

Yesterday, it was a very busy day with very limited access to my desk. So I was not able to continue with the challenge of the day. This morning I saw a tweet of Kim Engel about day 7 of the challenge.

color_tweetI did not know that extension, so I did have a look at it and installed it in my Chrome browser. After the extension is installed, a little color wheel is present in the toolbox area of my browser. The first thing I did is clicking on it. I saw some weird names that I did not recognize at first. But after a little searching, I found a article on Wikipedia about color blindness. The weird names where also mentioned there, so this are types of color blindness.

color_blindnessI never did think about color blindness when testing. But is can be a problem, specially for males. Because our machine is mostly operated by males, it is worth looking at it. The red-green color blindness is between 7 and 10 per cent of the male population in most countries according to the article.

To look what the spectrum extension does, I googled an image with a colored wheel on it. And I played a little bit with the check boxes.

Original color wheel
Original color wheel

Let’s try Deuteranomaly. Persons with deuteranomaly have a mutated form of the medium-wavelength (green) pigment.

The red is completely gone and it became a kind of brown. Also some colors in the green spectrum have changed.

Interesting, because in our interface we have a feature that we can simulate what our hardware is going to do. In this way, our customers can setup the machine without production first and simulate what is going to happen. But what is detected is put on an overlay with green color, and the color can not change.

That is a problem if you can not see that kind of green, specially if you are detecting colors that are displayed in the yellow spectrum, because those people see it as the same color.

30days_day7Thanks to the challenge, I found a new tool and found some bugs. This means that I now completed Challenge 7 and 19.

Challenge 7 was for me an eye-opener. It is not because I can see some items on the screen, that everybody sees the same items in the same way. I think that the spectrum extension will be used a lot more in my testing activities.