Valuable testing oracles you need to know as software tester

Yesterday, there was a discussion about our testing framework. The discussion ended soon in discussing testing oracles. I create this post to help with that discussion.

What is a test oracle?

Everybody knows about the Sphinx, the oracle in an ancient times. Oedipus went to Thebes, and the Sphinx asked him her riddle. He was able to solve it.  But that is not the oracle I am talking about. In the testing world of bits and bytes, we can state the following. A testing oracle is a mechanism by which we determine if the software under test is behaving reasonably.

An oracle is a mechanism for determining whether the program has passed or failed a test.
Cem Kamer

Cem Kamer has another definition, but it is in fact still the same definition.  The discussion yesterday was more or less about one type of testing oracle. There are many oracles. I will explain a few of them in this post.

No Oracle

The first oracle that I found is a simple one. Use no oracle. Just run the program you are testing, do not check for correctness but only check if it crashes. You can do this by hand . Just take random keyboard inputs. Call functions via an API and do not check the return values.

This oracle is easy to create and runs fast. Only running this oracle is not enough. This oracle only detects spectacular things like crashes. Be sure, this oracle is not the only oracle you use.

Independent Implementation

Create a separate program that generates the expected results. The test checks if the outcome of the oracle and the system are the same. This oracle is completely customizable for the system under test, so the results can be exact. The disadvantage is that it can be expensive to create.

Consistency oracle

This is an oracle that is also not difficult to create. The oracle compares the results of the system with a “golden master” or a reference. It detects that results change. The oracle can check without knowing what the results should be. Like every oracle, also here are some disadvantages. One of them is that if the system has a legacy bug. It is not discovered, and it can give a false feeling of confidence. This kind of oracle is good in conjunction with unit tests that are verifying an algorithm.

In our case, we have an algorithm in our software that calculates values. The software sends these values to hardware. Unit tests should verify if the algorithm is correct. This means that the integration tests should not verify the correctness of the algorithm. The integration test checks if software sends a value to hardware. The test stores that value to check for it in the next run.

Only when the algorithm changes the test result changes too. This oracle is good for regression testing.

Constraint based oracle

Sometimes a test validates simple valid or invalid combinations. This oracle is the one for such tests. One example is for example that the test can verify if a user’s birth date is before the date he started working.

These are easy checks. Implementing these checks has a low cost. The test code can become invasive. That is one drawback.


Test oracles are useful while testing. There are many of them. I only did mentioned a few of them. Using a few of them will improve testing.  Only using one of them is not a good idea, every oracle has some disadvantages. One oracle to rule them all does not apply here.

How I clone very large repositories with git

We have problems lately to clone a repository from our git server. It seems that the one of our repositories is too big to clone from scratch. We are suffering with network timeouts. We use git as version control, so it is not a problem. There are alternatives.

  1. You can copy the closed source directory from a colleague.
  2. You can clone it from a bundle.

This post is about the second option. I am a fan of bundling a repository to share it. How does that work?

Person A has already cloned a repository of the project on his system. What should he do?

Git created a file called closed-source.bundle in the current directory. This file is a copy of the complete repository now. It is a single file. Person A can share this file via a network drive. Person B can then copy that file and just take a clone of it.

Person B has now a checked out version of the project with a remote that points to the bundle. Remove that remote and set to the correct location.

Now person B can work like he just cloned it directly from the remote git server.

Found an awesome tool to create wonderful screen captures

One of the next challenges in the 30 days of testing challenge is to find and use a new tool. I already had some tests with a new tool like I explained in another blog post. I also told yesterday that I make for bugs some screen movies to explain what happens to the developers. A picture says more than thousand words.

I use a tool to create nice screen captures. It is LICEcap. That tool records some animated gif. I always use the animated gif and not the lcf format.

ScreenShot00219When opening the application, you can see a window with a transparent layer. LICEcap records what occurs in this part of the screen.

ScreenShot00218There is a button on the bottom right with the text Record that you click when starting a new screen capture. Then you see a dialog, where you can put some options and a name for the output file. Just pressing save in the dialog and the tool starts with recording what you are doing.

A nice tool that makes my life better. Developers understand the bug reports better when they see a nice animated gif.

How to write a fantastic bug report

Today I found a question in my mailbox.

Bart, I am solving a bug report. According to developer X (who created the bug report) you said that ZZZ happen. How can I reproduce this?

A normal question, no? In this particular case it is not.

Last week I got some feedback from one of our stakeholders. I was testing a feature that was failing. We call the feature that was failing feature simulation in our application. We also have criterion simulation and criterion trigger. The stakeholder told me that criterion trigger did not work. I still needed to investigate it. I believe in facts, and not what some stakeholder tells me that happens.
Later that day, I told our scrum master what I was doing and what I discovered so far. I told him that I discovered that the simulation was not working anymore. I also told him that one of our stakeholder had an issue with triggering.

Then it was time to leave for me. The weekend passes. On Monday I looked at our electronic scrum board. I saw that someone added a new story was to our scrum board! Our sprint ends in two days. We did not finish all our planned stories yet. How is this possible? What has happened?

The new story is a bug report. The bug report is a strange one. It does not contain any description. It only contains a title.

What do I write in a bug report?

  • A good title:
    • The title summarizes the problem as best as possible. Because it it often put in bug lists, it is important to be as specific as possible. But also not too long.
  • A description:
    • The description describes the problem as it happens. I always place steps to reproduce the problem in a description. When reading those steps, the developers should be able to reproduce the problem. This is why I never create a new report on rumors. I also attach a picture or a little screencast to the bug report. Our bug reports have some meta data. There are fields for severity or the status of that bug. What is lacking is a field to fill in a software version. The software version is also present in the description of the bug report

The product owner decides afterwards what the priority of this new bug will be. He talks to know the impact to several people about this new bug.

Remember that a bug report is to help the developers to fix a problem. Developers can fix problems faster if the description of that problem is better described.

Sharing a quote

I found a quote that  can be applied both in my personal life and in my life as software tester.

When nothing is sure,
everything is possible.

It applies in my personal life. I do long distance trails. This weekend I took for a walk of 50 km. In august, it will be 100 km.

But it also applies to software development and testing. Sometimes I do not feel confident about the quality of a software version. Suddenly, it gets better and better. It is also possible that it get worse.

Automatic testing framework

We have a build chain. After each code change in our version control (git) there is a build on our continuous integration server. This build creates each time an installable executable and runs some integration tests.

The integration tests will interact with our backend. We have a frontend and a backend. That backend talks to another service, plc’s and other hardware. For the moment there is not yet communication with PLC’s implemented, but it will be soon. For the other hardware, I wrote a hardware simulator in python. I did prefer erlang, but my colleagues objected about it, because the erlang is too difficult. I find that a stupid reason, but not worth arguing about. So I wrote the simulator in python. It was not that hard, because we have some commands that just interrogates some memory in hardware. If a register in memory is set to a value, the real hardware interacts with it. My simulator does not interact with all registers, only with the registers that have state in it.

After I wrote the simulator, I wrote some tests. Integration tests that calls the backend. Why not the frontend? Because it is easier to script calls to the backend, and they are less error prone. At first I choose for the robot framework. This is a keyword driven framework written in python. All tests can be put in version control, so this means that we also can do code reviews and other stuff that the developers do.

I did write the tests in robot framework, but the implementation of the keywords did have an implementation in a library. And now I am happy that I did it this way. Because at a retrospective in the past, we decided that our developers should also write integration tests. The objection was, 2 languages for testing? Can it not be only in python? So I searched a little bit and found py.test. This unit test framework can also run other tests that run longer. It is a nice alternative and the impact to convert from robot to py.test was not that hard. Only the tests itself must be rewritten, because I did implement the driver code already in python. A few sprints later, we had a decent testing framework. And I am also happy, because every team member can now write integration tests, and they are, so I am a happy tester now.

Challenge 11 and 12 ready

A few days that I had no time at all to do anything for the daily challenge. However, I also can say that I performed 2 tasks. Be it, that I posted it on twitter. The Challenge of yesterday was easy.

11: Take A picture of your team.

So I took a picture of our team while we where in our planning and design session. That was an easy task and I posted it on twitter. This is the picture that I took.

teamIt was not our complete team, because we are in holiday season this and next month in Belgium, but half a team is also good. In a few weeks, our team is also divided into two, because we are too big to be efficient. Now we are with 12 people. Way to big to be successful. We are hiring an extra software tester and then we can move on with two complete teams. For the moment we are lacking a software tester.

The next challenge is also not that difficult. It is:

12: Doodle a problem.

We have a problem. Part of our team is in another location and therefore it is not easy to have communication. Therefore I doodled a problem that I see at this moment and also in the near future when we are divided in two or maybe three teams. One team will be on a different location as the other, so I see a problem there with inter-team communication.


And it is true, we are already working with slack, having skype meetings and so. But still, it is not so easy to talk to each other if you are in a different location. I always go to the other location on Fridays. That is helping, but not enough.

Found accessibility bug with a new tool

Yesterday, it was a very busy day with very limited access to my desk. So I was not able to continue with the challenge of the day. This morning I saw a tweet of Kim Engel about day 7 of the challenge.

color_tweetI did not know that extension, so I did have a look at it and installed it in my Chrome browser. After the extension is installed, a little color wheel is present in the toolbox area of my browser. The first thing I did is clicking on it. I saw some weird names that I did not recognize at first. But after a little searching, I found a article on Wikipedia about color blindness. The weird names where also mentioned there, so this are types of color blindness.

color_blindnessI never did think about color blindness when testing. But is can be a problem, specially for males. Because our machine is mostly operated by males, it is worth looking at it. The red-green color blindness is between 7 and 10 per cent of the male population in most countries according to the article.

To look what the spectrum extension does, I googled an image with a colored wheel on it. And I played a little bit with the check boxes.

Original color wheel
Original color wheel

Let’s try Deuteranomaly. Persons with deuteranomaly have a mutated form of the medium-wavelength (green) pigment.

The red is completely gone and it became a kind of brown. Also some colors in the green spectrum have changed.

Interesting, because in our interface we have a feature that we can simulate what our hardware is going to do. In this way, our customers can setup the machine without production first and simulate what is going to happen. But what is detected is put on an overlay with green color, and the color can not change.

That is a problem if you can not see that kind of green, specially if you are detecting colors that are displayed in the yellow spectrum, because those people see it as the same color.

30days_day7Thanks to the challenge, I found a new tool and found some bugs. This means that I now completed Challenge 7 and 19.

Challenge 7 was for me an eye-opener. It is not because I can see some items on the screen, that everybody sees the same items in the same way. I think that the spectrum extension will be used a lot more in my testing activities.

Let’s go crazy

The next Challenge of the 30 days of testing is to perform a crazy test. What is crazy? I was thinking and it was maybe time to have a capacity test on our machine. Capacity tests are for us not that easy because it needs a lot of equipment. But Let’s try do do it anyway.
standalone_smallNormally my test setup in our lab is just the machine. I can place product at the back side on it, and in the front, there are 2 outputs, one for good product, and one for bad product. Yes, the machine I have to test is a sorting machine. We sort food. And because we do not want stones or glass in our daily food, we developed machines for taking that out of the food chain. I do not want that, so a lot of testing is needed. But today, I had a challenge to do.

feedback_smallLet’s go crazy and let’s drive to our other plant, where we could use a feedback loop. So we can do a capacity test with it.  After that was installed, I needed some test product. We have square cubes for that, but also Lego bricks. Nice, lets put a lot of them on our machine. After that was done, I saw that the spread of the product was nicely spread over 2 camera’s in our system. That was not what I wanted, because the system was not stressed that way. Maybe it was time do do something different this time.

top_smallI too a little of carbon and tried to manage the stream on our shaker. And it worked! Now the stream of bricks are just over one camera. That is what I wanted. Now I can stress a part of our system. Let’s see what happens then. I first started without any sorting. That went well and did not notice any problem. Then it was time to do something more. What if I sorted the Lego bricks out of the square cubes. Lego is rectangular and we also have little square bricks. Our detection can handle that. So lets do that.

After the parameters were set correctly, the output changed. Yes it works! The rectangular bricks can be sorted out of the square ones.

bug_smallThen a warning popped up in our logs. He, what is that? The warning “Shape processing SW too slow” together with warnings about cameras. But there was more, some warnings about cameras can not be true. There is no shape processing attached to some cameras in our machine. And they are also mentioning problems. That is not good at all. No, it is a bug! Because this test was so spectacular, the team was also looking what we did, and they all agree, that there is a problem in our system. So I think it was worth the effort. I only found 1 problem this time. A problem that I never had found if I did not use the feedback loop.

Unfortunately I can not always use this system, because I have to share it with other departments.

Introvert people

Today I want to talk about introverts. John Stevenson explains in his blog post that we should putting a label on people. We should be who and what we are. That is true.

But I also read Quiet from Susan Cain. She explained in her book that we are living in a world that is dominated by extroverts. That is true, and it is also very logical. If you look at our roots, to our ancestors in prehistoric times. They lived in groups, like we now also do. And each group has one leader. Mostly that was the most aggressive one, the one that could dominate the other (males). In present times, this is also true. Now we do not fight with our fists, but with words. And the most verbal persons are by accident the ones that are labelled as extravert.

Susan also explains that most of times people are not 100% extravert or 100% introvert. No, we do have a mix. And most people find that true.

Some signs that you could be an introvert are:

  • You feel exhausted after spending time with a group of people. After a day interacting with others, you need to retreat in a quiet place. I do so after some meetings. That is taken my energy completely. And more energy is taken from me if there are more people in the meeting room.
  • You enjoy being alone. For me that is also true. One of my hobbies is hiking in woods. I mostly go for a long walk of 40 or 50 km. I do not walk with other people, but alone. And if I return at home, I am fully recharged.
  • I am a quiet person. That is also another sign for introverts. But this is because I must most of the time prepare a conversation. I can not think out of the box without a reflexion first. There is however an exception. On domains that I know, like testing, or hiking, I can do any conversation I assume.

To conclude. If I am introvert or not, for me it does not matter, I only know since I read Susan’s book that I am as normal as another person and that I can live with my times that I want to be alone.