False Positives and False Negatives
because of improper environment set up
Let’s begin
with a real life scenario, I went to a doctor for a regular check up and I got
to know that I had been suffering from ‘Cancer’. I had never imagined that I would
be suffering from CANCER I believed it just because the report said.
Now my friend Mr. XXX consoled me and told me to visit the doctor again and
tell him that I wanted a Recheck up. I found his idea a good way to confirm if
I was really suffering from CANCER. I went to the doctor and after we were done
with Recheck up, surprisingly but luckily I had no CANCER, actually there were
some flaw in the instruments which generated a false result.
This is a
false positive, because on account of some flaw in the instruments, I had been
informed that I had CANCER.
Why called
False Positive?
Expected
Result as per me: I was totally fine and healthy.
Actual Result
as per Doctor: I had CANCER.
As there is
a deviation between the ‘Expected Result’ and ‘Actual Result’, it means there is
a failure as per me.
When I asked
the doctor what I would call such a fake report then he suggested me to call it
a ‘False Positive’. False because it was
not as per my expectation and Positive because doctors are born to identify
problems and symptoms in our body which my doctor has done by giving me a
report of CANCER which he believed was true because he referred to a medical
report.
Now what
would be a False Negative??
A test which
is marked as PASS but there still is a problem and under ideal conditions the
test should FAIL.
It means if
I had CANCER and the doctor left me by saying that I am fine just because of
some instrumental flaw, then that would be called as a ‘False Negative’.
Now let’s
come to the Software’s False Positives and False Negatives scenarios.
We a group
of software testers are testing an application ‘A’ under a set of environments
say B, C, D etc. We set up the environments
properly and we started testing our AUT (Application under Test) under the
desired environment. We started finding bugs and reported simultaneously. But
over a week we noticed that the ‘Defect Rejection Rate’ was too high as per
previous statistics. The development end people just updated their status as ‘Working
Fine at Our End’. Then we planned to reset the environments and did it
accordingly but with more cautions this time. Now all our own issues started to
pass at our end also.
This was ‘False
Positive’ in software testing. False because ‘Actual and Expected’ were not
matching. And Positive because we found
the bugs at our end (Our job as a tester is to find bugs and we did it so a
positive for our side).
But it is for granted that if we found some false positives then there must be some false negatives. It means we passed some tests just because of environment set up issue. This is where the quality of software is affected. Because False positive when goes for fixing or debugging to the development guys they found it working and hence we can close the issue after proper environment set up and retesting.
But what will happen to the False Negatives that we didn’t find and hence didn’t report because it was working fine in our environment which was not properly set up?
We should always try to cross check the environment set up by us and the environment needed to test the application. Once we commit mistake at this point, it means we already degraded the quality of application because it would give rise to a lot of false positives that leads to wastage of time and money (resources) for debugging and retesting and similar number of false negatives which are very difficult to find out, because no one notices it.
