Alexey Zakhlestins blog

My experience exactly. I refuse to ever again take one of their poorly organized and thought out tests.

On November 15th of 2006, I took the Brainbench
Java 2 test.

When I went into the test, I was expecting
something that would test my general knowledge
of the Java language and object oriented
programming. The test that I took did not do
that.

There were a large number of questions on
special purpose API's that I have never used.
There were some questions on development tools
that I have never used. And there were a large
number of "brain teaser" questions on code
snippets which I could have answered, if I would
have had more time.

As a result, my test score was very low (2.60).

That score shows that I'm not good at guessing
at API's and tools that I've never used. And it
shows that I'm not good at brain teasers with a
three-minute time limit. However, it shows
ABSOLUTELY NOTHING about my Java and object
oriented programming skills.

In 2002, I studied a book on Java, and then I
took a college course on it. I got an A in the
course. According to my Brainbench score, I
should not have been an "A" student.

For over a year, I was the sole designer and
developer of web software that my employer believes is marketable. That software involves
thousands of lines of Java code, a large number
of API's, and a large number of advanced object
oriented constructs. According to my Brainbench
score, I should not be able to accomplish what I
have just accomplished.

When I was in college, I took secondary
education courses. Later, while I was working
for a former employer, I designed two computer
courses which I taught for several years. As a
trained and experienced trainer and tester,
it is my opinion that the Brainbench Java 2 test
has a number of very serious flaws.

It appears that there have been no sound
scientific studies regarding the Brainbench
claim that their tests predict employee
success. And it appears that there are a large
number of companies that are blindly accepting
these unsubstantiated claims.

It seems to me that a sound scientific study for
the Java 2 test would include the following
elements: have thousands of working and
successful Java programmers take various Java 2
tests; have thousands of inexperienced people
with Java knowledge take the same tests; for
individual test takers, have tests with a large
number of questions on API's and tools that they
have never used; for individual test takers,
have tests with a large number of questions on
API's and tools that they have used; for the
latter, follow their careers as Java programmers
for at least five years.

It appears to me that the creators of the
Brainbench Java 2 test do not know what a
typical Java programmer does, and they have no
understanding of the art of testing.

I wonder how many careers have been derailed as
a result of flawed Brainbench tests. I would
like to see a scientific survey on that.