Testing multi-threaded code is hard. The main problem is that you
invoke your assertions either too soon (and they fail for no good
reason) or too late (in which case the test runs for a long time,
frustrating you). A possible solution is to declare an interface like
the following:

interface ActivityWatcher {
 void before();
 void after(); 
 void await(long time, TimeUnit timeUnit) throws InterruptedException, TimeoutException;
}

It is intended to be used as follows:

• “before” is called before the asynchronous task is delegated to an
  execution mechanism (threadpool, fork-join framework, etc) and it
  increments an internal counter.
• “after is called after the asynchronous task has completed and it decrements the counter.
• “await” waits for the counter to become zero

The net result is that when the counter is zero, all your asynchronous tasks have executed and you can run your assertions. See the example code. A couple more considerations:

• There should be a single ActivityWatcher per test (injected trough constructors or a dependency injection framework)
• In production code you will use a dummy/noop implementation which removes any overhead.
• This only works for situations where the asynchronous are kicked of
  immediately. Ie. it doesn’t work for situations where we have
  periodically executing tasks (like every 5 seconds) and we would want to
  wait for the 7th tasks to be executed for example.

One thing the above code doesn’t do is collecting exceptions: if the
exceptions happen on different threads than the one executing the
testrunner, they will just die and the testrunner will happily report
that the tests passs. You can work around this in two ways:

• use the default UncaughtExceptionHandler
  to capture all exceptions and rethrow them in the testrunner if they
  arrise (not so nice because it introduces global state – you can’t have
  two such tests running in parallel for example)
• Extend activity watcher and code calling activity watcher such that it has a “collect(Throwable)” method which gets called with the uncaught exceptions and “await” rethrows them.

Implementing this is left as an exercise to the reader :-).

JSON is a good alternative when you need a lightweight format to specify structured data. But sometimes (for example when you want the user to specify JSON manually) you would like to relax the formalism required to specify "valid" JSON data. For example the following snippet is not valid as per the spec, although its intent is quite clear:

[{ foo: 'bar' }]

To make this standard compliant we would need to write it as:

[{ "foo": "bar" }]

We shouldn’t run out and blame the standard of course since it needs to balance many contradictory requirements (ambiguity of encoded data, ease of understanding, ease of writing parsers, etc). If you decide that you want to strike the balance differently (make the definition of valid data more relaxed) you can do this easily with the Jackson parser:

JsonParser parser = new JsonFactory()
	.createJsonParser("[{ foo: 'bar' }]")
		.enable(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES)
		.enable(JsonParser.Feature.ALLOW_SINGLE_QUOTES);
JsonNode root = new ObjectMapper().readTree(parser);

assertEquals("bar", root.get(0).get("foo").asText());

If your tool of choice is gson, it is slightly more complicated but still doable. See the linked source code for a complete example.

JSON is a good tool for semi-structured data and using a relaxed parsing can make the programs you write easier to use.

To do this:

• Have Jenkins installed
• Intstall the Performance plugin (or upgrade to the latest version, since there was a bug in earlier versions which prevented the parsing of the JUnit reports)
• For your build check “Publish Performance test result report” and add locations where the reports should be collected from.
• That’s it! Future builds will collect the performance data and you can access it using the “Performance Trend” link (at the job level) or the “Performance Report” link (at the build level)

More details / caveats:

• The paths are defined as ANT file expressions (that is you can use “**” to specify an arbitrary level of directories, for example: target/surefire-reports/**/TEST*.xml)
• JUnit performance is grouped at the test-class level, thus it probably makes sense create separate project / module which group the performance test cases.
• Benchmarking is hard and JUnit doesn’t give you any provisions to do warmup or to repeat the tests multiple times. To make your test as relevant as possible you should do this manually (warmup code can be placed in the @Before method for example). A properly set up JMeter task accounts for this already.
• TestNG tests can also be parsed as long as the test run is set to produce a JUnit compatible report.
• Slightly off-topic: to integrate a JMeter run into your maven build, you can use the AntRun plugin:

  <build>
   <plugins>
    <plugin>
     <artifactId>maven-antrun-plugin</artifactId>
     <version>1.6</version>
     <executions>
      <execution>
       <phase>test</phase>
       <configuration>
        <target>
         <taskdef name="jmeter" classpath="C:workantlibant-jmeter-1.1.0.jar"
          classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>
         <jmeter jmeterhome="C:jakarta-jmeter-2.5"
          testplan="${basedir}/src/test/resources/example.jmx"
          resultlog="${basedir}/target/JMeterResults.jtl"/>
        </target>
       </configuration>
       <goals>
        <goal>run</goal>
       </goals>
      </execution>
     </executions>
    </plugin>
   </plugins>
  </build>

Article originally posted to the Transylvania JUG blog.