Sometimes it is necessary to impose certain order on the tasks in a threadpool. Issue 206 of the JavaSpecialists newsletter presents one such case: we have multiple connections from which we read using NIO. We need to ensure that events from a given connection are executed in-order but events between different connections can be freely mixed.

I would like to present a similar but slightly different situation: we have N clients. We would like to execute events from a given client in the order they were submitted, but events from different clients can be mixed freely. Also, from time to time, there are “rollup” tasks which involve more than one client. Such tasks should block the tasks for all involved clients (but not more!). Let’s see a diagram of the situation:

As you can see tasks from client A and client B are happily processed in parallel until a “rollup” task comes along. At that point no more tasks of type A or B can be processed but an unrelated task C can be executed (provided that there are enough threads). The skeleton of such an executor is available in my repository. The centerpiece is the following interface:

public interface OrderedTask extends Runnable {
    boolean isCompatible(OrderedTask that);
}

Using this interface the threadpool decides if two tasks may be run in parallel or not (A and B can be run in parallel if A.isCompatible(B) && B.isComaptible(A)). These methods should be implemented in a fast, non locking and time-invariant manner.

The algorithm behind this threadpool is as follows:

• If the task to be added doesn’t conflict with any existing tasks, add it to the thread with the fewest elements.
• If it conflicts with elements from exactly one thread, schedule it to be executed on that thread (and implicitly after the conflicting elements which ensures that the order of submission is maintained)
• If it conflicts with multiple threads, add tasks (shown with red below) on all but the first one of them on which a task on the first thread will wait, after which it will execute the original task.

More information about the implementation:

• The code is only a proof-of-concept, some more would would be needed to make it production quality (it needs code for exception handling in tasks, proper shutdown, etc)
• For maximum performance it uses lock-free* structures where available: each worker thread has an associated ConcurrentLinkedQueue. To achieve the sleep-until-work-is-available semantics, an additional Semaphore is used**
• To be able to compare a new OrderedTask with currently executing ones, a copy of their reference is kept. This list of copies is updated whenever new elements are enqueued (this is has the potential of memory leaks and if tasks are infrequent enough alternatives – like an additional timer for weak references – should be investigated)
• Compared to the solution in the JavaSpecialists newsletter, this is more similar to a fixed thread pool executor, while the solution from the newsletter is similar to a cached thread pool executor.
• This implementation is ideal if (a) the tasks are (mostly) short and (mostly) uniform and (b) there are few (one or two) threads submitting new tasks, since multiple submissions are mutually exclusive (but submission and execution isn’t)
• If immediately after a “rollup” is submitted (and before it can be executed) tasks of the same kind are submitted, they will unnecessarily be forced on one thread. We could add code rearrange tasks after the rollup task finished if this becomes an issue.

Have fun with the source code! (maybe some day I’ll find the time to remove all the rough edges).

* somewhat of a misnomer, since there are still locks, only at a lower – CPU not OS – level, but this is the accepted terminology

** – benchmarking indicated this to be the most performant solution. This was inspired from the implementation of the ThreadPoolExecutor.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Testing multi-threaded code is hard. The main problem is that you
invoke your assertions either too soon (and they fail for no good
reason) or too late (in which case the test runs for a long time,
frustrating you). A possible solution is to declare an interface like
the following:

interface ActivityWatcher {
 void before();
 void after(); 
 void await(long time, TimeUnit timeUnit) throws InterruptedException, TimeoutException;
}

It is intended to be used as follows:

• “before” is called before the asynchronous task is delegated to an
  execution mechanism (threadpool, fork-join framework, etc) and it
  increments an internal counter.
• “after is called after the asynchronous task has completed and it decrements the counter.
• “await” waits for the counter to become zero

The net result is that when the counter is zero, all your asynchronous tasks have executed and you can run your assertions. See the example code. A couple more considerations:

• There should be a single ActivityWatcher per test (injected trough constructors or a dependency injection framework)
• In production code you will use a dummy/noop implementation which removes any overhead.
• This only works for situations where the asynchronous are kicked of
  immediately. Ie. it doesn’t work for situations where we have
  periodically executing tasks (like every 5 seconds) and we would want to
  wait for the 7th tasks to be executed for example.

One thing the above code doesn’t do is collecting exceptions: if the
exceptions happen on different threads than the one executing the
testrunner, they will just die and the testrunner will happily report
that the tests passs. You can work around this in two ways:

• use the default UncaughtExceptionHandler
  to capture all exceptions and rethrow them in the testrunner if they
  arrise (not so nice because it introduces global state – you can’t have
  two such tests running in parallel for example)
• Extend activity watcher and code calling activity watcher such that it has a “collect(Throwable)” method which gets called with the uncaught exceptions and “await” rethrows them.

Implementing this is left as an exercise to the reader :-).

To illustrate the problem, I’ve created a small program. It does the following:

1. It creates a Date object and sets it to certain values
2. Schedules multiple Runnable’s which examine the value of the object on a threadpool

Everything looks fine and dandy, right? The object isn’t changed (apparently) after being handed of to the threadpool, yet sometimes wrong answers still appear (it takes around ~30 min on my laptop for such an event). So what are the lessons here?

• Get your API right! If the user doesn’t seem to be doing writing, don’t do writing!
• You can still do lazy initialization (if you really want to), but be sure to make it thread-correct (volatile, synchronized, etc) or at least document it (even though nobody reads the documentation)
• Source code FTW! I couldn’t have debugged this without source code. Ok, maybe I could (decompiling class files is not that hard), but probably I wouldn’t have bothered.
• Finally, the solution (hack) in this particular situation is to call getTime() after setting the values, which preemptively normalizes the internal representation. Of course the proper solution would be to pass around truly immutable objects (like timestamps or value objects from Joda Time).

I was profiling a rather old legacy codebase (since the first rule of performance optimization is “profile it” with the close second of “have clear goals in mind” – but that’s an other post) and – after optimizing the first few hotspots, Thread.yield appeared at the top of the most timely methods. I was intrigued, since I didn’t use yield since I wrote “cooperative multitasking” programs for Windows 3.1 in VB 3 (and I’m not a big Python programmer either). So I scoured the ‘net for Information on why/when you should use Thread.yield, but came up with relative few pieces of information:

• I found indications that some old Linux kernels behaved poorly on single processor machines if Thread.yield wasn’t used (as in: one thread consuming all the CPU)
• There were discussion about using Thread.sleep(0) vs. Thread.yield (apparently there is a difference regarding the treatment of remaining time-quantum by the scheduler)
• … this was pretty much it …

So I’ve decided to do some micro-benchmarks. They consisted of a producer and a consumer thread, connected by an unbounded queue (a LinkedBlockingQueue to be more exact) and I measure the number of items produced / consumer in 10 seconds. The first set of measurements were performed on a dual-core machine, while the second set in a VM to simulate a single-CPU system (it’s kind of ironic that one has to perform simulation to evaluate single-core systems). This isn’t meant to be a performance, evaluation, thus all the numbers are normalized to the produced/direct number.

(My) conclusions:

• These days there is no need for “helping” the OS scheduler out. Both of the proposed methods (yield and sleep) reduced the throughput of the system considerably.
• Speaking of throughput: make a decision about the (performance) numbers your system should achieve. This includes both throughput and delay. Concrete (and realistic!) numbers. “As good as possible” is not a number! Neither is “better than the current”. Then profile and optimize it until the numbers are achieved and no further.
• In the case of a single CPU system there was a big imbalance between the speed of the producer and consumer which Thread.yield (or Thread.sleep) seemed to solve. Consider however, that this “solution” comes at the price of (almost) two orders of magnitude reduction in the throughput. A much better solution would be (in case you are in the rare situation of single CPU – maybe you’re on a VPS, or you have multiple CPUs, but the number of threads far outweigh the number of CPUs) to use a bounder queue. This way the producer gets slowed down (by blocking repeatedly) if it produces faster than the consumer can consume. Then again, you need consider if this is acceptable for your application. Maybe the “overflow” situation is rare and it is more important to handle each element (and there are enough resources for it in the long run) than response speed. You have to know your application and its priorities. There is no way around it.

Finally I would like to leave you with the following short (~30 min) presentation about performance optimization on the JVM: Making every millisecond count! JVM performance tuning in the real-world.

priv.addAll(common);

Exception in thread "Thread-1" java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at java.util.ArrayList.toArray(Unknown Source)
	at java.util.ArrayList.addAll(Unknown Source)
	at TestConcurrentList$ConsumeThread.run(TestConcurrentList.java:34)

Answer: because of bad synchronization. The scenario is the following: one thread is continuously modifying the list “common” while the second thread tries to perform the “addAll” operation on it. The testcode is shown below:

import java.util.*;

public class TestConcurrentList {
	private static List common = new ArrayList();
	
	private static class GenerateThread extends Thread {
		private List common;
		
		GenerateThread(List common) {
			this.common = common;
		}
		
		@Override
		public void run() {
			while (true) {
				common.add("foo");
				if (common.size() > 1000) common.clear();
			}
		}
	}
	
	private static class ConsumeThread extends Thread {
		private List common;
		
		ConsumeThread(List common) {
			this.common = common;
		}
		
		@Override
		public void run() {
			while (true) {
				List priv = new ArrayList();
				priv.addAll(common);
			}
		}
	}

	public static void main(String[] args) throws Exception {
		Thread gen = new GenerateThread(common),
			consume = new ConsumeThread(common);
		gen.start();
		consume.start();
		System.out.println("Waiting...");
		gen.join();
	}

}

What makes this so hard debug is that (a) the exception doesn’t say anything about concurrency (it’s not like it throws an ConcurrentModificationException), (b) the exception actually occurs in the native Java libraries and (c) the source of the concurrent modifications may not be so obvious as in the reduced test case.

Conclusion? When possible, avoid concurrency or delegate it (to an RDBMS with proper transaction / locking support for example).

PS. This bug is not found by FindBugs (admittedly the support for checking concurrency bugs is fairly low at the moment) and is dependent on the version of the runtime. I reproduced it on 1.5.14, but not on 1.6.07.

Picture taken from yimhafiz’s photostream with permission.