Benchmarking servlet containers

Christopher L Merrill of webperformance put together
a really extensive test of servlet containers,
the Servlet Performance Report. The servers tested
were as follows.

• Apache Tomcat 5.0.25
• IronFlare Orion 2.0.2
• MortBay Jetty 4.2.20
• Caucho Resin 3.0.8
• IBM WebSphere 5.1
• Macromedia JRun 4 (update 3)

The aim was to show how each server behaved in
particular situations, such as serving static content and handling sessions, without declaring a winner. Those
aspects were measured with several tests.

It’s a well-executed benchmark, though there were a few missteps. The first thing that
surprised me was Jetty’s poor showing. We normally
use it in production without the slightest trouble.

It turned out all the servers were tested in their default configurations. Jetty,
with 50 worker threads by default (threads that handle incoming requests),
was compared with servers such as Tomcat, which defaults to 150.

Jetty’s creator and lead developer Greg Wilkins naturally responded
to the benchmark in Servlet Performance Report — The Jetty Response. There was more discussion
of the benchmark and results on TSS and in the benchmark’s discussion forum.