Restlet 1.1 - Benchmark

Introduction

This section describes some performance tests involving the various standalone HTTP server connectors distributed with Restlet Engine. For comparison purpose, we also tested the Servlet adapter with popular Servlet containers. The tests have been made with the release 1.1.3 of the Restlet.

This benchmark doesn't aim to provide an absolute comparison between the various connectors and the servlet containers tested for at least one reason: only their default configuration was used, with no particular optimization or performance tuning except for the connector based on Jetty. We would like to show the behavior of the Restlet framework in different situations where the server load varies from light to heavy, and prove that the various connectors are scalable and usable in production environments.

Environment

Specifications

Stress tool

The following tests have been made with Apache Bench delivered by Apache HTTP Server 2.0. Ab especially shows how many requests per second the web server is capable of serving. For this benchmark, one file is served: a single text file (2.1 kb). The number of concurrent requests varied from 1 to 200 for one serie of tests, and the number of requests (with only one client) varied from 1 to 500 for the other.

Apache Bench command line:

ab -q -c %concurrency% -n 1000 %url_test%
   

where %concurrency% varied from 1 to 200.

ab -q -c 1 -n %nbRequests% %url_test%
   

where %nbRequests% varied from 1 to 500.

Restlet connectors configuration

The Restlet connectors have been tested with their default configuration except for the connector based on Jetty. The maximum number of threads has been increased to 1000 ("maxThread" parameter).

Please refer to the connectors page for more details about the available parameters for each tested connectors.

The application using the Restlet connectors has been launched via the Tanuki wrapper, that provides several nice services especially "write once run everywhere" service. It manages a dedicated JVM instance where your application is running. For these tests, the JVM has been configured with the following parameters:

-server
-Djava.awt.headless=true
-Djava.util.logging.config.file=%PROD_HOME%/data/config/logging.properties
   

Servlet containers configuration

As written in the introduction, both tested Servlet containers Apache Tomcat and Jetty haven't been especially configured for this benchmark. They were just installed, and then ready to use.

Results

First series of tests: variation of the concurrency

Click on the image to enlarge it.

In this series of tests, an increasing number of concurrent clients that ranges from 1 to 200 are asked to send 1000 requests. This test tends to illustrate the scalability of the server connector. What can explain the upper limit of 1000 requests by second? At first sight, it could be the number of threads (1000 for Jetty), but actually the pool of threads of the internal and the Grizzly connectors are not limited at all. The main reason must be that there is only 1000 requests to achieve.

We notice that the average time per request increases linearly with the number of concurrent clients varies, although the response rate seems more or less constant.

Except for the connector based on Simple 3, the connnectors seem to perform homogenously, regarding the number of requests handled per second. Unsurprisingly, the response time is increasing with the number of concurrent requests.

Second series of tests: variation of the number of requests

Click on the image to enlarge it.

In this series of tests, a single client is asked to launch a number of requests that ranges from 1 to 500. This test tends to illustrate how the connector performs when there is no processing overload on the server since the content is not generated but only delivered.

When the number of requests varies, we notice that the time spent per request is more or less constant and does not exceed 20 ms. At the same time, the response rate degrades quite regularly which seems to denote an increasing latency.

The graphs seem to establish a hierarchy between the connector based on Simple 3 and the others. We can notice than the former has not been updated since May 2006 (since the team is focused on Simple 4, that we will support in Restlet 2.0), whereas the ones based on Grizzly and Jetty are pretty recent (March 2009). We can also notice that the performance of all connectors looks generally good.

Conclusion

These simple tests show that under normal conditions, each connector responds quickly and correctly. The response time are correct and each of them seems reasonably scalable.