Restlet 1.0 - Benchmark

Introduction

This section describes some performance tests involving the various standalone HTTP server connectors distributed with Noelios Restlet Engine. For comparison purpose, we also tested the Servlet adapter with popular Servlet containers. The tests have been made with the release 1.0 RC6 of the Restlet. As the RC6 is the latest release before the final 1.0 one, the performance results shouldn't change in any way.

This benchmark doesn't aim to provide an absolute comparison between the various connectors and Servlet containers tested for at least one reason: only their default configuration was used, with no particular optimization or performance tuning. We would like to show the behavior of the Restlet framework in different situations where the server's load varies from light to heavy, and prove that the various connectors are scalable and usable in production environments.

Environment

Specifications

Stress tool

The following tests have been made with Apache Bench delivered by Apache HTTP Server 2.0. Ab especially shows how many requests per second the web server is capable of serving. For this benchmark, two files were served: a single HTML page (35 kb), and a Zip file (8.8 mb). The maximum number of seconds spent benchmarking was limited to 60 seconds and the number of concurrent requests varied from 1 to 300.

Apache Bench command line:

ab -q -t 60 -c %concurrency% -e %csv_test_file%.csv %url_test%
   

where %concurrency% took the following values: 1, 20, 80, 300.

Servlet containers configuration

As written in the introduction, both tested Servlet containers Apache Tomcat and Jetty haven't been especially configured for this benchmark. They were just installed, and then ready to use.

Restlet connectors configuration

The Restlet connectors have been tested with their default behaviour. Please refer to the connectors page for more details about the available parameters for each tested connectors.

The application using the restlet connectors has been launched via the Tanuki wrapper, that provides several kind of nice services especially write once run everywhere configuration service. It manages a dedicated JVM instance where your application is running. For these tests, the JVM has been configured with the following parameters:

-server
-Djava.awt.headless=true
-Djava.util.logging.config.file=%PROD_HOME%/data/config/logging.properties
   

Results

Results obtained for an HTML page (35 kb)

Requests per second (mean)

Time per request (mean)

Results obtained for a Zip file (8.8 mb)

Requests per second (mean)

Time per request (mean)

Conclusion

These simple tests show that under normal conditions, each connector or servlet container responds quickly and correctly. The response time does not exceed 12ms for a single HTML page, and under normal circumstances is less than 500ms for the zip file. Whatever kind of requested file, we can unsurprisingly notice that as the number of concurrent requests grows, the time necessary to serve them increases.

The results obtained for the single web page are homogeneous between each HTTP connectors and between the two tested servlet containers. As an exception, the AsyncWeb connector (which has not reached the 1.0 version yet) does not support a very high number of concurrent requests.

The results obtained for the zip file show a bit more differences between the HTTP connectors. We can also notice that the AsyncWeb connector shows a better resistance. This may be due to the intrinsec quality of the underlying MINA NIO framework.

At this point, we must recognize that the tests should be completed, especially with graphs showing for example the evolution of the throughput and/or latency for a long period of time, etc. But, the aim of these tests was only to show that the Noelios Restlet Engine provides robust connectors and can also be efficiently integrated inside a servlet container.

It would also be interesting to compare those results with those of the full NIO HTTP connector based on Sun's Grizzly NIO framework, which is planned for version 1.1.