Joshua Zhu's Blog

Apache 2.4 Faster Than Nginx?

Joshua — Tue, 28 Feb 2012 15:46:25 +0000

Some reports came out recently after Apache 2.4 was released, saying it’s “as fast, and even faster than Nginx”. To check it out if it’s true, I ran a benchmark by myself. Here are the benchmark results.

It turned out that it’s a false claim. Apache 2.4 is actually much slower than Nginx!

The benchmark was run on the same Linux box (localhost), to avoid possible network affection. I used ab (ApacheBench) as the benchmark client. Apache listened on port 80 and Nginx listened on port 81. In the whole benchmark process, Apache was stressed first then Nginx. There was a 60 seconds sleep between each test, which was taken by five times per concurrency (from 100 to 1000). I gave up the tests with higher concurrencies because Apache was so unstable with concurrency greater than 1000 that there would be some failures. While Nginx was very stable without problems.

I’ve tried my best to fully “unleash the power of Apache”:
1) configured with apr-1.4.6 and apr-util-1.4.1, using the fastest atomic API:

$ ./configure --prefix=/home/shudu/apache --with-included-apr \
--enable-nonportable-atomics=yes

2) least modules were enabled:

$ apache/bin/httpd -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 mpm_event_module (static)
 authz_core_module (shared)
 filter_module (shared)
 mime_module (shared)
 unixd_module (shared)

3) MaxRequestWorkers was raised to 800 and ServerLimit to 32.

Nginx was just compiled with its default options:

$ ./configure --prefix=/home/shudu/bench/nginx

The common features of Apache and Nginx:
1) Sendfile on.
2) KeepAlive off.
3) AccessLog off.

The configuration files of Apache and Nginx are as followings:

# Apache 2.4.1
ServerRoot "/home/shudu//bench/apache"
 
KeepAlive Off
 
ServerLimit 32
MaxRequestWorkers 800
 
Listen 80
ServerName localhost
 
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule filter_module modules/mod_filter.so
LoadModule mime_module modules/mod_mime.so
LoadModule unixd_module modules/mod_unixd.so
 
<IfModule unixd_module>
User shudu
Group shudu
IfModule>
 
ServerAdmin you@example.com
 
<Directory />
    AllowOverride none
    Require all denied
Directory>
 
DocumentRoot "/home/shudu/bench/apache/htdocs"
<Directory "/home/shudu/bench/apache/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
Directory>
 
ErrorLog "logs/error_log"
LogLevel warn
 
<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
IfModule>
 
EnableSendfile on

# Nginx-1.0.12
user  shudu users;
worker_processes  2;
 
events {
    worker_connections  10240;
    accept_mutex_delay  100ms;
}
 
http {
    include             mime.types;
    default_type        application/octet-stream;
 
    sendfile            on;
    tcp_nopush          on;
 
    keepalive_timeout   0;
 
    access_log off;
 
    server {
        listen          81;
        server_name     localhost;
 
        location / {
            root        html;
            index       index.html index.htm;
        }
    }
}

My Ubuntu-10.04 box:

$ uname -a
Linux shudu-desktop 2.6.32-38-generic #83-Ubuntu SMP Wed Jan 4 11:13:04 UTC 2012 i686 GNU/Linux

$ grep "model\ name" /proc/cpuinfo 
model name	: Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
model name	: Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz

$ free -m
             total       used       free     shared    buffers     cached
Mem:          1995       1130        864          0         80        341
-/+ buffers/cache:        708       1286
Swap:         2491          0       2491

$ cat /etc/security/limits.conf
root soft nofile 65535
root hard nofile 65535
shudu soft nofile 65535
shudu hard nofile 65535

Comments are welcome

A Mechanism to Help Write Web Application Firewalls for Nginx

Joshua — Mon, 16 Jan 2012 14:38:09 +0000

Developing a Web Application Firewall module for Nginx is not an easy task. The lack of input body filters makes it harder. Nginx is an outstanding web server, but it is not perfect. Actually, nothing is perfect.

So we added the input body filter mechanism to our own Nginx distribution, which is named Tengine. By taking advantage of this mechanism, processing the request body is not that complicated anymore (In standard Nginx, request body may be buffered to disk file and you have to deal with up to two buffers)

Here I have an example to demonstrate how to write an input body filter. It is a simple module to fight hash collision DoS attacks.

/*
 * Copyright (C) Joshua Zhu, http://www.zhuzhaoyuan.com
 */
 
 
#include 
#include 
#include 
 
 
typedef struct {
    ngx_flag_t                            enable;
    ngx_uint_t                            max_post_params;
} ngx_http_anti_hashdos_loc_conf_t;
 
 
typedef struct {
    ngx_uint_t                            post_params;
} ngx_http_anti_hashdos_ctx_t;
 
 
static ngx_int_t ngx_http_anti_hashdos_input_body_filter(ngx_http_request_t *r,
    ngx_buf_t *buf);
 
static void *ngx_http_anti_hashdos_create_loc_conf(ngx_conf_t *cf);
static char *ngx_http_anti_hashdos_merge_loc_conf(ngx_conf_t *cf, void *parent,
    void *child);
static ngx_int_t ngx_http_anti_hashdos_init(ngx_conf_t *cf);
 
 
static ngx_command_t ngx_http_anti_hashdos_filter_commands[] = {
 
    { ngx_string("anti_hashdos"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_FLAG,
      ngx_conf_set_flag_slot,
      NGX_HTTP_LOC_CONF_OFFSET,
      offsetof(ngx_http_anti_hashdos_loc_conf_t, enable),
      NULL },
 
    { ngx_string("anti_hashdos_max_post_params"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
      ngx_conf_set_num_slot,
      NGX_HTTP_LOC_CONF_OFFSET,
      offsetof(ngx_http_anti_hashdos_loc_conf_t, max_post_params),
      NULL },
 
      ngx_null_command
};
 
 
static ngx_http_module_t ngx_http_anti_hashdos_filter_module_ctx = {
    NULL,                                 /* preconfiguration */
    ngx_http_anti_hashdos_init,           /* postconfiguration */
 
    NULL,                                 /* create main configuration */
    NULL,                                 /* init main configuration */
 
    NULL,                                 /* create server configuration */
    NULL,                                 /* merge server configuration */
 
    ngx_http_anti_hashdos_create_loc_conf,/* create location configuration */
    ngx_http_anti_hashdos_merge_loc_conf  /* merge location configuration */
};
 
 
ngx_module_t ngx_http_anti_hashdos_filter_module = {
    NGX_MODULE_V1,
    &ngx_http_anti_hashdos_filter_module_ctx, /* module context */
    ngx_http_anti_hashdos_filter_commands,/* module directives */
    NGX_HTTP_MODULE,                      /* module type */
    NULL,                                 /* init master */
    NULL,                                 /* init module */
    NULL,                                 /* init process */
    NULL,                                 /* init thread */
    NULL,                                 /* exit thread */
    NULL,                                 /* exit process */
    NULL,                                 /* exit master */
    NGX_MODULE_V1_PADDING
};
 
 
static ngx_http_input_body_filter_pt  ngx_http_next_input_body_filter;
 
 
static ngx_int_t
ngx_http_anti_hashdos_input_body_filter(ngx_http_request_t *r,
    ngx_buf_t *buf)
{
    u_char                           *p;
    ngx_http_anti_hashdos_ctx_t      *ctx;
    ngx_http_anti_hashdos_loc_conf_t *ahlf;
 
    ahlf = ngx_http_get_module_loc_conf(r, ngx_http_anti_hashdos_filter_module);
 
    if (!ahlf->enable) {
        return ngx_http_next_input_body_filter(r, buf);
    }
 
    ctx = ngx_http_get_module_ctx(r, ngx_http_anti_hashdos_filter_module);
    if (ctx == NULL) {
        ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_anti_hashdos_ctx_t));
        if (ctx == NULL) {
            return NGX_HTTP_INTERNAL_SERVER_ERROR;
        }
 
        ctx->post_params = 1;
    }
 
    for (p = buf->pos; p < buf->last; p++) {
 
        if (*p == '&') {
            ctx->post_params++;
        }
    }
 
    if (ctx->post_params > ahlf->max_post_params) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                      "anti hashdos: \"%V\" blocked, too many post params: %d",
                      &r->connection->addr_text,
                      ctx->post_params);
        return NGX_HTTP_BAD_REQUEST;
    }
 
    return ngx_http_next_input_body_filter(r, buf);
}
 
 
static void *
ngx_http_anti_hashdos_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_anti_hashdos_loc_conf_t *conf;
 
    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_anti_hashdos_loc_conf_t));
    if (conf == NULL) {
        return NULL;
    }
 
    conf->enable = NGX_CONF_UNSET;
    conf->max_post_params = NGX_CONF_UNSET_UINT;
 
    return conf;
}
 
 
static char *
ngx_http_anti_hashdos_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_anti_hashdos_loc_conf_t *prev = parent;
    ngx_http_anti_hashdos_loc_conf_t *conf = child;
 
    ngx_conf_merge_value(conf->enable, prev->enable, 0);
    ngx_conf_merge_uint_value(conf->max_post_params, prev->max_post_params,
                              120);
 
    return NGX_CONF_OK;
}
 
 
static ngx_int_t
ngx_http_anti_hashdos_init(ngx_conf_t *cf)
{
    ngx_http_next_input_body_filter = ngx_http_top_input_body_filter;
    ngx_http_top_input_body_filter = ngx_http_anti_hashdos_input_body_filter;
 
    return NGX_OK;
}

The code looks quite straight forward, right? And it is similar to an output body filter, just a few steps:

1) Implement your own input body filter function. e.g.

static ngx_int_t
ngx_http_anti_hashdos_input_body_filter(ngx_http_request_t *r, ngx_buf_t *buf)
{
    /* Do the input body filtering here */
}

2) In your input body filter function, return an HTTP error code if something is wrong. Otherwise, call ngx_http_next_input_body_filter(r, buf) directly to pass the buf to the next input body filters.

3) Install your input body filter in the post_configuration hook function. Push your input body filter to the head of the input body filter chain. e.g.

static ngx_int_t
ngx_http_anti_hashdos_init(ngx_conf_t *cf)
{
    ngx_http_next_input_body_filter = ngx_http_top_input_body_filter;
    ngx_http_top_input_body_filter = ngx_http_anti_hashdos_input_body_filter;
 
    return NGX_OK;
}

NOTE: This is just a demonstration to show how to write input body filters. If you want to fight hash collision DoS attacks completely, you have to write more code and various POST content types should be processed.

Download the code here:
http://www.zhuzhaoyuan.com/download/tengine/anti_hashdos.tar.gz

Tengine, a customized Nginx, goes to open source

Joshua — Fri, 02 Dec 2011 11:31:15 +0000

We’re glad to announce that Tengine, our home-baked Nginx at Taobao now becomes an open source project.

Taobao is the largest e-commerce website in Asia and ranked #12 on Alexa’s top global sites list. Our website serves billions of pageviews per day. For busy website as us, Nginx is obviously the best choice. Thanks to Nginx’s high performance, small footprint and flexibility, we have done more with less.

We first learned the Nginx internals by using it as a traditional web server and developing dozens of modules. Then from June of this year we started hacking the Nginx core to expand its capabilities. As some of the features we have developed may also benefit other Nginx users and websites, so why not open source them? We do not want to be just open source software users, but also open source contributors. That’s why the Tengine open source project came out.

Tengine is based on the latest stable version of Nginx (Nginx-1.0.10). There are a few features and bug fixes you may be interested in Tengine:

Logging enhancement. It supports syslog (local and remote) and pipe logging. You can also do log sampling, i.e. not all requests have to be written.
Protects the server when the system load and memory use goes high.
Combines multiple CSS or JavasScript requests into one request to reduce the downloading time.
Sets the worker process number and CPU affinities automatically. Setting Nginx’s worker_cpu_affinity is not a pain any more.
Enhanced limit_req module with whitelist support and more limit_req directives in one location.
More operations engineer friendly server information, so host can be located easily when error happens.
More command lines support. You can list all modules compiled in and the directives supported, even the content of configuration file itself.
Set expiration for files according to specific content type.
Error pages can be set back to ‘default’.

Basically, Tengine can be considered as a better or superset of Nginx. You can download the tar ball here:
http://tengine.taobao.org/download/tengine-1.2.0.tar.gz

We want to say thank you to the Nginx team, especially to Igor. Thank you very much for your great work! We would love to donate the patches against the Nginx-1.1 branch later if you think the patches are okay.

Frankly, I’m not sure whether the features in Tengine right now can impress you guys or not. It’s the first step we moving towards open source after all. We have built a team working on Tengine and have quite a long to-do list. I promise you more enhancements are coming out.

I’m Back

Joshua — Wed, 16 Nov 2011 09:57:27 +0000

I’m so terrible about keeping my blog up-to-date. This blog has not been updated for 2 years. Sorry guys.

Yet today I have good news for you, if you are an Nginx fan. We are going to open source our home-baked Nginx! This Nginx fork is named Tengine. It serves on thousands of production servers at our website taobao.com. (For those know nothing about Taobao: Taobao is the largest e-commerce website in Asia, and is ranked #15 on Alexa’s top global sites list)

There are quite a few features that you may be interested. I’ll release the more detailed announcement soon.

Nginx Internals (Slides & Video)

Joshua — Tue, 22 Sep 2009 08:14:57 +0000

Last Saturday I gave the talk “Nginx Internals” in Guangzhou. Here are the presentation slides and the video of the talk.

Nginx Internals

View more documents from Joshua Zhu.

Nginx Internals Video part 1 (in Chinese):

Nginx Internals Video part 2 (in Chinese):
UPDATE: This part is lost, saddly, and I have no backup

Nginx Internals Video part 3 (in Chinese):

Nginx Internals Talk in Guangzhou, China

Joshua — Mon, 31 Aug 2009 03:37:52 +0000

nginx map (click to view large image)

I’m going to give a free talk on nginx’s internals next month (September 19), in Guangzhou, China.

I’ve been reading the source code of nginx for a few days. Digging into this charming code is really a pleasant experience, though at first glance it appeared a little bit difficult to understand. Nginx becomes more and more popular, but unfortunately there is not enough documentation on its architecture and implementation. Now that I have spent a considerable amount of time reading the source code and have gained some knowledge, why not share it with those who want to know things under the hood?

So, if you are interested in this talk and you can be in Guangzhou that day, feel free to join in. Please comment on this post or drop me an email to let me know which parts you are interested in (see the mind map above, draft version though).

There might be a thousand Hamlets in a thousand people’s eyes. Note that I’m not Igor, and the only way I try to understand the nuts and bolts is by reverse engineering it, hence I can’t guarantee you no mistakes or misunderstandings in my talk. And frankly, it is not a trivial topic after all, not only because of the size of nginx’s code base, but also its elaborate design.

The speech will be in Chinese while slides will be in English. Specifics of time and location are coming soon. Stay tuned.

Update:
Time: 14:30-17:30, September 19, 2009
Location: Netease Building Tower E, Guangzhou Information Port #16 Keyun RD. Tianhe District, Guangzhou
Registration: http://blog.laiyonghao.com/2009/09/programming-tech-party/370

A Handy Strace Option

Joshua — Thu, 20 Aug 2009 07:39:32 +0000

I didn’t notice the ‘-ff’ option of strace until I came across it today. By turning it on, not only fork(2)s can be followed, but also each process’s trace will be written to tracefile.pid, where pid is the process id of each process. Typical usage might look like this:

# strace -o tracelog.txt -ff -T command

This option can be quite handy, when debugging programs that spawn child processes.

Creating a Hello World! Nginx Module

Joshua — Thu, 13 Aug 2009 07:53:09 +0000

Some of my friends think nginx modules are very difficult to write. Sure it’s not so easy but it’s not that hard either. The only problem is that the documentation is not enough. But don’t let this scare yourself away, the situation is improving.

So I write a hello world nginx module here. It’s pretty short and with enough comments. If you’re a new nginx module developer, feel free to take it as an example and replace hello with whatever your module name is, then start your happy nginx module hacking journey. And I’ll write more topics on nginx soon. Look for it!

ngx_http_hello_module.c:

 
/*
 * Copyright (C) Joshua Zhu, http://www.zhuzhaoyuan.com
 */
 
 
#include 
#include 
#include 
 
 
static char *ngx_http_hello(ngx_conf_t *cf, ngx_command_t *cmd,
    void *conf);
 
static ngx_command_t ngx_http_hello_commands[] = {
    { ngx_string("hello"),
      NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS,
      ngx_http_hello,
      0,
      0,
      NULL },
 
    ngx_null_command
};
 
 
static u_char ngx_hello_string[] = "Hello, world!";
 
 
static ngx_http_module_t ngx_http_hello_module_ctx = {
    NULL,                          /* preconfiguration */
    NULL,                          /* postconfiguration */
 
    NULL,                          /* create main configuration */
    NULL,                          /* init main configuration */
 
    NULL,                          /* create server configuration */
    NULL,                          /* merge server configuration */
 
    NULL,                          /* create location configuration */
    NULL                           /* merge location configuration */
};
 
 
ngx_module_t ngx_http_hello_module = {
    NGX_MODULE_V1,
    &ngx_http_hello_module_ctx,    /* module context */
    ngx_http_hello_commands,       /* module directives */
    NGX_HTTP_MODULE,               /* module type */
    NULL,                          /* init master */
    NULL,                          /* init module */
    NULL,                          /* init process */
    NULL,                          /* init thread */
    NULL,                          /* exit thread */
    NULL,                          /* exit process */
    NULL,                          /* exit master */
    NGX_MODULE_V1_PADDING
};
 
 
static ngx_int_t
ngx_http_hello_handler(ngx_http_request_t *r)
{
    ngx_int_t    rc;
    ngx_buf_t   *b;
    ngx_chain_t  out;
 
    /* we response to 'GET' and 'HEAD' requests only */
    if (!(r->method & (NGX_HTTP_GET|NGX_HTTP_HEAD))) {
        return NGX_HTTP_NOT_ALLOWED;
    }
 
    /* discard request body, since we don't need it here */
    rc = ngx_http_discard_request_body(r);
 
    if (rc != NGX_OK) {
        return rc;
    }
 
    /* set the 'Content-type' header */
    r->headers_out.content_type_len = sizeof("text/html") - 1;
    r->headers_out.content_type.len = sizeof("text/html") - 1;
    r->headers_out.content_type.data = (u_char *) "text/html";
 
    /* send the header only, if the request type is http 'HEAD' */
    if (r->method == NGX_HTTP_HEAD) {
        r->headers_out.status = NGX_HTTP_OK;
        r->headers_out.content_length_n = sizeof(ngx_hello_string) - 1;
 
        return ngx_http_send_header(r);
    }
 
    /* allocate a buffer for your response body */
    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
 
    /* attach this buffer to the buffer chain */
    out.buf = b;
    out.next = NULL;
 
    /* adjust the pointers of the buffer */
    b->pos = ngx_hello_string;
    b->last = ngx_hello_string + sizeof(ngx_hello_string) - 1;
    b->memory = 1;    /* this buffer is in memory */
    b->last_buf = 1;  /* this is the last buffer in the buffer chain */
 
    /* set the status line */
    r->headers_out.status = NGX_HTTP_OK;
    r->headers_out.content_length_n = sizeof(ngx_hello_string) - 1;
 
    /* send the headers of your response */
    rc = ngx_http_send_header(r);
 
    if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
        return rc;
    }
 
    /* send the buffer chain of your response */
    return ngx_http_output_filter(r, &out);
}
 
 
static char *
ngx_http_hello(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_loc_conf_t *clcf;
 
    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
    clcf->handler = ngx_http_hello_handler; /* handler to process the 'hello' directive */
 
    return NGX_CONF_OK;
}

config:

ngx_addon_name=ngx_http_hello_module
HTTP_MODULES="$HTTP_MODULES ngx_http_hello_module"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_hello_module.c"

hello.conf:

# hello
 
worker_processes 1;
 
events {
       worker_connections 1024;
}
 
http {
     server {
     	    listen 8080;
	    server_name localhost;
 
	    location / {
	    	     hello;
	    }
     }     
}

If you want more detailed information, please consult Evan Miller’s Guide To Nginx Module Development.

Learn from Memcached’s Success

Joshua — Mon, 13 Jul 2009 06:51:41 +0000

Memcached becomes more and more popular nowadays. It is widely used by many heavy loaded sites. Why does it succeed?

Well, of course the first and the most important reason is that it meets the need for speed of the web 2.0 sites, by caching data and objects in memory. However, from the point of view of a server developer, what I want to emphasize is that it is the simplicity of memcached’s protocol design makes it more successful. Take a look at memcached’s protocol:

storage: ("set", "add", "replace", "append", "prepend", "cas")
          [noreply]rn
         cas [noreply]rn
     reply: ("ERRORrn", "CLIENT_ERROR rn", "SERVER_ERROR rn",
         "STOREDrn", "NOT_FOUNDrn", "EXISTSrn", "NOT_FOUNDrn")

retrieval: ("get", "gets")
         get rn
         gets rn
     reply: ("ENDrn",
         "VALUE []rnrn")

deletion:
delete [] [noreply]rn
reply: ("DELETEDrn", "NOT_FOUNDrn")

increment/decrement: ("incr", "decr")
         incr [noreply]rn
         decr [noreply]rn
     reply: ("NOT_FOUNDrn",
         "rn")

statistics: ("stat")
         statsrn
         stats rn
     reply: ("STAT rn",
         "STAT items:: rn"
         "ENDrn")

other:
     flush_all
     reply: ("OKrn")
     versionrn
     reply: ("VERSION rn")
     verbosity
     reply: ("OKrn")
     quit

With the textual protocol as shown above, memcache can be easily supported and implemented in various programming languages. No wonder dozens of different memcache clients appear. And then it consequently boosts memcached’s use. Simple thing usually will withstand the test of time. The old simple textual protocols, e.g., HTTP, FTP, SMTP and POP3 are still in use on the modern Internet. Not only because textual protocols can be easily parsed and extended, but also they are convenient for human being to read and debug. This is where the UNIX philosophy shines.

In conclusion, always prefer textual protocol when designing your own application. It would turn out to be really a wise decision.

Tips on High Performance Server Programming

Joshua — Thu, 11 Jun 2009 02:58:10 +0000

Yesterday, I gave a talk entitled “Tips on High Performance Server Programming” to some computer science graduate students at Jinan University. Below are the slides I used for the talk.

Tips on High Performance Server Programming

View more Microsoft Word documents from Joshua Zhu.