Nginx Internals Video part 1 (in Chinese):

Nginx Internals Video part 2 (in Chinese):

Nginx Internals Video part 3 (in Chinese):

nginx map (click to view large image)

I’m going to give a free talk on nginx’s internals next month (September 19), in Guangzhou, China.

I’ve been reading the source code of nginx for a few days. Digging into this charming code is really a pleasant experience, though at first glance it appeared a little bit difficult to understand. Nginx becomes more and more popular, but unfortunately there is not enough documentation on its architecture and implementation. Now that I have spent a considerable amount of time reading the source code and have gained some knowledge, why not share it with those who want to know things under the hood?

So, if you are interested in this talk and you can be in Guangzhou that day, feel free to join in. Please comment on this post or drop me an email to let me know which parts you are interested in (see the mind map above, draft version though).

There might be a thousand Hamlets in a thousand people’s eyes. Note that I’m not Igor, and the only way I try to understand the nuts and bolts is by reverse engineering it, hence I can’t guarantee you no mistakes or misunderstandings in my talk. And frankly, it is not a trivial topic after all, not only because of the size of nginx’s code base, but also its elaborate design.

The speech will be in Chinese while slides will be in English. Specifics of time and location are coming soon. Stay tuned.

Update:
Time: 14:30-17:30, September 19, 2009
Location: Netease Building Tower E, Guangzhou Information Port #16 Keyun RD. Tianhe District, Guangzhou
Registration: http://blog.laiyonghao.com/2009/09/programming-tech-party/370

# strace -o tracelog.txt -ff -T command

This option can be quite handy, when debugging programs that spawn child processes.

So I write a hello world nginx module here. It’s pretty short and with enough comments. If you’re a new nginx module developer, feel free to take it as an example and replace hello with whatever your module name is, then start your happy nginx module hacking journey. And I’ll write more topics on nginx soon. Look for it! 

ngx_http_hello_module.c:

 
/*
 * Copyright (C) Joshua Zhu, http://www.zhuzhaoyuan.com
 */
 
 
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
 
 
static char *ngx_http_hello(ngx_conf_t *cf, ngx_command_t *cmd,
    void *conf);
 
static ngx_command_t ngx_http_hello_commands[] = {
    { ngx_string("hello"),
      NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS,
      ngx_http_hello,
      0,
      0,
      NULL },
 
    ngx_null_command
};
 
 
static u_char ngx_hello_string[] = "Hello, world!";
 
 
static ngx_http_module_t ngx_http_hello_module_ctx = {
    NULL,                          /* preconfiguration */
    NULL,                          /* postconfiguration */
 
    NULL,                          /* create main configuration */
    NULL,                          /* init main configuration */
 
    NULL,                          /* create server configuration */
    NULL,                          /* merge server configuration */
 
    NULL,                          /* create location configuration */
    NULL                           /* merge location configuration */
};
 
 
ngx_module_t ngx_http_hello_module = {
    NGX_MODULE_V1,
    &ngx_http_hello_module_ctx,    /* module context */
    ngx_http_hello_commands,       /* module directives */
    NGX_HTTP_MODULE,               /* module type */
    NULL,                          /* init master */
    NULL,                          /* init module */
    NULL,                          /* init process */
    NULL,                          /* init thread */
    NULL,                          /* exit thread */
    NULL,                          /* exit process */
    NULL,                          /* exit master */
    NGX_MODULE_V1_PADDING
};
 
 
static ngx_int_t
ngx_http_hello_handler(ngx_http_request_t *r)
{
    ngx_int_t    rc;
    ngx_buf_t   *b;
    ngx_chain_t  out;
 
    /* we response to 'GET' and 'HEAD' requests only */
    if (!(r->method & (NGX_HTTP_GET|NGX_HTTP_HEAD))) {
        return NGX_HTTP_NOT_ALLOWED;
    }
 
    /* discard request body, since we don't need it here */
    rc = ngx_http_discard_request_body(r);
 
    if (rc != NGX_OK) {
        return rc;
    }
 
    /* set the 'Content-type' header */
    r->headers_out.content_type_len = sizeof("text/html") - 1;
    r->headers_out.content_type.len = sizeof("text/html") - 1;
    r->headers_out.content_type.data = (u_char *) "text/html";
 
    /* send the header only, if the request type is http 'HEAD' */
    if (r->method == NGX_HTTP_HEAD) {
        r->headers_out.status = NGX_HTTP_OK;
        r->headers_out.content_length_n = sizeof(ngx_hello_string) - 1;
 
        return ngx_http_send_header(r);
    }
 
    /* allocate a buffer for your response body */
    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
        return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
 
    /* attach this buffer to the buffer chain */
    out.buf = b;
    out.next = NULL;
 
    /* adjust the pointers of the buffer */
    b->pos = ngx_hello_string;
    b->last = ngx_hello_string + sizeof(ngx_hello_string) - 1;
    b->memory = 1;    /* this buffer is in memory */
    b->last_buf = 1;  /* this is the last buffer in the buffer chain */
 
    /* set the status line */
    r->headers_out.status = NGX_HTTP_OK;
    r->headers_out.content_length_n = sizeof(ngx_hello_string) - 1;
 
    /* send the headers of your response */
    rc = ngx_http_send_header(r);
 
    if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
        return rc;
    }
 
    /* send the buffer chain of your response */
    return ngx_http_output_filter(r, &out);
}
 
 
static char *
ngx_http_hello(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_core_loc_conf_t *clcf;
 
    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
    clcf->handler = ngx_http_hello_handler; /* handler to process the 'hello' directive */
 
    return NGX_CONF_OK;
}

config:

ngx_addon_name=ngx_http_hello_module
HTTP_MODULES="$HTTP_MODULES ngx_http_hello_module"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_hello_module.c"

hello.conf:

# hello
 
worker_processes 1;
 
events {
       worker_connections 1024;
}
 
http {
     server {
     	    listen 8080;
	    server_name localhost;
 
	    location / {
	    	     hello;
	    }
     }     
}

If you want more detailed information, please consult Evan Miller’s Guide To Nginx Module Development.

Well, of course the first and the most important reason is that it meets the need for speed of the web 2.0 sites, by caching data and objects in memory. However, from the point of view of a server developer, what I want to emphasize is that it is the simplicity of memcached’s protocol design makes it more successful. Take a look at memcached’s protocol:

With the textual protocol as shown above, memcache can be easily supported and implemented in various programming languages. No wonder dozens of different memcache clients appear. And then it consequently boosts memcached’s use. Simple thing usually will withstand the test of time. The old simple textual protocols, e.g., HTTP, FTP, SMTP and POP3 are still in use on the modern Internet. Not only because textual protocols can be easily parsed and extended, but also they are convenient for human being to read and debug. This is where the UNIX philosophy shines.

In conclusion, always prefer textual protocol when designing your own application. It would turn out to be really a wise decision.

A screenshot of Dstat

By using Dstat, now not only all my needs can be met but also it is pretty easy to do resource usage analysis and graphing.

As the author described, “Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.” So why use three or more tools when one tool can give you everything you need?

Here is an example showing how to use it.

First, capture the resource usage information (CPU and memory) to a file, e.g. stat.dat:

$ dstat -tcmn > stat.dat

Then use the scripts below to create CPU and memory usage graphs:

#!/usr/bin/gnuplot
 
set terminal png
set output "cpu.png"
set title "CPU usage"
set xlabel "time"
set ylabel "percent"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"
plot "stat.dat" using 1:4 title "system" with lines, 
"stat.dat" using 1:3 title "user" with lines, 
"stat.dat" using 1:5 title "idle" with lines

#!/usr/bin/gnuplot
 
set terminal png
set output "memory.png"
set title "memory usage"
set xlabel "time"
set ylabel "size(M Bytes)"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"
plot "stat.dat" using 1:9 title "used" with lines, 
"stat.dat" using 1:10 title "buff" with lines, 
"stat.dat" using 1:11 title "cach" with lines, 
"stat.dat" using 1:12 title "free" with lines

Resource usage graph examples:

Download the scripts.
http://www.zhuzhaoyuan.com/download/dstat/cpu.sh
http://www.zhuzhaoyuan.com/download/dstat/memory.sh

It is true, however, TIME_WAIT and CLOSE_WAIT are the most confusing two among TCP’s 11 states (CLOSED, LISTEN, SYN_SENT, SYN_RECV, ESTABLISHED, CLOSE_WAIT, LAST_ACK, FIN_WAIT1, FIN_WAIT2, CLOSING, and TIME_WAIT), which are displayed by netstat(1).

Before going into detail, let me explain the terms first:
Active Open: An end sends a SYN segment to the other end by calling connect(2). This end is usually called a client.
Passive Open: An end issues a passive open by calling socket(2), bind(2), and listen(2) so that it can accept(2) the clients’ connections. This end is usually called a server.
Active Close: An end performs active close when it calls close(2) first. It results a FIN segment being sent.
Passive Close: The other end receives the FIN segment performs the passive close.
MSL: Maximum Segment Lifetime, the maximum time a segment can live in the network before being discarded.

Now take a look at the TCP state transition diagram.

TCP State Transition Diagram 1

TCP State Transition Diagram 2

In the diagrams above, I denote the active open/close transitions with a green line and the passive open/close transitions with a blue line (Normal transitions are in bold). The end that performs the active close goes through the TIME_WAIT state; while the end performs the passive close enters the CLOSE_WAIT state. Note that either end (the server or the client) can perform the active close!

What’s the TIME_WAIT state for?
1) When the final ACK being sent by the end that performed the active close was lost, the other end would resend a FIN. TCP maintains the information of the connection, hence it would resend the lost ACK rather than respond with an RST (interpreted as error).
2) To allow wandering duplicates to expire in the network, so that the duplicate segments would not corrupt later connections.

The value of TIME_WAIT delay is 2MSL, which is TCP implementation dependent and is generally about 1-4 minutes.

CLOSE_WAIT happens when an end has received a FIN segment from the other end, to wait for the program to close the socket. Long duration of CLOSE_WAIT suggests that there might be a bug in your program. In fact, a TCP connection can stay in the CLOSE_WAIT state forever unless it is explicitly closed. It is the application’s responsibility to close its socket after use, to release the resource of the connection.

Conclusion:
In most cases, there is no need to worry about TIME_WAIT. It is a safety feature of TCP. But you really should take care of CLOSE_WAIT. Also note, if a connection is in CLOSE_WAIT, it would never end up in the TIME_WAIT state, and vice versa. BTW, if you haven’t read Richard Stevens’ Unix Network Programming, go out and buy one. It is a must-have for all network programmers, even if you do Windows programming.

Download the dot files and msc file here:
http://www.zhuzhaoyuan.com/download/tcp/tcp-state1.dot
http://www.zhuzhaoyuan.com/download/tcp/tcp-state2.dot
http://www.zhuzhaoyuan.com/download/tcp/tcp_close.msc

MySQL Proxy has only long options. Therefore, it is a little bit inconvenient to type the long command line. For example, suppose we are going to setup a MySQL Proxy server forwarding connections to two MySQL backends. We also want to 1) use a Lua script, 2) start the proxy in daemon mode, 3) log all messages of level debug and higher to a specific file, and 4) write the proxy’s pid to its pid file. What would the command line look like?

$ mysql-proxy --proxy-address=192.168.0.189:3307 \
--proxy-backend-addresses=192.168.0.189:3306 \
--proxy-backend-addresses=192.168.0.192:3306 \
--proxy-lua-script=/home/josh/mysql-proxy/dispatch-by-client-address.lua \
--daemon \
--log-file=/home/josh/mysql-proxy/mysql-proxy.log --log-level=debug \
--pid-file=/home/josh/mysql-proxy/mysql-proxy.pid

As you can see, it’s quite long. Not to mention, MySQL Proxy can be plugged into many plugins, which could import more options.

Is there any way to make it less repetitive? Yes. By taking advantage of MySQL Proxy’s configuration file, we no longer need to type such long command line.

MySQL Proxy uses GLib’s key file (GKeyFile) as its configuration file format. A key file is very similar to a .ini file. It consists of groups of key value pairs which can be strings, booleans, integers and lists of these. Each key value pair must be contained in a group, where the name appears between enclosed square brackets.

Here goes the proxy’s configuration file of the example above:

# MySQL Proxy's configuration file (mysql-proxy.cnf)
 
[mysql-proxy]
daemon = true
pid-file = /home/josh/mysql-proxy/mysql-proxy.pid
log-file = /home/josh/mysql-proxy/mysql-proxy.log
log-level = debug
proxy-address = 192.168.0.189:3307
proxy-backend-addresses = 192.168.0.192:3306,192.168.0.189:3306
proxy-lua-script = /home/josh/mysql-proxy/dispatch-by-client-address.lua

And the command line:

$ mysql-proxy --defaults-file=mysql-proxy.cnf

It is apparently much shorter. We now just need to specify the name of the configuration file!

Some notes:
1) GLib uses the ‘;’ character as the default list separator, while MySQL Proxy uses ‘,’.
2) Ungrouped keys are not allowed in key files.
3) The –version and –defaults-file options can not appear in MySQL Proxy’s configuration file.
4) A handy feature: the values in MySQL Proxy’s configuration file can be overridden by the command line options.

To learn more about GKeyFile, please visit:
http://www.gtkbook.com/tutorial.php?page=keyfile
http://library.gnome.org/devel/glib/2.18/glib-Key-value-file-parser.html
http://standards.freedesktop.org/desktop-entry-spec/desktop-entry-spec-1.0.html

First, if you don’t have the EPEL (Extra Packages for Enterprise Linux) repository enabled, you should enable it:

# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-2.noarch.rpm

Make sure you have GNU Autotools, flex, pkg-config, and bazaar, as well as MySQL client libraries installed. If not, please follow this command:

# yum install autoconf automake libtool flex pkgconfig bzr mysql-devel

Since the version of libevent that CentOS ships is too old, you need to build a newer one (>= 1.4.0, for better threading support):

$ wget http://monkey.org/~provos/libevent-1.4.9-stable.tar.gz
$ tar zvfx libevent-1.4.9-stable.tar.gz
$ cd libevent-1.4.9-stable
$ ./configure
$ make
# make install

Again, CentOS 5.2 ships with an old version of GLib, a newer one is required (>= 2.16.0, for the GLib testing framework):

$ wget http://ftp.gnome.org/pub/gnome/sources/glib/2.18/glib-2.18.4.tar.gz
$ tar zvfx glib-2.18.4.tar.gz
$ cd glib-2.18.4
$ ./configure
$ make
# make install

And Lua 5.1 should be installed:

$ wget http://www.lua.org/ftp/lua-5.1.4.tar.gz
$ tar zvfx lua-5.1.4.tar.gz
$ cd lua-5.1.4
$ make linux
# make install
# cp etc/lua.pc /usr/local/lib/pkgconfig/

Important: you should make pkg-config know where the libraries are!

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig

Now, check out the latest source code of MySQL Proxy and build:

$ bzr branch lp:mysql-proxy
$ cd mysql-proxy
$ ./autogen.sh
$ ./configure
$ make
# make install

Run it and see if everything is okay:

$ mysql-proxy -V

Done!