Running a High-Performance Web Server for BSD

Running a High-Performance Web Server for BSD

Running a High-Performance Web Server for

This document assumes that you have read the appropriate

manual page contains lots of wisdom, especially regarding sysctl

Like other OS''s, the listen queue is often the first

limit hit. The following are comments from "Aaron

Gifford " on how to fix this on

BSDI 1.x, 2.x, and FreeBSD 2.0 (and earlier):


In each file, look for the following:

* Maximum queue length specifiable by listen.

Just change the "5" to whatever appears to work. I bumped the

two machines I was having problems with up to 32 and haven''t

After the edit, recompile the kernel and recompile the

FreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN set

very heavily loaded BSD servers from Chuck Murcko

If you''re running a really busy BSD Apache server, the

following are useful things to do if the system is acting

Run vmstat to check memory usage, page/swap rates,

Run netstat -m to check mbuf usage

Run fstat to check file descriptor usage

These utilities give you an idea what you''ll need to tune in

your kernel, and whether it''ll help to buy more RAM. Here are

some BSD kernel config parameters (actually BSDI, but pertinent

to FreeBSD and other 4.4-lite derivatives) from a system

getting heavy usage. The tools mentioned above were used, and

the system memory was increased to 48 MB before these tuneups.

Maxusers drives a lot of other kernel parameters:

The actual formulae for these derived parameters are in

/usr/src/sys/conf/param.c. These calculated parameters

can also be overridden (in part) by specifying your own values

# Network options. NMBCLUSTERS defines the number of mbuf clusters and

# defaults to 256. This machine is a server that handles lots of traffic,

options CHILD_MAX=512 # maximum number of child processes

options OPEN_MAX=512 # maximum fds (breaks RPC svcs)

In many cases, NMBCLUSTERS must be set much larger than

would appear necessary at first glance. The reason for this is

that if the browser disconnects in mid-transfer, the socket fd

associated with that particular connection ends up in the

TIME_WAIT state for several minutes, during which time its

mbufs are not yet freed. Another reason is that, on server

timeouts, some connections end up in FIN_WAIT_2 state forever,

because this state doesn''t time out on the server, and the

browser never sent a final FIN. For more details see the FIN_WAIT_2 page.

Some more info on mbuf clusters (from sys/mbuf.h):

* Mbufs are of a single size, MSIZE (machine/machparam.h), which

* includes overhead. An mbuf may add a single "mbuf cluster" of size

* MCLBYTES (also in machine/machparam.h), which has no additional overhead

* and is used instead of the internal data area; this is done when

* at least MINCLSIZE of data must be stored.

CHILD_MAX and OPEN_MAX are set to allow up to 512 child

processes (different than the maximum value for processes per

user ID) and file descriptors. These values may change for your

particular configuration (a higher OPEN_MAX value if you''ve got

modules or CGI scripts opening lots of connections or files).

If you''ve got a lot of other activity besides httpd on the same

machine, you''ll have to set NPROC higher still. In this

example, the NPROC value derived from maxusers proved

To increase the size of the listen() queue, you

need to adjust the value of SOMAXCONN. SOMAXCONN is not derived

from maxusers, so you''ll always need to increase that yourself.

We use a value guaranteed to be larger than Apache''s default

for the listen() of 128, currently. The actual value for

SOMAXCONN is set in sys/socket.h. The best way to

adjust this parameter is run-time, rather than changing it in

this header file and thus hardcoding a value in the kernel and

elsewhere. To do this, edit /etc/rc.local and add

/usr/sbin/sysctl -w kern.somaxconn=256

We used 256 but you can tune it for your own

setup. In many cases, however, even the default value of

128 (for later versions of FreeBSD) is OK.

Be aware that your system may not boot with a kernel that is

configured to use more resources than you have available system

RAM. ALWAYS have a known bootable kernel

available when tuning your system this way, and use the system

tools beforehand to learn if you need to buy more memory before

RPC services will fail when the value of OPEN_MAX is larger

than 256. This is a function of the original implementations of

the RPC library, which used a byte value for holding file

descriptors. BSDI has partially addressed this limit in its 2.1

release, but a real fix may well await the redesign of RPC

Finally, there''s the hard limit of child processes

For versions of Apache later than 1.0.5 you''ll need to

change the definition for HARD_SERVER_LIMIT in

httpd.h and recompile if you need to run more than the

# Limit on total number of servers running, i.e., limit on the number

# of clients who can simultaneously connect --- if this limit is ever

# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.

# It is intended mainly as a brake to keep a runaway server from taking

# Unix with it as it spirals down...

Know what you''re doing if you bump this value up, and make sure

you''ve done your system monitoring, RAM expansion, and kernel

tuning beforehand. Then you''re ready to service some serious

Thanks to Tony Sanders and Chris Torek at

BSDI for their helpful suggestions and information.

"M. Teterin" writes:

It really does help if your kernel and frequently used

utilities are fully optimized. Rebuilding the FreeBSD kernel

on an AMD-133 (486-class CPU) web-server with-m486 -fexpensive-optimizations -fomit-frame-pointer

helped reduce the number of "unable" errors, because the CPU

Versions of FreeBSD from August 2000 onwards include a

feature called "accept filters" which delay the return from

accept() until a condition has been met, e.g. an HTTP request

has arrived. This postpones the requirement for a child process

to handle the new connection which therefore increases the

number of connections that a given number of child processes

can handle. It also allows a child process to accomplish more

immediately after accept() returns (because the request is

already available to be read) so there is less context

Accept filters provide the most benefit on servers that are

already so busy that they are configured with "KeepAlive

persistent connections) avoids the cost of setting up a new

connection for every request, but connections that are being

kept alive use up one of the available child processes. Since

there is a limited number of child processes this can

significantly reduce the capacity of the server. The viewers of

a web site will still get a lot of the benefit of persistent

connections even with a very small

KeepAliveTimeout so you should try reducing it

To enable accept filtering, you must either load the

appropriate accept filter module, e.g. with the command

kldload accf_http, or compile a kernel with

options ACCEPT_FILTER_HTTP. Apache will then

enable filtering when it is restarted.

Accept filters are compiled in if the symbol

SO_ACCEPTFILTER is defined on the machine on which

Apache is built. Additionally there is a directive AcceptFilter to switch

the filters on or off. The default is on; except when apache is

compiled with -D AP_ACCEPTFILTER_ON.

If you have tips to contribute, send mail to