Helion-Prime Solutions blog » openBSD

Deployment of Ruby on Rails applications on OpenBSD

alex.shapovalov — Mon, 05 Apr 2010 12:35:56 +0000

Preamble

In this post I will define typical production environment on OpenBSD OS for deployment of Ruby on Rails applications.

There are few common things:
1. The Ruby on Rails framework doesn’t support concurrent running in multiple threads within the same process, and so to scale it and fully utilize available hardware we need to execute application in several processes.
2. We need load-balancer to spread incoming requests between application instances.
3. We need separate web-server to serve static content.

Overall configuration

At first all incoming HTTP requests from a clients come to httpd web-server, it servers all static content, and send other requests to HAProxy.
HAProxy receives requests and selects free Thin instance, forwards the request to it, receives a response and passes it back to httpd.

Following diagram should give you basic understanding about common work of components :

OpenBSD Httpd – standard OpenBSD web-server

I suggest OpenBSD standard web-server as you can find it as part of OpenBSD base installation, it checked for security issues and being updated as part of OpenBSD. We will use it to serve static content and don’t bother our Thin servers.

file: httpd.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

BindAddress SERVER_IP_ADDRESS

# Dynamic Shared Object (DSO) Support
# caching proxy
LoadModule proxy_module /usr/lib/apache/modules/libproxy.so

# allow Symbolic Links for root of our static content and all sub-directories

Options +FollowSymLinks
ServerAdmin ADMIN_EMAIL

# path to root of our static content
DocumentRoot /var/www/railsdocs/RAILS_PROJECT/public
ServerName SERVER_NAME
ServerAlias www.SERVER_NAME

# directories that contain static content (they excluded from dispatching to HAProxy)
ProxyPass /images !
ProxyPass /stylesheets !
ProxyPass /javascripts !
ProxyPass /500.html !
ProxyPass /503.html !

# address where to send and from receive requests (HAProxy listens that address)
ProxyPass / http://127.0.0.1:4000/
ProxyPassReverse / http://127.0.0.1:4000/

# Disallows remote servers to be mapped into the space of the local server.
ProxyRequests Off

# Don't use incoming Host HTTP request header for proxy request.
ProxyPreserveHost Off

ErrorLog logs/SERVER_NAME-error_log
CustomLog logs/SERVER_NAME-access_log common

see for configuration details: man httpd

For OpenBSD 4.6/4.6 -Stable
It’s the hard part, OpenBSD4.6 has a bug in mod_proxy module so ‘!’ directive doesn’t work.
You have to edit following file: /usr/src/usr.sbin/httpd/src/modules/proxy/mod_proxy.c

Find method: static int proxy_trans(request_rec *r)
in that method after condition: if (len > 0) {
add 2 string:

1 2	if (ent[i].real[0] == '!' && ent[i].real[1] == '\0') return DECLINED;

so final part of code:
…

1
2
3

if (len > 0) {
if (ent[i].real[0] == '!' && ent[i].real[1] == '\0')
return DECLINED;

…

Then recompile your system, it’s common procedure for following -stable so you should already know it otherwise see for details for building instructions: http://www.openbsd.org/faq/faq5.html

Thin – high performance ruby web server

We need some Ruby web-server, and it seems that at this time Thin provides best performance.
At least we see such results on Thin homepage: http://code.macournoyer.com/thin/

file: start.sh

1 2	# start work instances thin start -C thin-production.yml

file: thin-production.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

---
environment: production

port: 4001
address: 127.0.0.1
daemonize: true
servers: 4

chdir: /var/www/railsdocs/RAILS_PROJECT
pid: tmp/pids/thin.pid
log: log/thin.log

user: myuser
group: mygroup

require: []

see for configuration details: http://code.macournoyer.com/thin/usage/

HAproxy – TCP/ HTTP load balancer

As Rails doesn’t support concurrent running each incoming request should be assigned to a separate process. HAProxy can be configured to send only one request at a time to every Thin server, it will always pick instance that is not busy with something.

It provides bunch of other useful things like:
– route HTTP requests depending on statically assigned cookies ;
– switch to backup servers in the event a main one fails ;
– accept connections to special ports dedicated to service monitoring ;
– add/modify/delete HTTP headers both ways ;
– block requests matching a particular pattern ;
for full documentation see: http://haproxy.1wt.eu/#docs

note:
If someone thinks that we could use nginx for that purpose check following performance comparison of HAProxy and Nginx:
http://affectioncode.wordpress.com/2008/06/28/another-comparison-of-haproxy-and-nginx/

file: haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

defaults
log global
mode http

# provides more detailed information about HTTP contents, such as the request and some cookies
option httplog
# do not to log any session which didn't transfer any data
option dontlognull
# allow the proxy to break their persistence and redistribute connections in case of failure
option redispatch

# the number of attempts to reconnect after a connection failure to a server
retries 3

# the time we accept to wait for a connection to establish on a server
contimeout 100000
# the time we accept to wait for data from the client, or for the client to accept data
clitimeout 100000
# the time we accept to wait for data from the server, or for the server to accept data
srvtimeout 100000

listen project_proxy 127.0.0.1:4000
balance roundrobin

# creates an HTTP 'X-Forwarded-For' header which contains the client's IP address.
# This is useful to let the final web server know what the client address was
option forwardfor

# using “maxconn 1″ improves performance with Rails.
# As Rails instance can process only 1 request “maxconn 1″ force HAProxy to select next free instance

server app1_1 127.0.0.1:4001 check inter 60000 rise 2 fall 5 maxconn 1
server app1_2 127.0.0.1:4002 check inter 60000 rise 2 fall 5 maxconn 1
server app1_3 127.0.0.1:4003 check inter 60000 rise 2 fall 5 maxconn 1
server app1_4 127.0.0.1:4004 check inter 60000 rise 2 fall 5 maxconn 1

# httpd web-server will handle it in case of 503, 504 errors due to it's static content
errorloc 503 http://DOMAIN_NAME/503.html
errorloc 504 http://DOMAIN_NAME/504.html

# statistics page thru http://127.0.0.1:8080
listen stats 127.0.0.1:8080
balance roundrobin
mode http
stats uri /

see for configuration details: http://haproxy.1wt.eu/#docs

Tuning of Postgresql under OpenBSD

alex.shapovalov — Thu, 25 Feb 2010 15:04:00 +0000

Preamble

I assume that you already made your best with help of your favorite programming language
and recommended postgresql performance tips: http://www.postgresql.org/docs/8.4/static/performance-tips.html

Postgresql resources

No doubt that standard postgresql configuration is far from modern production environments.
Therefore you need to spend enough time with following sources.

1. resource consumption documentation:
http://www.postgresql.org/docs/8.4/static/runtime-config-resource.html

The most important parameters are:
work_mem (integer)
shared_buffers (integer)

2. Query Planning documentation:
http://www.postgresql.org/docs/current/static/runtime-config-query.html

The most important parameters are:
effective_cache_size (integer)
random_page_cost (floating point)

OpenBSD resources

The default sizes in the GENERIC kernel are insignificant also and waiting for your tuning as well.
Posgtresql doesn’t start without enough memory size so always know when you need to increase kern.shminfo.shmmax.

Setting that we can change in /etc/sysctl.conf
the maximum number of System V IPC system-wide semaphore sets (and identifiers) which can exist at any given time:
kern.seminfo.semmni

the maximum total individual System V IPC semaphores which can be assigned by applications:
kern.seminfo.semmns

the amount of shared memory available in the system (bytes):
kern.shminfo.shmmax

the maximum number of shared memory segments:
sysctl kern.shminfo.shmseg

Full list of setting you can see with:
# man sysctl

OpenBSD kernel parameters
So, there are set of parameters that can be tuned only with kernel rebuild.

You should tune them only if system works unstable with default values and you have:
kernel warnings: “uvm_mapent_alloc: out of static map entries”
or panics like: “panic: malloc: out of space in kmem_map”

NKMEMPAGES
This option defines number of pages in kernel kmem_map structure.

MAX_KMAPENT
It defines number of static entries in kernel kmem_map (kernel virtual memory).

They can be changed in:
/usr/src/sys/arch/conf/GENERIC

As start you need to recheck ‘Building the System from Source’ part of OpenBSD documentation:
http://openbsd.org/faq/faq5.htm

Usually administrators select these parameters using set of tests on dedicated testing box where
they emulate load of production servers.

Example

our test server: 1x Intel Quad core CPU, 2GB RAM
software: Ruby on Rails application, postgresql DB, memcached.
load: about 15.000 users/day, peak load: 10 users/sec.

postgresql_dir/data/postgresql.conf

1
2
3
4
5
6
7
8
9

# RESOURCE USAGE
shared_buffers = 738MB
max_prepared_transactions = 30
work_mem = 16MB
max_fsm_pages = 2000000

# QUERY TUNING
effective_cache_size = 512MB
random_page_cost = 1.7

/etc/sysctl.conf

1
2
3

kern.seminfo.semmni = 256
kern.seminfo.semmns = 2048
kern.shminfo.shmmax = 805306368 # Shared memory segment size is 768M

/usr/src/sys/arch/conf/GENERIC

1
2
3

## custom settings
option MAX_KMAPENT = 3072
option NKMEMPAGES = 32768

Fast scalability of Ruby on Rails with mongrel under OpenBSD

alex.shapovalov — Wed, 12 Aug 2009 15:00:30 +0000

preamble

Everybody who works with ruby on rails more then month knows at least 2 things:
it’s great framework, but can’t handle multiple requests simultaneously due to it still doesn’t use threads.

fast start

As scalability is common issue, rubyonrails site wiki provides full set of solutions:
[http://wiki.rubyonrails.org/#deployment_stacks]

But we as fast solution will use great OpenBSD pf (Packet Filter) that capable to do many cool things.
First issue is standard mongrail cluster that can start several instances of mongrail can only create them on one IP with different ports, and pf can’t distribute requests among different ports on same IP.

Here we can create 2 simple scripts that start and destroy server instances:
# cat start.sh

1
2
3
4
5
6
7
8
9
10

count=1
for ip in 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5
do
mongrel_rails start -e production -d --user myapp --group myapp --chdir /var/www/railsdocs/myapp \
--address ${ip} \
--pid /var/www/railsdocs/myapp/tmp/pids/mongrel.${count}.pid \
--log /var/www/railsdocs/myapp/log/mongrel.${count}.log

count=$(($count + 1))
done

# cat stop.sh

1
2
3
4
5

for instance in {1..5}
do
mongrel_rails stop --wait 3 --chdir /var/www/railsdocs/myapp \
--pid /var/www/railsdocs/myapp/tmp/pids/mongrel.${instance}.pid
end

And we need to enable pf:
#/etc/rc.conf.local

1 2	pf=YES #pflogd_flags= # add more flags, ie. "-s 256"

add one string to configuration file for load distribution:
#/etc/pf.conf

1 2	rdr pass log on re0 proto tcp from any to SOME_IP port 80 -> {10.0.0.1, 10.0.0.2, 10.0.0.3, 10.0.0.4, 10.0.0.5 } port 3000 round-robin

check work with:
# pfctl -s nat

monitor with:
tcpdump -n -e -ttt -i pflog0

for more info on pf:
http://www.openbsd.org/faq/pf/
man pf.conf (especially ‘TRANSLATION’ and ‘POOL OPTIONS’ sections)

Then when you see you need more advanced solution for load distribution you can install haproxy.
Check it with [http://www.openbsd.org/4.5_packages/i386/haproxy-1.3.15.7.tgz-long.html]

Authorization with LDAP on OpenBSD

vasiliy.kiryanov — Thu, 07 May 2009 11:22:46 +0000

preamble

common case: you have LDAP server and want to use it to authorize your users on OpenBSD.

First thing is to understand if you really want to use LDAP server for authorization due to OpenBSD doesn’t have build-in support for it.

But has such support for many others authorization styles:
# passwd local password file
# krb5 Kerberos V password
# radius radius authentication
# skey S/Key authentication
# activ activCard X9.9 token authentication
# crypto CRYPTOCard X9.9 token authentication
# snk Digital Pathways SecureNet Key authentication
# token Generic X9.9 token authentication

see for details: man login.conf

setup as pain

1. login_ldap – contact ldap directory server for authentication

install login_ldap package:
# pkg_add -iv login_ldap

use example files in: [/usr/local/share/examples/login_ldap/]
configure it in /etc/login.conf

you should add something like this:

1
2
3
4
5
6

ldap:\
:auth=-ldap:\
:x-ldap-server=127.0.0.1,,ssl:\
:x-ldap-basedn=ou=Users,ou=auth,dc=helion-prime,dc=com:\
:x-ldap-filter=(&(objectclass=posixAccount)(uid=%u)):\
:tc=default:

look for details: man login_ldap

test it with: # /usr/libexec/auth/login_-ldap -d -s login USERNAME ldap

2. ypldap – YP map server using LDAP backend (provide users’ info)

as OpenBSD has great support for YP using of ypldap provides soft integration of LDAP server.

use example in man: man ypldap.conf
configure it in /etc/ypldap.conf

you should have something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

interval 100
domain "helion-prime.com"

provide map "passwd.byname"
provide map "passwd.byuid"
provide map "group.byname"
provide map "group.bygid"

directory "127.0.0.1" {
# directory options
binddn "cn=Manager,dc=helion-prime,dc=com"
bindcred "password" #we don't need it if anonymous searches are allowed
basedn "ou=Users,ou=auth,dc=helion-prime,dc=com"

# passwd maps configuration
passwd filter "(objectClass=posixAccount)"

attribute name maps to "uid"
fixed attribute passwd "*" # we do no need passwords - we use login_ldap for authentication
attribute uid maps to "uidNumber"
attribute gid maps to "gidNumber"
attribute gecos maps to "cn"
attribute home maps to "homeDirectory"
fixed attribute shell "/bin/ksh" # no bash in default install (check it)
fixed attribute change "0" # we can have issues with time format (check it)
fixed attribute expire "0" # we can have issues with time format (check it)
fixed attribute class "ldap" # class of login.conf

# group maps configuration
group filter "(objectClass=posixGroup)"

attribute groupname maps to "cn"
fixed attribute grouppasswd "*"
attribute groupgid maps to "gidNumber"
list groupmembers maps to "memberUid"
}

test it with: # ypldap -dv

as ypldad currently doesn’t support ldap over ssl, you should configure your ldap server to listen over regular ldap.
I believe developer soon will create appropriate support for it.

3. ypbind – create and maintain a binding to a YP server

add your domainname to /etc/defaultdomain
# echo DOMAINNAME > /etc/defaultdomain

the standard way to enable YP passwd support in /etc/master.passwd is to add string: +:::::::::/bin/ksh
use vipw to edit master.passwd

see for details: man 5 passwd

same with groups:
# echo “+:::” >> /etc/group

see for details: man 5 group

4. automate execution

worst part is: we should modify: /etc/rc script

1
2
3
4
5
6
7
8
9
10
11
12

if [ X`domainname` != X ]; then
if [ -d /var/yp/`domainname` ]; then
# YP server capabilities needed...
echo -n ' ypserv'; ypserv ${ypserv_flags}
#echo -n ' ypxfrd'; ypxfrd
fi

#if [ -d /var/yp/binding ]; then
# # YP client capabilities needed...
# echo -n ' ypbind'; ypbind
#fi
....

second entry will run ypbind before ypldap that we exec according to OpenBSD rules in rc.local

Then we should add something like this to: /etc/rc/local

1
2
3
4
5
6
7

if [ X"${ypldap_flags}" != X"NO" ]; then
echo -n ' ypldap'; /usr/sbin/ypldap ${ypldap_flags} 1> /dev/null &
fi

if [ -d /var/yp/binding ]; then
echo -n ' ypbind'; ypbind
fi

And to /etc/rc.conf.local:

1 2	portmap=YES ypldap_flags=""

When I review text I see that it is quite easy and it is hard to believe someone can spend more then hour on this ..
Good luck, guys..