helion-prime
home about us blogs contacts

Archive for the ‘general programming’ Category

GitHub good working practice

Monday, June 10th, 2013

GitHub is wonderful place to learn new things and help open source community by participating in various projects. GitHub newcomers often have similar problems with git usage pattern, and I want to describe simple strategy to avoid them.

Work with a fork

The first thing is to fork the repository in your GitHub account. Go thru https://help.github.com/articles/fork-a-repo instructions on how to “Configure Remotes” and “Pull in upstream changes” to keep your fork in sync with changes that happen in the official repository.

Never commit on master branch

Each time you want to commit bug fix or a feature you need to create a branch for it. There is no problem for maintainer of a project to accept your pull request from you master branch but it is problematic for your fork when you want to pull the changes back and your master branch has diverged from upstream.

With this strategy you can think of your master branch as a “landing place” for upstream changes. Even your own commits will end up on master after they have been merged in upstream. Also when you do a pull request on a branch, you can continue to work on another branch and make another pull request.

Before create a new branch pull the changes from upstream, your master need to be uptodate.

$ git fetch upstream
# Pulls in changes not present in your local repository, without modifying
$ git merge upstream/master
# Merges any changes fetched into your working files

Create new branch on your local machine:

$ git branch <name_of_your_branch>

Switch to your new branch :

$ git checkout <name_of_your_branch>

Check current branch you are working on:

$ git branch

Push the branch/changes from your commit on github :

$ git push origin <name_of_your_branch>

Delete a branch on your local filesytem :

$ git branch -d <name_of_your_branch>

Delete the branch on github (you can also make it on github site):

$ git push origin :<name_of_your_branch>

Useful hacks

If you accidentally commit on master but have not pushed your changes:

$ git reset --hard upstream/master
# resets your master branch to the same state as upstream/master.

If you need to merge several commits to one commit:
Say your bug fix branch is called bugfix then on the master branch issue the following command:

$ git merge --squash bugfix
$ git commit
# take all the commits from the bugfix branch, squash them into 1 commit and then merge it with your master branch.

If you need to amend last commit:
Make the fixes. (If you just want to change the log, skip this step.)
Commit the changes in “amend” mode:

$ git commit --all --amend
# your editor will come up asking for a log message (by default, the old log message).

devenv – development environment

Sunday, November 20th, 2011

It’s quite common when developers share information about environment, libraries or tools that they use. And it should be work practice for every good software engineer, manager or architect to constantly evaluate new technologies, ideas, fraimworks, etc.

As software development director I always look into new things and want to share my current understanding of important things in Java and Ruby development, architecture design, and performance.

All the companies are telling us that they are developing the best products to support the development of the software but we will rely on a variety of awards, and statistics of usage whenever possible + my experience. This does not mean it will be possible to do without investing your own time but will hopefully make this job a little easier. I will be glad to get your thoughts, critics, and information about things that empower your IT development process.

So, welcome: http://devenv.in

Deployment of Ruby on Rails applications on OpenBSD

Monday, April 5th, 2010

Preamble

In this post I will define typical production environment on OpenBSD OS for deployment of Ruby on Rails applications.

There are few common things:
1. The Ruby on Rails framework doesn’t support concurrent running in multiple threads within the same process, and so to scale it and fully utilize available hardware we need to execute application in several processes.
2. We need load-balancer to spread incoming requests between application instances.
3. We need separate web-server to serve static content.

Overall configuration

At first all incoming HTTP requests from a clients come to httpd web-server, it servers all static content, and send other requests to HAProxy.
HAProxy receives requests and selects free Thin instance, forwards the request to it, receives a response and passes it back to httpd.

Following diagram should give you basic understanding about common work of components :
deployment diagram

OpenBSD Httpd – standard OpenBSD web-server

I suggest OpenBSD standard web-server as you can find it as part of OpenBSD base installation, it checked for security issues and being updated as part of OpenBSD. We will use it to serve static content and don’t bother our Thin servers.

file: httpd.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
BindAddress SERVER_IP_ADDRESS

# Dynamic Shared Object (DSO) Support
# caching proxy
LoadModule proxy_module /usr/lib/apache/modules/libproxy.so

# allow Symbolic Links for root of our static content and all sub-directories

Options +FollowSymLinks
ServerAdmin ADMIN_EMAIL

# path to root of our static content
DocumentRoot /var/www/railsdocs/RAILS_PROJECT/public
ServerName SERVER_NAME
ServerAlias www.SERVER_NAME

# directories that contain static content (they excluded from dispatching to HAProxy)
ProxyPass /images !
ProxyPass /stylesheets !
ProxyPass /javascripts !
ProxyPass /500.html !
ProxyPass /503.html !

# address where to send and from receive requests (HAProxy listens that address)
ProxyPass / http://127.0.0.1:4000/
ProxyPassReverse / http://127.0.0.1:4000/

# Disallows remote servers to be mapped into the space of the local server.
ProxyRequests Off

# Don't use incoming Host HTTP request header for proxy request.
ProxyPreserveHost Off

ErrorLog logs/SERVER_NAME-error_log
CustomLog logs/SERVER_NAME-access_log common

see for configuration details: man httpd

For OpenBSD 4.6/4.6 -Stable
It’s the hard part, OpenBSD4.6 has a bug in mod_proxy module so ‘!’ directive doesn’t work.
You have to edit following file: /usr/src/usr.sbin/httpd/src/modules/proxy/mod_proxy.c

Find method: static int proxy_trans(request_rec *r)
in that method after condition: if (len > 0) {
add 2 string:

1
2
if (ent[i].real[0] == '!' && ent[i].real[1] == '\0')
return DECLINED;

so final part of code:

1
2
3
if (len > 0) {
if (ent[i].real[0] == '!' && ent[i].real[1] == '\0')
return DECLINED;

Then recompile your system, it’s common procedure for following -stable so you should already know it otherwise see for details for building instructions: http://www.openbsd.org/faq/faq5.html

Thin – high performance ruby web server

We need some Ruby web-server, and it seems that at this time Thin provides best performance.
At least we see such results on Thin homepage: http://code.macournoyer.com/thin/

file: start.sh

1
2
# start work instances
thin start -C thin-production.yml

file: thin-production.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
---
environment
: production

port
: 4001
address
: 127.0.0.1
daemonize
: true
servers
: 4

chdir
: /var/www/railsdocs/RAILS_PROJECT
pid
: tmp/pids/thin.pid
log
: log/thin.log

user
: myuser
group
: mygroup

require
: []

see for configuration details: http://code.macournoyer.com/thin/usage/

HAproxy – TCP/ HTTP load balancer

As Rails doesn’t support concurrent running each incoming request should be assigned to a separate process. HAProxy can be configured to send only one request at a time to every Thin server, it will always pick instance that is not busy with something.

It provides bunch of other useful things like:
– route HTTP requests depending on statically assigned cookies ;
– switch to backup servers in the event a main one fails ;
– accept connections to special ports dedicated to service monitoring ;
– add/modify/delete HTTP headers both ways ;
– block requests matching a particular pattern ;
for full documentation see: http://haproxy.1wt.eu/#docs

note:
If someone thinks that we could use nginx for that purpose check following performance comparison of HAProxy and Nginx:
http://affectioncode.wordpress.com/2008/06/28/another-comparison-of-haproxy-and-nginx/

file: haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
defaults
log     global
mode    http

# provides more detailed information about HTTP contents, such as the request and some cookies
option  httplog
# do not to log any session which didn't transfer any data
option  dontlognull
# allow the proxy to break their persistence and redistribute connections in case of failure
option  redispatch

# the number of attempts to reconnect after a connection failure to a server
retries 3

# the time we accept to wait for a connection to establish on a server
contimeout      100000
# the time we accept to wait for data from the client, or for the client to accept data
clitimeout      100000
# the time we accept to wait for data from the server, or for the server to accept data
srvtimeout      100000

listen project_proxy 127.0.0.1:4000
balance roundrobin

# creates an HTTP 'X-Forwarded-For' header which contains the client's IP address.
# This is useful to let the final web server know what the client address was
option forwardfor

# using “maxconn 1″ improves performance with Rails.
# As Rails instance can process only 1 request “maxconn 1″ force HAProxy to select next free instance

server  app1_1 127.0.0.1:4001 check inter 60000 rise 2 fall 5 maxconn 1
server  app1_2 127.0.0.1:4002 check inter 60000 rise 2 fall 5 maxconn 1
server  app1_3 127.0.0.1:4003 check inter 60000 rise 2 fall 5 maxconn 1
server  app1_4 127.0.0.1:4004 check inter 60000 rise 2 fall 5 maxconn 1

# httpd web-server will handle it in case of 503, 504 errors due to it's static content
errorloc    503  http://DOMAIN_NAME/503.html
errorloc    504  http://DOMAIN_NAME/504.html

# statistics page thru http://127.0.0.1:8080
listen stats 127.0.0.1:8080
balance roundrobin
mode http
stats uri   /

see for configuration details: http://haproxy.1wt.eu/#docs

Tuning of Postgresql under OpenBSD

Thursday, February 25th, 2010

Preamble

I assume that you already made your best with help of your favorite programming language
and recommended postgresql performance tips: http://www.postgresql.org/docs/8.4/static/performance-tips.html

Postgresql resources

No doubt that standard postgresql configuration is far from modern production environments.
Therefore you need to spend enough time with following sources.

1. resource consumption documentation:
http://www.postgresql.org/docs/8.4/static/runtime-config-resource.html

The most important parameters are:
work_mem (integer)
shared_buffers (integer)

2. Query Planning documentation:
http://www.postgresql.org/docs/current/static/runtime-config-query.html

The most important parameters are:
effective_cache_size (integer)
random_page_cost (floating point)

OpenBSD resources

The default sizes in the GENERIC kernel are insignificant also and waiting for your tuning as well.
Posgtresql doesn’t start without enough memory size so always know when you need to increase kern.shminfo.shmmax.

Setting that we can change in /etc/sysctl.conf
the maximum number of System V IPC system-wide semaphore sets (and identifiers) which can exist at any given time:
kern.seminfo.semmni

the maximum total individual System V IPC semaphores which can be assigned by applications:
kern.seminfo.semmns

the amount of shared memory available in the system (bytes):
kern.shminfo.shmmax

the maximum number of shared memory segments:
sysctl kern.shminfo.shmseg

Full list of setting you can see with:
# man sysctl

OpenBSD kernel parameters
So, there are set of parameters that can be tuned only with kernel rebuild.

You should tune them only if system works unstable with default values and you have:
kernel warnings: “uvm_mapent_alloc: out of static map entries”
or panics like: “panic: malloc: out of space in kmem_map”

NKMEMPAGES
This option defines number of pages in kernel kmem_map structure.

MAX_KMAPENT
It defines number of static entries in kernel kmem_map (kernel virtual memory).

They can be changed in:
/usr/src/sys/arch/conf/GENERIC

As start you need to recheck ‘Building the System from Source’ part of OpenBSD documentation:
http://openbsd.org/faq/faq5.htm

Usually administrators select these parameters using set of tests on dedicated testing box where
they emulate load of production servers.

Example

our test server: 1x Intel Quad core CPU, 2GB RAM
software: Ruby on Rails application, postgresql DB, memcached.
load: about 15.000 users/day, peak load: 10 users/sec.

postgresql_dir/data/postgresql.conf

1
2
3
4
5
6
7
8
9
# RESOURCE USAGE
shared_buffers = 738MB
max_prepared_transactions = 30
work_mem = 16MB
max_fsm_pages = 2000000

# QUERY TUNING
effective_cache_size = 512MB
random_page_cost = 1.7

/etc/sysctl.conf

1
2
3
kern.seminfo.semmni = 256
kern.seminfo.semmns = 2048
kern.shminfo.shmmax = 805306368    # Shared memory segment size is 768M

/usr/src/sys/arch/conf/GENERIC

1
2
3
## custom settings
option MAX_KMAPENT = 3072
option NKMEMPAGES = 32768

Log rotation in Ruby on Rails applications

Thursday, November 19th, 2009

preamble

Even on mid-size applications logs grow enormously fast, it’s a pity Rails doesn’t provide built-in functionality for that like Apache web-server does. Fortunately there is an easy and fast solution for that issue.

1. Install cronolog

“cronolog [http://cronolog.org/] is a simple filter program that reads log file entries from standard input and writes each entry to the output file specified by a filename template and the current date and time.”

For details, and list of parameters see: [http://cronolog.org/usage.html]

On GNU/Linux Debian:

# apt-get install cronolog

2. Setup your Rails application

Edit necessary environment file: like ‘config/environments/production.rb’ file

1
2
3
4
5
# system logger configuration
# config.logger = SyslogLogger.new

config.logger = Logger.new(IO.popen("FULL_PATH_TO_CRONOLOG log/production.log.%Y%m%d", "w"))
config.logger.level = Logger::INFO

That’s all, fast and easy .. Enjoy!

©2010 Helion-Prime Solutions Ltd.
Custom Software Development Agile Company.