Wednesday, January 30, 2013

Setting up Nginx for Socket.io Loadbalancing

This is basically the documentary for what i tried to use nginx to load balance socket.io.

PCRE (Perl Compatible Regular Expressions)


# download PCRE 8.32
[Unix] [http://sourceforge.net/projects/pcre/files/pcre/]
[Windows] [http://gnuwin32.sourceforge.net/packages/pcre.htm]
# extract the tarball
# ./configure
# make
# sudo make install (it'll install the library under /usr/local/lib)

OpenSSL


# download OpenSSL 1.0.1c [http://www.openssl.org/source/openssl-1.0.1c.tar.gz]
# extract the tarball
# ./config
# make
# sudo make install

Nginx


# download Nginx 1.2.6 [http://nginx.org/en/download.html]
# extract the tarball
# ./configure \--with-http_ssl_module
# make
# sudo make install

To start Nginx


# sudo LD_LIBRARY_PATH=/usr/local/lib /usr/local/nginx/sbin/nginx
[or] set LD_LIBRARY_PATH in environment
# open url [http://localhost] should get the default Nginx welcome page


Welcome to nginx\!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.


Reference installation guide


[http://www.thegeekstuff.com/2011/07/install-nginx-from-source/] [http://nginx.org/en/docs/install.html]

Enable SSL on Nginx

  •  edit nginx.config, under server add
server {
        listen 80;
        listen 443 ssl;

        ssl_certificate /some/location/ssl.crt;
        ssl_certificate_key /some/location/privateKey.key;
        ssl_protocols        SSLv3 TLSv1;
        ssl_ciphers HIGH:!aNULL:!MD5;

        server_name host;
  • please note that this setup only works for normal http traffic but not socket.io traffic.  please refer to attempt 4 for the nginx.conf setup.

Using Nginx as websocket reverse proxy


Node.js setup


This is the sample socket.io example from http://socket.io

server.js
var app = require('express')()
  , server = require('http').createServer(app)
  , io = require('socket.io').listen(server);

var port = (typeof process.argv[2] !== 'undefined') ? process.argv[2] : 8080;
console.log( 'port ' + port );
server.listen(port);

app.get('/', function(req, res) {
  res.sendfile(__dirname + '/index.html');
});

io.sockets.on('connection', function(socket) {
  socket.emit('news', {hello: 'world'} );
  socket.on('my other event', function(data) {
    console.log(data);
  });
});

index.html

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('https://host');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });
</script>
note, https://host when testing SSL, http://host when testing without SSL.  you will get lots of warning messages on the browser if using http in the socket while the index.html is accessing through https.
The page at https://host/ displayed insecure content from http://host/socket.io/1/xhr-polling/S-vUnEsPSCzECUcZIITT?t=1359502588702.

First attempt: forward websocket using Nginx 1.2.6 directly to Node.js

nginx.conf

server {
        listen 80;
        server_name host;
        location / {
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header Host $http_host;
          proxy_set_header X-NginX-Proxy true;

          proxy_pass http://node;
          proxy_redirect off;
        }
}

upstream node {
        server 127.0.0.1:7010;
}

socket.io debug output without nginx

   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized 5VBe_ZAT56ApdzVLIITR
   debug - setting request GET /socket.io/1/websocket/5VBe_ZAT56ApdzVLIITR
   debug - set heartbeat interval for client 5VBe_ZAT56ApdzVLIITR
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }

socket.io debug output with nginx

   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/websocket/LeOmKCCLQYk37baLIITS
   debug - set heartbeat interval for client LeOmKCCLQYk37baLIITS
   warn  - websocket connection invalid
   info  - transport end (undefined)
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - cleared close timeout for client LeOmKCCLQYk37baLIITS
   debug - cleared heartbeat interval for client LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394012
   debug - setting poll timeout
   debug - client authorized for
   debug - clearing poll timeout
   debug - xhr-polling writing 1::
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394181
   debug - setting poll timeout
   debug - clearing poll timeout
   debug - xhr-polling writing 5:::{"name":"news","args":[{"hello":"world"}]}
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - discarding transport
   debug - cleared close timeout for client LeOmKCCLQYk37baLIITS
   debug - xhr-polling received data packet 5:::{"name":"my other event","args":[{"my":"data"}]}
{ my: 'data' }
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394355
   debug - setting poll timeout
   debug - discarding transport

Same result for https. 

Conclusions

Websocket failed and fall back to XHR.  and it is noticibly slower than websocket.

Second Attempt: Nginx 1.3.11

Same as 1.2.6

Thrid Attempt: Nginx 1.2.6 with websocket upgrade header

nginx.conf

location / {
    chunked_transfer_encoding off;
    proxy_http_version 1.1;
    proxy_pass        http://localhost:9001;
    proxy_buffering   off;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host $host:9001;  #probaby need to change this
    proxy_set_header  Connection "Upgrade";
    proxy_set_header  Upgrade websocket;
}

Result

req url:/
   debug - destroying non-socket.io upgrade
req url:/favicon.ico
   debug - destroying non-socket.io upgrade

socket.io - manager.js

Manager.prototype.handleUpgrade = function (req, socket, head) {
  var data = this.checkRequest(req)
    , self = this;
  if (!data) {
    if (this.enabled('destroy upgrade')) {
      socket.end();
      this.log.debug('destroying non-socket.io upgrade');
    }
    return;
  }
  req.head = head;
  this.handleClient(data, req);
};

Manager.prototype.checkRequest = function (req) {
  var resource = this.get('resource');
console.log( 'req url:' + req.url );
  var match;
  if (typeof resource === 'string') {
    match = req.url.substr(0, resource.length);
    if (match !== resource) match = null;
  } else {
    match = resource.exec(req.url);
    if (match) match = match[0];
  }

socket.io check the url if it's started with socket.io, like below (without nginx).  note that all socket.io communication is gone with nginx.

Socket.io debug without nginx

req url:/
req url:/socket.io/socket.io.js
   debug - served static content /socket.io.js
req url:/socket.io/1/?t=1359507255627
   debug - client authorized
   info  - handshake authorized tjAKZXz4h8Cv3EeK7xyq
req url:/favicon.ico
req url:/socket.io/1/websocket/tjAKZXz4h8Cv3EeK7xyq
   debug - setting request GET /socket.io/1/websocket/tjAKZXz4h8Cv3EeK7xyq

Conclusions

it appears normal websocket will work, but socket.io is looking for something which the setup doesn't provide.

Disable destroy upgrade

var app = require('express')()
  , server = require('http').createServer(app)
  , io = require('socket.io').listen(server);
io.set("destroy upgrade",false);

the page won't load.  probably because that option only means socket.io won't destroy the connection but won't serve it either.

Only do websocket upgrade for /socket.io

        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;

            proxy_pass http://node;
            proxy_buffering off;
        }

        location /socket.io {
            chunked_transfer_encoding off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Upgrade websocket;

            proxy_http_version 1.1;
            proxy_pass http://node;
            proxy_buffering off;
            proxy_redirect off;
        }

socket.io debug log

req url:/
req url:/socket.io/socket.io.js
   warn  - unknown transport: "undefined"
req url:/favicon.ico

Socket.io - manager.js

Manager.prototype.handleClient = function (data, req) {
....
  if (!~this.get('transports').indexOf(data.transport)) {
    this.log.warn('unknown transport: "' + data.transport + '"');
    req.connection.end();
    return;
  }
looks like data is lost.

Forth Attempt: Nginx 1.2.6 with TCP proxy module


  1. # goto http://yaoweibin.github.com/nginx_tcp_proxy_module/
  2. # download tarball
  3. # extract tarball
  4. # cd nginx-1.2.6
  5. # patch -p1 < ../yaoweibin-nginx_tcp_proxy_module-f1d5c62/tcp.patch
patching file src/core/ngx_log.c
Hunk #1 succeeded at 67 (offset 1 line).
patching file src/core/ngx_log.h
Hunk #1 succeeded at 30 (offset 1 line).
Hunk #2 succeeded at 38 (offset 1 line).
patching file src/event/ngx_event_connect.h
Hunk #1 succeeded at 33 (offset 1 line).
Hunk #2 succeeded at 44 (offset 1 line).
  1. # ./configure --add-module=../yaoweibin-nginx_tcp_proxy_module-f1d5c62 --with-http_ssl_module
  2. # make
  3. # sudo make install

Nginx.config

tcp {
        upstream node {
                server 127.0.0.1:7010;
                server 127.0.0.1:7020;
                check interval=3000 rise=2 fall=5 timeout=1000;
        }

        server {
                listen 80;
                listen       443 ssl;
                server_name  host;

                ssl_certificate /home/arthur/ssl/ssl.crt;
                ssl_certificate_key /home/arthur/ssl/privateKey.key;
                ssl_protocols SSLv3 TLSv1;
                ssl_ciphers HIGH:!aNULL:!MD5;

                tcp_nodelay on;
                proxy_pass node;
        }
}
http {
...
    server {
        listen 7000;
        location /websocket_status {
            check_status;
        }

Socket.io debug log

req url:/
req url:/socket.io/socket.io.js
   debug - served static content /socket.io.js
req url:/socket.io/1/?t=1359508983909
   debug - client authorized
   info  - handshake authorized Ky0M5xgxULfhGesu85-D
req url:/favicon.ico
req url:/socket.io/1/websocket/Ky0M5xgxULfhGesu85-D
   debug - setting request GET /socket.io/1/websocket/Ky0M5xgxULfhGesu85-D
   debug - set heartbeat interval for client Ky0M5xgxULfhGesu85-D
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }
   debug - emitting heartbeat for client Ky0M5xgxULfhGesu85-D
   debug - websocket writing 2::
   debug - set heartbeat timeout for client Ky0M5xgxULfhGesu85-D
   debug - got heartbeat packet
   debug - cleared heartbeat timeout for client Ky0M5xgxULfhGesu85-D
   debug - set heartbeat interval for client Ky0M5xgxULfhGesu85-D

HTTPS

https resulted with the same output.  didn't fall back to XHR.

Fail over

browser conenct to one of the node.js in the upstream server.  kill the node.js, the connection reconnected to the other node.js immediately.

Socket.io debug output for node listening on port 7020

$ node server 7020
   info  - socket.io started
port 7020
   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized AvAMO0vBRo3BbzCBGntx
   debug - setting request GET /socket.io/1/websocket/AvAMO0vBRo3BbzCBGntx
   debug - set heartbeat interval for client AvAMO0vBRo3BbzCBGntx
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }
{code}

kill node 7020.

Socket.io debug output for node listening on port 7010

$ node server 7010
   info  - socket.io started
port 7010
   debug - client authorized
   info  - handshake authorized GpQU1HmA7Ei8bZt1GmSD
   debug - setting request GET /socket.io/1/websocket/GpQU1HmA7Ei8bZt1GmSD
   debug - set heartbeat interval for client GpQU1HmA7Ei8bZt1GmSD
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }

note that the log for the first time connect & reconnect on the server side is exactly the same.  it even execute the init code to send out the test events.

Concerns: Splitting websocket & non-websocket traffic on the same port

[http://www.exratione.com/2012/07/proxying-websocket-traffic-for-nodejs-the-present-state-of-play/]
The TCP Proxy Module for Nginx Doesn't Solve the Problem

The excellent nginx_tcp_proxy_module allows Nginx to be used as a proxy for websocket traffic, as outlined in a post from last year. Unfortunately it is really only intended for load balancing - you can't use it to split out websocket versus non-websocket traffic arriving on the same port and send them to different destinations based on URI. Additionally, this requires a custom build to include the module into Nginx - which might be an issue for future support and upgrades.
so, if we want to serve static content from the same nginx too, it has to be on another port.  i.e. you'll have to open up another port for the static content.

Licensing Concern

Not sure how valid the concern is, but Weibin mentioned that he borrowed code from Igor & Jack's work.

https://github.com/yaoweibin/nginx_tcp_proxy_module
Copyright & License
    This README template copy from agentzh (<http://github.com/agentzh>).

    I borrowed a lot of code from upstream and mail module from the nginx
    0.7.* core. This part of code is copyrighted by Igor Sysoev. And the
    health check part is borrowed the design of Jack Lindamood's healthcheck
    module healthcheck_nginx_upstreams
    (<http://github.com/cep21/healthcheck_nginx_upstreams>);

    This module is licensed under the BSD license.

    Copyright (C) 2012 by Weibin Yao <yaoweibin@gmail.com>.

    All rights reserved.