Wednesday, January 30, 2013

Setting up Nginx for Socket.io Loadbalancing

This is basically the documentary for what i tried to use nginx to load balance socket.io.

PCRE (Perl Compatible Regular Expressions)


# download PCRE 8.32
[Unix] [http://sourceforge.net/projects/pcre/files/pcre/]
[Windows] [http://gnuwin32.sourceforge.net/packages/pcre.htm]
# extract the tarball
# ./configure
# make
# sudo make install (it'll install the library under /usr/local/lib)

OpenSSL


# download OpenSSL 1.0.1c [http://www.openssl.org/source/openssl-1.0.1c.tar.gz]
# extract the tarball
# ./config
# make
# sudo make install

Nginx


# download Nginx 1.2.6 [http://nginx.org/en/download.html]
# extract the tarball
# ./configure \--with-http_ssl_module
# make
# sudo make install

To start Nginx


# sudo LD_LIBRARY_PATH=/usr/local/lib /usr/local/nginx/sbin/nginx
[or] set LD_LIBRARY_PATH in environment
# open url [http://localhost] should get the default Nginx welcome page


Welcome to nginx\!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.


Reference installation guide


[http://www.thegeekstuff.com/2011/07/install-nginx-from-source/] [http://nginx.org/en/docs/install.html]

Enable SSL on Nginx

  •  edit nginx.config, under server add
server {
        listen 80;
        listen 443 ssl;

        ssl_certificate /some/location/ssl.crt;
        ssl_certificate_key /some/location/privateKey.key;
        ssl_protocols        SSLv3 TLSv1;
        ssl_ciphers HIGH:!aNULL:!MD5;

        server_name host;
  • please note that this setup only works for normal http traffic but not socket.io traffic.  please refer to attempt 4 for the nginx.conf setup.

Using Nginx as websocket reverse proxy


Node.js setup


This is the sample socket.io example from http://socket.io

server.js
var app = require('express')()
  , server = require('http').createServer(app)
  , io = require('socket.io').listen(server);

var port = (typeof process.argv[2] !== 'undefined') ? process.argv[2] : 8080;
console.log( 'port ' + port );
server.listen(port);

app.get('/', function(req, res) {
  res.sendfile(__dirname + '/index.html');
});

io.sockets.on('connection', function(socket) {
  socket.emit('news', {hello: 'world'} );
  socket.on('my other event', function(data) {
    console.log(data);
  });
});

index.html

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('https://host');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });
</script>
note, https://host when testing SSL, http://host when testing without SSL.  you will get lots of warning messages on the browser if using http in the socket while the index.html is accessing through https.
The page at https://host/ displayed insecure content from http://host/socket.io/1/xhr-polling/S-vUnEsPSCzECUcZIITT?t=1359502588702.

First attempt: forward websocket using Nginx 1.2.6 directly to Node.js

nginx.conf

server {
        listen 80;
        server_name host;
        location / {
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header Host $http_host;
          proxy_set_header X-NginX-Proxy true;

          proxy_pass http://node;
          proxy_redirect off;
        }
}

upstream node {
        server 127.0.0.1:7010;
}

socket.io debug output without nginx

   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized 5VBe_ZAT56ApdzVLIITR
   debug - setting request GET /socket.io/1/websocket/5VBe_ZAT56ApdzVLIITR
   debug - set heartbeat interval for client 5VBe_ZAT56ApdzVLIITR
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }

socket.io debug output with nginx

   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/websocket/LeOmKCCLQYk37baLIITS
   debug - set heartbeat interval for client LeOmKCCLQYk37baLIITS
   warn  - websocket connection invalid
   info  - transport end (undefined)
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - cleared close timeout for client LeOmKCCLQYk37baLIITS
   debug - cleared heartbeat interval for client LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394012
   debug - setting poll timeout
   debug - client authorized for
   debug - clearing poll timeout
   debug - xhr-polling writing 1::
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394181
   debug - setting poll timeout
   debug - clearing poll timeout
   debug - xhr-polling writing 5:::{"name":"news","args":[{"hello":"world"}]}
   debug - set close timeout for client LeOmKCCLQYk37baLIITS
   debug - discarding transport
   debug - cleared close timeout for client LeOmKCCLQYk37baLIITS
   debug - xhr-polling received data packet 5:::{"name":"my other event","args":[{"my":"data"}]}
{ my: 'data' }
   debug - setting request GET /socket.io/1/xhr-polling/LeOmKCCLQYk37baLIITS?t=1359502394355
   debug - setting poll timeout
   debug - discarding transport

Same result for https. 

Conclusions

Websocket failed and fall back to XHR.  and it is noticibly slower than websocket.

Second Attempt: Nginx 1.3.11

Same as 1.2.6

Thrid Attempt: Nginx 1.2.6 with websocket upgrade header

nginx.conf

location / {
    chunked_transfer_encoding off;
    proxy_http_version 1.1;
    proxy_pass        http://localhost:9001;
    proxy_buffering   off;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host $host:9001;  #probaby need to change this
    proxy_set_header  Connection "Upgrade";
    proxy_set_header  Upgrade websocket;
}

Result

req url:/
   debug - destroying non-socket.io upgrade
req url:/favicon.ico
   debug - destroying non-socket.io upgrade

socket.io - manager.js

Manager.prototype.handleUpgrade = function (req, socket, head) {
  var data = this.checkRequest(req)
    , self = this;
  if (!data) {
    if (this.enabled('destroy upgrade')) {
      socket.end();
      this.log.debug('destroying non-socket.io upgrade');
    }
    return;
  }
  req.head = head;
  this.handleClient(data, req);
};

Manager.prototype.checkRequest = function (req) {
  var resource = this.get('resource');
console.log( 'req url:' + req.url );
  var match;
  if (typeof resource === 'string') {
    match = req.url.substr(0, resource.length);
    if (match !== resource) match = null;
  } else {
    match = resource.exec(req.url);
    if (match) match = match[0];
  }

socket.io check the url if it's started with socket.io, like below (without nginx).  note that all socket.io communication is gone with nginx.

Socket.io debug without nginx

req url:/
req url:/socket.io/socket.io.js
   debug - served static content /socket.io.js
req url:/socket.io/1/?t=1359507255627
   debug - client authorized
   info  - handshake authorized tjAKZXz4h8Cv3EeK7xyq
req url:/favicon.ico
req url:/socket.io/1/websocket/tjAKZXz4h8Cv3EeK7xyq
   debug - setting request GET /socket.io/1/websocket/tjAKZXz4h8Cv3EeK7xyq

Conclusions

it appears normal websocket will work, but socket.io is looking for something which the setup doesn't provide.

Disable destroy upgrade

var app = require('express')()
  , server = require('http').createServer(app)
  , io = require('socket.io').listen(server);
io.set("destroy upgrade",false);

the page won't load.  probably because that option only means socket.io won't destroy the connection but won't serve it either.

Only do websocket upgrade for /socket.io

        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;

            proxy_pass http://node;
            proxy_buffering off;
        }

        location /socket.io {
            chunked_transfer_encoding off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Upgrade websocket;

            proxy_http_version 1.1;
            proxy_pass http://node;
            proxy_buffering off;
            proxy_redirect off;
        }

socket.io debug log

req url:/
req url:/socket.io/socket.io.js
   warn  - unknown transport: "undefined"
req url:/favicon.ico

Socket.io - manager.js

Manager.prototype.handleClient = function (data, req) {
....
  if (!~this.get('transports').indexOf(data.transport)) {
    this.log.warn('unknown transport: "' + data.transport + '"');
    req.connection.end();
    return;
  }
looks like data is lost.

Forth Attempt: Nginx 1.2.6 with TCP proxy module


  1. # goto http://yaoweibin.github.com/nginx_tcp_proxy_module/
  2. # download tarball
  3. # extract tarball
  4. # cd nginx-1.2.6
  5. # patch -p1 < ../yaoweibin-nginx_tcp_proxy_module-f1d5c62/tcp.patch
patching file src/core/ngx_log.c
Hunk #1 succeeded at 67 (offset 1 line).
patching file src/core/ngx_log.h
Hunk #1 succeeded at 30 (offset 1 line).
Hunk #2 succeeded at 38 (offset 1 line).
patching file src/event/ngx_event_connect.h
Hunk #1 succeeded at 33 (offset 1 line).
Hunk #2 succeeded at 44 (offset 1 line).
  1. # ./configure --add-module=../yaoweibin-nginx_tcp_proxy_module-f1d5c62 --with-http_ssl_module
  2. # make
  3. # sudo make install

Nginx.config

tcp {
        upstream node {
                server 127.0.0.1:7010;
                server 127.0.0.1:7020;
                check interval=3000 rise=2 fall=5 timeout=1000;
        }

        server {
                listen 80;
                listen       443 ssl;
                server_name  host;

                ssl_certificate /home/arthur/ssl/ssl.crt;
                ssl_certificate_key /home/arthur/ssl/privateKey.key;
                ssl_protocols SSLv3 TLSv1;
                ssl_ciphers HIGH:!aNULL:!MD5;

                tcp_nodelay on;
                proxy_pass node;
        }
}
http {
...
    server {
        listen 7000;
        location /websocket_status {
            check_status;
        }

Socket.io debug log

req url:/
req url:/socket.io/socket.io.js
   debug - served static content /socket.io.js
req url:/socket.io/1/?t=1359508983909
   debug - client authorized
   info  - handshake authorized Ky0M5xgxULfhGesu85-D
req url:/favicon.ico
req url:/socket.io/1/websocket/Ky0M5xgxULfhGesu85-D
   debug - setting request GET /socket.io/1/websocket/Ky0M5xgxULfhGesu85-D
   debug - set heartbeat interval for client Ky0M5xgxULfhGesu85-D
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }
   debug - emitting heartbeat for client Ky0M5xgxULfhGesu85-D
   debug - websocket writing 2::
   debug - set heartbeat timeout for client Ky0M5xgxULfhGesu85-D
   debug - got heartbeat packet
   debug - cleared heartbeat timeout for client Ky0M5xgxULfhGesu85-D
   debug - set heartbeat interval for client Ky0M5xgxULfhGesu85-D

HTTPS

https resulted with the same output.  didn't fall back to XHR.

Fail over

browser conenct to one of the node.js in the upstream server.  kill the node.js, the connection reconnected to the other node.js immediately.

Socket.io debug output for node listening on port 7020

$ node server 7020
   info  - socket.io started
port 7020
   debug - served static content /socket.io.js
   debug - client authorized
   info  - handshake authorized AvAMO0vBRo3BbzCBGntx
   debug - setting request GET /socket.io/1/websocket/AvAMO0vBRo3BbzCBGntx
   debug - set heartbeat interval for client AvAMO0vBRo3BbzCBGntx
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }
{code}

kill node 7020.

Socket.io debug output for node listening on port 7010

$ node server 7010
   info  - socket.io started
port 7010
   debug - client authorized
   info  - handshake authorized GpQU1HmA7Ei8bZt1GmSD
   debug - setting request GET /socket.io/1/websocket/GpQU1HmA7Ei8bZt1GmSD
   debug - set heartbeat interval for client GpQU1HmA7Ei8bZt1GmSD
   debug - client authorized for
   debug - websocket writing 1::
   debug - websocket writing 5:::{"name":"news","args":[{"hello":"world"}]}
{ my: 'data' }

note that the log for the first time connect & reconnect on the server side is exactly the same.  it even execute the init code to send out the test events.

Concerns: Splitting websocket & non-websocket traffic on the same port

[http://www.exratione.com/2012/07/proxying-websocket-traffic-for-nodejs-the-present-state-of-play/]
The TCP Proxy Module for Nginx Doesn't Solve the Problem

The excellent nginx_tcp_proxy_module allows Nginx to be used as a proxy for websocket traffic, as outlined in a post from last year. Unfortunately it is really only intended for load balancing - you can't use it to split out websocket versus non-websocket traffic arriving on the same port and send them to different destinations based on URI. Additionally, this requires a custom build to include the module into Nginx - which might be an issue for future support and upgrades.
so, if we want to serve static content from the same nginx too, it has to be on another port.  i.e. you'll have to open up another port for the static content.

Licensing Concern

Not sure how valid the concern is, but Weibin mentioned that he borrowed code from Igor & Jack's work.

https://github.com/yaoweibin/nginx_tcp_proxy_module
Copyright & License
    This README template copy from agentzh (<http://github.com/agentzh>).

    I borrowed a lot of code from upstream and mail module from the nginx
    0.7.* core. This part of code is copyrighted by Igor Sysoev. And the
    health check part is borrowed the design of Jack Lindamood's healthcheck
    module healthcheck_nginx_upstreams
    (<http://github.com/cep21/healthcheck_nginx_upstreams>);

    This module is licensed under the BSD license.

    Copyright (C) 2012 by Weibin Yao <yaoweibin@gmail.com>.

    All rights reserved.

Monday, January 28, 2013

JBoss OSGi

We have migrated to JBoss from Tomcat+Equinox recently.  There's a challenge to do the migration basically the documenation in jboss is a bit less than desired.

Here is the osgi subsystem in standalone.xml we finally use.

        <subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="eager">
            <properties>
                <property name="org.jboss.osgi.system.modules.extra">
                    org.apache.log4j, org.apache.commons.logging, org.slf4j, javax.xml.bind.api
                </property>
                <property name="org.osgi.framework.startlevel.beginning">
                    5
                </property>
                <property name="org.osgi.framework.system.packages.extra">
                    org.apache.log4j;version=1.2.15, org.apache.commons.logging;version=1.1.1, org.slf4j;version=1.6, javax.activation;version=1.1.1, javax.xml.bind;version=2.2, sun.security.provider;version=1.0
                </property>
            </properties>
            <capabilities>
                <capability name="javax.servlet.api"/>
                <capability name="javax.transaction.api"/>
                <capability name="javax.xml.bind.api"/>
                <capability name="javax.api"/>
                <capability name="javax.enterprise.api"/>
                <capability name="org.apache.felix.log" startlevel="1"/>
                <capability name="org.apache.felix.configadmin" startlevel="1"/>
                <capability name="org.jboss.osgi.logging" startlevel="1"/>
                <capability name="org.jboss.logmanager"/>
                <capability name="javax.ws.rs.api" startlevel="1"/>
                <capability name="org.jboss.resteasy.resteasy-jaxrs" startlevel="3"/>
                <capability name="ch.qos.cal10n"/>
                <capability name="org.apache.commons.beanutils"/>
                <capability name="org.apache.commons.codec"/>
                <capability name="org.apache.commons.collections"/>
                <capability name="org.apache.commons.lang"/>
                <capability name="org.apache.commons.io"/>
                <capability name="org.codehaus.jackson.jackson-core-asl"/>
                <capability name="org.codehaus.jackson.jackson-jaxrs"/>
                <capability name="org.codehaus.jackson.jackson-mapper-asl"/>
                <capability name="org.codehaus.jackson.jackson-xc"/>
            </capabilities>
        </subsystem>

The first thing we discovered is we have to use JBoss' logging.  It's well integrated, easy to use, but if you tried to deploy your own slf4j, log4j... it just won't work.

The 2nd thing is the org.jboss.osgi.system.modules.extra & org.osgi.framework.system.packages.extra, without them, some of the package are exported without version.  And what you see here is a list of all that I can find on different blog posts or forum.

Also, there're two version of javax.servlet.api, one being javax.servlet.api:v25 and the one i've used here is for servlet api 3.0 which is required by resteasy.

We also use resteasy-jackson-provider, but the one in jboss module exported a package which "masks" the same package exported in resteasy-jaxrs.  I've raised a JIRA about it.  Meanwhile, i've repackaged the resteasy-jackson-provider so it won't export that package.

At last, jboss also comes with a jboss config admin integration, which will add the configurations to the configadmin subsystem in standalone.xml.  However, it doesn't support list as value (Apache Felix FileInstall supports that).  Thus, if you are using FileInstall, you'll have to disable the jboss config admin (by not adding the capability to osgi subsystem)


Sunday, January 27, 2013

How to add service properties to services created by Managed Service Factory

When using managed-service-factory,

The bean created will be registered as OSGi service having bean-name set to <factory id>.<uuid> which is of no use to tell the service apart.

In this example, the configuration has a name property and we'd like to set the name to the cache instance so that it can be imported like this,

<reference interface="org.sprintframework.cache.Cache" filter="(cacheName=abcde)"/>

Gemini Blueprint 1.0.2 does allow <service-property> in the managed-service-factory, but it appears it only support simple value, not programatically.

To do that, we need to get the ServiceRegistration for the service (which is returned when a service is added to the registry) as that is the only way to update a service's properties.  Luckily, the managed-service-factory bean will actually return a list of ServiceRegistration that it created.

You can simply get the list of ServiceRegistration from the factory bean if you are sure the configuration is always set before you make the call to get the list but in our case, we use something like FileInstall which will add configuration automatically, we'll have to register a service listener and update the new service's property accordingly. 

Another issue is that the serviceChanged is actually called before the new ServiceRegistration is added to the list.  thus, if we get the list of service registrations in the serviceChanged call, we will always miss the newly created one.

The service properties dictionary returned (in JBoss) is a JBoss internal class which is immutable, so i have to create a new dictionary (didn't use the internal class to avoid dependency) and overwrite the internal implementation.  Didn't test if JBoss will convert the hashtable to its internal implementation when I call setProperties but it seems to work fine.

public class ManagedServicePropertyUpdater implements ApplicationContextAware, InitializingBean, ServiceListener {
    private OsgiBundleXmlApplicationContext osgiAppCtx;
    private BundleContext bundleContext;
    private BeanFactory beanFactory;

    @Getter @Setter
    private String serviceInterfaceClassName;
    @Getter @Setter
    private String managedServiceFactoryBeanName;
    @Getter @Setter
    private String servicePropertyKey;
    @Getter @Setter
    private String servicePropertyValueMethodName;

    @Override
    public void afterPropertiesSet() throws Exception {
        assert (serviceInterfaceClassName!=null);
        assert (managedServiceFactoryBeanName!=null);
        assert (servicePropertyKey!=null);
        assert (servicePropertyValueMethodName!=null);
    }
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        osgiAppCtx = (OsgiBundleXmlApplicationContext)applicationContext;
        bundleContext = osgiAppCtx.getBundleContext();
        beanFactory = osgiAppCtx.getBeanFactory();
        String filter = "(" + Constants.OBJECTCLASS + "=" + serviceInterfaceClassName + ")";
        try {
            bundleContext.addServiceListener(this,filter);
        } catch( InvalidSyntaxException  e ) {
            throw new BeanCreationException("fail registering service listener for Cache", e);
        }
        Collection serviceRegistrations = (Collection)beanFactory.getBean(managedServiceFactoryBeanName);
        for( Object sr : serviceRegistrations) {
            addServiceProperty((ServiceRegistration) sr);
        }

    }

    @Override
    public void serviceChanged(ServiceEvent event) {
        final ServiceReference reference = event.getServiceReference();
        ExecutorService exec = Executors.newSingleThreadExecutor();
        exec.submit( new Runnable() {
            @Override
            public void run() {
                ServiceRegistration registration = mustFindServiceRegistration(reference);
                addServiceProperty(registration);
            }
        });
        exec.shutdown();
    }

    private ServiceRegistration findServiceRegistration(ServiceReference reference) {
        Collection serviceRegistrations = (Collection)beanFactory.getBean(managedServiceFactoryBeanName);
        for( Object srObj : serviceRegistrations ) {
            ServiceRegistration sr = (ServiceRegistration) srObj;
            ServiceReference ref = sr.getReference();
            if( ref.compareTo(reference) == 0 ){
                return sr;
            }
        }
        return null;
    }

    private ServiceRegistration mustFindServiceRegistration(ServiceReference serviceReference) {
        ServiceRegistration serviceRegistration = null;
        int count = 0;
        do{
            if( count > 0 ) {
                try {
                    Thread.sleep(1000);
                } catch( InterruptedException e ){}
            }
            count ++;
            serviceRegistration = findServiceRegistration(serviceReference);
        } while( serviceRegistration == null && count < 10);
        if( serviceRegistration == null )
            throw new IllegalArgumentException("cannot find service" );
        return serviceRegistration;
    }

    private void addServiceProperty(ServiceRegistration sr) {
        try {
            ServiceReference ref = sr.getReference();
            Object service = bundleContext.getService(ref);
            if( !(service instanceof Cache) ) {
                return;
            }
            Dictionary dict = OsgiServiceReferenceUtils.getServiceProperties(ref);
            if( dict.get(servicePropertyKey) == null ) {
                Dictionary newProps = new Hashtable();
                Enumeration keys = dict.keys();
                while( keys.hasMoreElements() ) {
                    String key = keys.nextElement().toString();
                    newProps.put(key, dict.get(key) );
                }
                Class serviceClass = service.getClass();
                Method getMethod = serviceClass.getMethod(servicePropertyValueMethodName);
                String servicePropertyValue = (String)getMethod.invoke(service);
                newProps.put( servicePropertyKey, servicePropertyValue );
                sr.setProperties(newProps);
            }
            bundleContext.ungetService(ref);
        } catch( Exception e ) {
            e.printStackTrace();
        }
    }
}

Using Spring Security in OSGi without WAB

This is actually inspired by another post that use Apache CXF in OSGi (but I couldn't find the link now).

Apache CXF D-OSGi, you can add a propety to the OSGi service so that it will use the filter (ServletFilter)

    <osgi:service ref="searchRS" interface="...search.webservice.SearchWS">       
        <osgi:service-properties>
            <entry key="service.exported.interfaces" value="*" />
            <entry key="service.exported.configs" value="org.apache.cxf.rs" />
            <entry key="org.apache.cxf.rs.httpservice.context" value="/search" />
            <entry key="org.apache.cxf.httpservice.requirefilter" value="true" />
               <entry key="org.apache.cxf.rs.provider">
                   <array>
                       <ref bean="jsonProvider" />
                   </array>
               </entry>
        </osgi:service-properties>
    </osgi:service>

and

    <osgi:service ref="customFilterChain" interface="javax.servlet.Filter">
        <osgi:service-properties>
            <entry key="org.apache.cxf.httpservice.filter" value="true" />
            <entry key="servletNames" value="none" />
        </osgi:service-properties>
    </osgi:service>

With this, the customFilterChain will be used everytime when /search has been accessed.

To setup the filter chain,

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:osgi="http://www.eclipse.org/gemini/blueprint/schema/blueprint"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:security="http://www.springframework.org/schema/security"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
                        http://www.springframework.org/schema/beans/spring-beans.xsd
                        http://www.springframework.org/schema/security           
                        http://www.springframework.org/schema/security/spring-security.xsd
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint/gemini-blueprint.xsd">

    <bean id="customSecurityFilter" class="...security.filter.CustomSecurityFilter"/>

    <bean id="requestContextFilter" class="org.springframework.web.filter.RequestContextFilter"/>
    <!-- disable url rewrite, change session key -->
    <bean id="httpSessionSecurityContextRepository" class="org.springframework.security.web.context.HttpSessionSecurityContextRepository" />
   
    <bean id="springSecurityFilter" class="org.springframework.security.web.context.SecurityContextPersistenceFilter" >
        <constructor-arg ref="httpSessionSecurityContextRepository" />
        <property name="forceEagerSessionCreation" value="false" />
    </bean>
   
    <bean id="basicAuthenticationFilter" class="org.springframework.security.web.authentication.www.BasicAuthenticationFilter" >
        <constructor-arg ref="customAuthenticationManager" />
    </bean>
   
    <bean id="http403ForbiddenEntryPoint" class="org.springframework.security.web.authentication.Http403ForbiddenEntryPoint"/>
    <bean id="exceptionTranslationFilter" class="org.springframework.security.web.access.ExceptionTranslationFilter">
        <constructor-arg ref="http403ForbiddenEntryPoint" /> <!-- can redirect to https here -->
    </bean>
   
    <bean id="sessionManagementFilter" class="org.springframework.security.web.session.SessionManagementFilter">
        <constructor-arg ref="httpSessionSecurityContextRepository"/>
        <constructor-arg ref="sessionFixationProtectionStrategy" />
    </bean>
   
    <bean id="sessionFixationProtectionStrategy" class="org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy">
        <property name="migrateSessionAttributes" value="true"/>
    </bean>

    <bean id="customFilterChain" class="org.springframework.security.web.FilterChainProxy">
        <security:filter-chain-map request-matcher="ant">
            <security:filter-chain pattern="/osgi/auth/**" filters="springSecurityFilter,requestContextFilter,basicAuthenticationFilter,sessionManagementFilter,exceptionTranslationFilter"/>
            <security:filter-chain pattern="/**" filters="springSecurityFilter,requestContextFilter,customSecurityFilter,exceptionTranslationFilter"/>
        </security:filter-chain-map>
    </bean>
   
</beans>

note that we have our own AuthenticationProvider and only /auth will perform Basic Authentication Filter.

After we've migrated to Resteasy, we're still keeping the filter chain, and instead of registering to OSGi and let Apache CXF handles it, we registered to the WebContainer directly (see my other post about Resteasy & Pax Web).

Resteasy & Gemini Blueprint (Springdm) & Pax Web

inspired by sarbarian, I have created a gemini blueprint version which registered to pax web (in fact any other HTTP service will do)

import webcontainer from pax web in osgi-context.xml.

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0
            http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
    <reference id="webContainer" interface="org.ops4j.pax.web.service.WebContainer" />
    <reference id="customFilterChain" interface="javax.servlet.Filter"/>
</blueprint>

define reateasy server bean in module-context.xml.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:osgi="http://www.eclipse.org/gemini/blueprint/schema/blueprint"
       xmlns:osgix="http://www.eclipse.org/gemini/blueprint/schema/blueprint-compendium"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
                        http://www.springframework.org/schema/beans/spring-beans.xsd
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint/gemini-blueprint.xsd
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint-compendium
                        http://www.eclipse.org/gemini/blueprint/schema/blueprint-compendium/gemini-blueprint-compendium.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <bean id="resteasy-server" class="...impl.ResteasyServiceImpl" init-method="init" destroy-method="cleanup">
        <property name="webContainer" ref="webContainer"/>
        <property name="securityFilter" ref="customFilterChain" />
        <property name="resteasyServlet" ref="resteasyServlet" />
        <property name="resteasyBootstrap" ref="resteasyBootstrap" />
    </bean>

    <bean id="resteasyServlet" class="...ResteasyServlet" />
    <bean id="resteasyBootstrap" class="...servlet.ResteasyBootstrap" />

</beans>
In the init method, register Resteasy as if you are doing that in the web.xml, which in our case, we don't have web.xml (we didn't use WAB)
            /*
                <listener>
                    <listener-class>org.jboss.resteasy.plugins.server.servlet.ResteasyBootstrap</listener-class>
                </listener>

                <servlet>
                    <servlet-name>Resteasy</servlet-name>
                    <servlet-class>org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher</servlet-class>
                </servlet>

                <servlet-mapping>
                    <ervlet-name>Resteasy</servlet-name>
                    <url-pattern>/osgi/*</url-pattern>
                </servlet-mapping>

                <context-param>
                    <param-name>resteasy.servlet.mapping.prefix</param-name>
                    <param-value>/osgi</param-value>
                </context-param>
                <context-param>
                    <param-name>resteasy.scan</param-name>
                    <param-value>false</param-value>
                </context-param>
             */

private void init() {
        try {
            HttpContext httpContext = webContainer.getDefaultSharedHttpContext();
            // context param must be set first to the http context
            Dictionary contextParam = new Hashtable();
            contextParam.put("resteasy.servlet.mapping.prefix", "/osgi" );
            contextParam.put("resteasy.scan", "false" );
            webContainer.setContextParam(contextParam, httpContext);

            // add security filter
            webContainer.registerFilter(securityFilter,new String[]{"/osgi/*"}, new String[] {"customSecurityFilterChain"}, null, httpContext );
            logger.info( "Security Filter has been registered to /osgi/*" );
            // register resteasy bootstrap listener which would initialize resteasy framework
            // it will also add the registry to the servlet context
            webContainer.registerEventListener( resteasyBootstrap, httpContext );

            // register resteasy dispatcher servlet
            webContainer.registerServlet( resteasyServlet, "Resteasy", new String[]{ "/osgi/*" }, null, httpContext );
            servletContext = resteasyServlet.getServletConfig().getServletContext();

            //ResteasyProviderFactory providerFactory = ResteasyProviderFactory.getInstance();
            ResteasyProviderFactory providerFactory = (ResteasyProviderFactory) servletContext.getAttribute(ResteasyProviderFactory.class.getName());
            ResteasyJacksonProvider jacksonProvider = new ResteasyJacksonProvider();
            ObjectMapper objectMapper = jaxbUtils.getJsonMapper();
            jacksonProvider.setMapper( objectMapper );
            providerFactory.addBuiltInMessageBodyWriter(jacksonProvider);
            providerFactory.addBuiltInMessageBodyReader(jacksonProvider);

            try {
                ByteArrayProvider byteArrayProvider = new ByteArrayProvider();
                providerFactory.addBuiltInMessageBodyWriter(byteArrayProvider);

                DefaultTextPlain defaultTextPlain = new DefaultTextPlain();
                providerFactory.addBuiltInMessageBodyWriter(defaultTextPlain);

                StringTextStar stringTextStar = new StringTextStar();
                providerFactory.addBuiltInMessageBodyWriter(stringTextStar);
            }catch( Throwable ignore ) {
                logger.debug( "defect in reasteasy-jackson-provider, should not export org.jboss.resteasy.plugins.providers");
            }
            logger.info("RESTEasy Framework started on osgi");

            /* register service listener to add JAX-RS annotated class to resteasy */
            bundleContext.addServiceListener(this, serviceFilter);
            ServiceReference[] references = bundleContext.getServiceReferences( (String)null, serviceFilter);
            for(ServiceReference reference : references) {
                try {
                    Object service = bundleContext.getService(reference);
                    addResource(service);
                    bundleContext.ungetService(reference);
                } catch ( Throwable ignore ) {
                    logger.debug( "exception in adding jaxrs resource to resteasy" );
                }
            }

        } catch (Throwable e) {
            logger.error("Error registering resteasy servlet", e);
        }

    }

    @Override
    public void serviceChanged(ServiceEvent event) {
        ServiceReference sr = event.getServiceReference();
        switch(event.getType()) {
            case ServiceEvent.REGISTERED:
                try {
                    addResource(bundleContext.getService(sr));
                    bundleContext.ungetService(sr);
                } catch (Exception e) {
                    e.printStackTrace();
                }
                break;

            case ServiceEvent.UNREGISTERING:
                try {
                    removeResource(bundleContext.getService(sr));
                    bundleContext.ungetService(sr);
                } catch (Exception e) {
                    e.printStackTrace();
                }
                break;
        }
    }

    @Override
    public void addResource(Object resource) {
        if (resource != null ) {
            for( Class _interface : resource.getClass().getInterfaces() ){
                if( _interface.isAnnotationPresent( Path.class ) ) {
                    logger.info("Adding JAX-RS resource: " + resource.getClass().getName());
                    Registry resteasyRegistry = getRegistry();
                    if( resteasyRegistry != null )
                        resteasyRegistry.addSingletonResource(resource);
                    logger.debug("resource " + resource.getClass().getName() + " added to resteasy");
                    break;
                }
            }
        }
    }

    @Override
    public void removeResource(Object resource) {
        if (resource != null && resource.getClass().isAnnotationPresent(Path.class)) {
            logger.info("Removing JAX-RS resource: " + resource.getClass().getName());
            getRegistry().removeRegistrations(resource.getClass());
        }
    }

    public Registry getRegistry() {
        if (servletContext != null) {
            return (Registry) servletContext.getAttribute(Registry.class.getName());
        }else{
            return null;
        }
    }

Please note that you'll have to implement BundleContextAware to get the BundleContext and ServiceListener to listen to new services.

at last, to clean up when the bundle is being unload (or refresh)


    private void cleanup() {
        try {
            webContainer.unregisterEventListener(resteasyBootstrap);
        }catch ( Exception ignored ) {}
        try {
            webContainer.unregisterServlet(resteasyServlet);
        }catch ( Exception ignored ) {}
        try {
            webContainer.unregisterFilter(securityFilter);
        }catch ( Exception ignored ) {}
        try {
            webContainer.getDefaultSharedHttpContext().deregisterBundle(bundleContext.getBundle());
        }catch ( Exception ignored ) {}
        try {
            bundleContext.removeServiceListener(this);
        }catch ( Exception ignored ) {}

    }



Wrapper bundle for non-OSGi packages

All we need is add Bundle-ClassPath to the wrapper's manifest.mf and packaged it with dependency plugin.

Below is an example of how to build a wrapper for Jedis.

Manifest-Version: 1.0
Bundle-Version: 2.1.0
Bundle-ManifestVersion: 2
Bundle-Name: Jedis Wrapper
Bundle-Description: Wrapper bundle for Jedis
Import-Package: org.slf4j, org.apache.commons.pool.impl, org.apache.commons.pool
Export-Package: redis.clients.jedis;version="2.1.0",redis.clients.util;version="2.1.0",redis.clients.jedis.exceptions;version="2.1.0"
Bundle-SymbolicName: wrapper-jedis_bundle
Bundle-ClassPath: .,jedis-2.1.0.jar

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>redis.clients</groupId>
    <artifactId>wrapper-jedis</artifactId>
    <packaging>jar</packaging>
    <version>2.1.0</version>
    <name>wrapper-jedis</name>
    <url>http://maven.apache.org</url>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>prepare-package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <includeArtifactIds>jedis</includeArtifactIds>
                            <outputDirectory>${project.build.directory}/classes</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
    <dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>2.1.0</version>
    </dependency>
    <dependency>
    <groupId>commons-pool</groupId>
    <artifactId>commons-pool</artifactId>
    <version>1.5</version>
</dependency>
    </dependencies>
</project>

Granted, the package exported or imported is a bit trail-and-error.  don't know if there's any good tools that can look into the bytecode of a jar file and generate the manifest file.  maybe bnd can, but haven't try that. 

OSGi class loading issue

One of the difficulty when using OSGi is that if there are more than one version (or copies) of the same class available in different bundle, it is often difficult to tell what went wrong.

java.lang.LinkageError: loader constraint violation in interface itable initialization: when resolving method "org.eclipse.jetty.server.Request.getServletContext()Ljavax/servlet/ServletContext;" the class loader (instance of org/jboss/osgi/framework/internal/HostBundleClassLoader) of the current class, org/eclipse/jetty/server/Request, and the class loader (instance of org/jboss/modules/ModuleClassLoader) for interface javax/servlet/ServletRequest have different Class objects for the type javax/servlet/ServletContext used in the signature
22:44:51,428 ERROR [stderr] (qtp1026914784-119 Selector0)     at org.eclipse.jetty.server.AbstractHttpConnection.<init>(AbstractHttpConnection.java:151)

org.springframework.aop.AopInvocationException: AOP configuration seems to be invalid: tried calling method [public abstract void org.ops4j.pax.web.service.WebContainer.registerServlet(javax.servlet.Servlet,java.lang.String,java.lang.String[],java.util.Dictionary,org.osgi.service.http.HttpContext) throws javax.servlet.ServletException] on target [org.ops4j.pax.web.service.internal.HttpServiceProxy@28a2f76f]; nested exception is java.lang.IllegalArgumentException: argument type mismatch
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:326)

These are two of the error I got recently because there are actually two copies of ServletContext.  The first exception happens when I tried to call the rest interface (resteasy with pax web jetty bundle), the framework threw a LinkageError which has a very good error message showing both are loaded with a different classloader.  The 2nd error is when my code tried to register a servlet (which get the ServletContext from HostBundleClassLoader) to resteasy (which gets it from ModuleClassLoader). 

I've developed a little code to print out more details about where the class is loaded.
private Map<String,Map<String,Object>> findClass(BundleContext context, String name) {
        Map<String,Map<String,Object>> result = new HashMap<String,Map<String,Object>>();
        for (Bundle b : context.getBundles()) {
            try {
                Class<?> c = b.loadClass(name);
                Dictionary<String,String> dict = b.getHeaders();
                Map<String,Object> value = new HashMap<String, Object>();
                Enumeration<String> keys = dict.keys();
                while( keys.hasMoreElements() ) {
                    String key = keys.nextElement();
                    value.put(key,dict.get(key));
                }
                value.put("class", c );
                value.put("classLoader", c.getClassLoader() );
                value.put("location", c.getClassLoader().getResource(name.replace('.','/')+".class"));
                result.put(b.getSymbolicName() + " " + c, value );
            } catch (ClassNotFoundException e) {
                // No problem, this bundle doesn't have the class
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        return result;
    }

The idea is simple.  For each bundle, call findClass (which use the classloader of the bundle) and if it's found, get the headers, class loader & the location of the class itself and put everything in a hashmap.

Note that it's running on JBoss, the header dictionary it returned is unmutable.

Turns out the problem is because "bundles" from jboss' module directory (/standalone/modules) are loaded by ModuleClassLoader.

<subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="eager">
...
<capabilities>
  <capability name="org.jboss.resteasy.resteasy-jaxrs"/>
...

and it will automatically loaded all dependencies defined in path\to\jboss\modules\org\jboss\resteasy\resteasy-jaxrs\main\module.xml which includes javax.servlet.api.

However, unless you specifically declares that as one of the capabilities
<capability name="javax.servlet.api"/>
it won't be available to OSGi.  Adding the javax.servlet.api to jboss' osgi capabilities does completely solve the classloading issue as another copy of ServletContext is available in the pax web jetty bundle.  Since both are exporting javax.servlet;version=3.0.0, we have an intermitten LinkageError problem.


Two options here:

1. do not use the resteasy in the jboss modules.  instead deploy it with the rest of the bundles so it won't load the servlet api with the module classloader.  thus, only the one loaded from pax web jetty bundle will be used.  Because resteasy in the central maven repo is not osgi ready, i'll have to wrap it in a bundle.
2.  instead of using pax web jetty bundle, use the individual pax web jetty, pax web runtime...  I did a simple test, looks like pax web runtime (2.1.2) has problem importing the javax.servlet;version=3.0.0 from jboss.

not sure which options is better, i'll update the post once i'm done with it.