Friday, March 29, 2013

Cucumber with fun

i'm a big fan of BDD and we are using cucumber-jvm on the server side. The normal BDD template is
Given something
And something else
When I do this
And I do that too
Then something good happens

And if you put this at the beginning of your feature file
# language : en-pirate 

you can do this

Gangway! something
Aye something else
Blimey! I do this
Aye I do that too
Let go an haul something good happens


The only thing is the locale en-pirate isn't really documented nicely (at least i couldn't find any documentation) and it's not that intuitive either. I'd expect it to be en_pirate as in en_US. I have to trace down the code and found the regular expression they use to match the locale to find out it should be a - instead of a _. 

Thursday, March 28, 2013

Registering a POJO as a Spring Bean

It's indeed part of the Dynamic Service Activator.  In it, we need to programmatically create a channel and register that to Spring application context so the router can route to that channel (it will look up the name of the channel from the application context.)  It's surprising easy to do that, a one liner!

Using ZooKeeper

We are using zookeeper to keep our configuration and when the configuration is updated, we'll be notified and we can update/restart the related module.  To do that, we leverage the watch feature of zookeeper.  And Yan in his blog stated very clear the things to watch for when using ZooKeeper's watch.

  1. Watches are one time triggers; if you get a watch event and you want to get notified of future changes, you must set another watch.
  2. Because watches are one time triggers and there is latency between getting the event and sending a new request to get a watch you cannot reliably see every change that happens to a node in ZooKeeper. Be prepared to handle the case where the znode changes multiple times between getting the event and setting the watch again. (You may not care, but at least realize it may happen.)
  3. A watch object, or function/context pair, will only be triggered once for a given notification. For example, if the same watch object is registered for an exists and a getData call for the same file and that file is then deleted, the watch object would only be invoked once with the deletion notification for the file.

The problem he is trying to solve here is that when the node has been deleted and recreate right away, we will lost the event for the creation of the new node as both the parent level and the child level events are triggered (node deleted) and before we can registered a new watch, the node had been recreated and the event will be forever lost.

He had also posted a solution for this.  It's pretty straightforward (it's the troubleshooting that's hard),  the watcher for the parent level only deals with 'add' event and each child watcher try to watch the "deleted" node in case it has been recreated.

if you're not familar with scalar, here's the java version

Another thing to watch for when using ZooKeeper is the ZooKeeper object from new ZooKeeper() is not readily usable.  You will get a ConnectionLossException if you try to use it before the connection is established.  To avoid that, you'll have to watch for the connection events and wait for the status KeeperState.SyncConnected (that's event.getState().ordinal() == 3 in the process method).

Wednesday, March 27, 2013

EasyMock with Spring Beans (Spring Integration)

Creating mock beans in spring is really easy.

All you have to do is to declare a bean with easymock and set the interface you want to mock as the constructor.

<bean id="somethingCool" class="org.eashmock.EasyMock" factory-method="createMock" primary="true">
   <constructor-arg value=">

and in the test, just autowired the bean and call expect & reply as usual.

This blog by DevGrok has detailed how that's done and more.

And when I'm try to mock an Spring Integration Gateway only to find that if the gateway is annotated with @Gateway, it can't be mocked.  Spring Integration somehow taken over the proxy and when I call, EasyMock.expect( someGateway.someMethod() ), it actually call Spring Integration and hung waiting for a reply message. I ended up mocking one of the service activator endpoints called by the gateway.

Monday, March 25, 2013

Spring custom namespace

The dynamic service activator that we created has a very clumsy definition.

it'd be nice if it's something like this

To do that we need to create a custom namespace for spring and we can have our bean definition under beans. I found this blog by carlo scarioni detailing how to do that.  Following it, I was able to create my namespace and put in under beans.  However, it is not added to the application context.

For that, I followed Spring's doc, all i have to do is
  1. the xsd needs to import spring beans and the element needs to extend beans:identifiedType.
  2. the bean definition parser should extends AbstractSingleBeanDefinitionParser instead.
everything else is the same.  but now you'll have id attribute in your bean definition (from beans:identifiedType)!  and it will be added to the application context.

 Another cool thing we can use is to use spring tool so you can verify the bean ref is of a certain class.  Import the following namespace and add the annotation to your attribute.

<xs:import namespace="" />

below is the gist we used to parse our custom tag. note that it set the scope to prototype at the end and it also shows you how to pass an array of beans.

Spring Integration Gateway

Gateway is very handy for communicating between modules.  However, it has a side effect that all headers not specified in the gateway will be lost to those behind the gateway.  i.e. it will only copy those headers that is listed/annotated in the method arguments. 

check GatewayMethodInboundMessageMapper.mapArgumentsToMessage.  Here it construct the message sent into the request channel and it only copy those headers that is in the argument.

But come to think of it, if endpoints behind gateway needs to use a specific header, it's not suppose to know the structure of the message, it should be specified as an input parameter to the gateway method.  It's just that we're developing both the module that calls the gateway and those behind the gateway, we kinda expect the headers will be carried into and behind the gateway. 

Sunday, March 24, 2013

JBoss OSGi doesn't export Log4j SPI

We are using spring-rabbit and others than 1.0.0.RELEASE, it imports org.apache.log4j.spi package.  However, JBoss OSGi (at least 7.1.1.Final) doesn't export that.  A workaround is to get log4j and build a wrapper just to export the spi package.

Create Apache Shell Command to check Gemini Blueprint Service

One of the annoying thing in using Eclipse Gemini is that if a dependency service is not available, it will wait for 5 mins (default) and threw timeout exception.  and even if the application context is not created (because of the timeout), the bundle's status is still active.  So I have decided to build a simple command to Apache Shell so that I can find out if there's any unsatisfied dependencies using Apache Felix WebConsole.

First we need to build a fragment bundle to gemini blueprint extender by adding this to the bundle's

Fragment-Host: org.eclipse.gemini.blueprint.extender

Next create a class extending OsgiBundleApplicationContextListener

in the onOsgiApplicationEvent, simply build a map to store all service dependency status.  Since it's collecting event data (and that it is a fragment), so the bundle have to be installed before starting the blueprint extender and other bundles that you want to capture the events.

and build another bundle which implements Apache Shell's Command and simply call the last bundle and prints out the result.

My command's name is "geministatus" (just return "geministatus" in the getName method of  your Command) so when you navigate to the shell tab of your felix webconsole, you can type geministatus and you'll find out if any bundles is waiting for any services to start the application context.

Scripting in Redis

Since Redis 2.6, it introduced scripting and is surprisingly easy.  It is using LUA 5.1 which I have never came across before but i can learn the basic syntax in an hour or so.

We are trying to create a cache that still has O(1) when retrieve entry with a certain id while maintaining the order of the entries.  Basically, we want a cache with feature of a hash and a list, so we decided to build a virtual cache where actually, it is implemented by a list cache an a hash cache.

when adding an entry to the cache, we'll have to add the id to the end of the list if it's not already in the list and put the (id,entry) pair to the hash.  In order to do that in an atomic fashion, we have to use script*.

the script look like this.

you can call any redis api using  Notice the space after do, then and end, that's there because we're concatenating the strings without a space, it'll join with the next line's first command.

To call the script using RedisTemplate (with Jedis as connection factory), you'll have to get the native connection (Jedis) and use Jedis.eval to execute the script.

also, you can test your script using redis-cli directly.  Please note that the syntax is ./redis-cli --eval <script name> <number of key> <keys>... , <argv>...
e.g. ./redis-cli --eval test.lua 1 x , 1 2
 in the script, you can do
local key = KEYS[1]
local test1 = ARGV[1]
local test2 = ARGV[2]
and the comma has to have a space before and after it.  neither x, 1 2 nor x ,1 2 will work. 

* multi/exec can't do what we wanted to do because command inputs cannot depends on result from previous commands. 

Raise condition when creating bean in OSGi managed property callback

Our application had been suffering with mysterious problem that some of the OSGi services are not registered and just by restarting the server, you'll have a 50-50 chance that it'll all start up.  It has been bothering us for a few months.  One of my team's engineer finally spotted two things.

  1. the bundle not registering the service registered some other services.
  2. application context of that bundle is not registered as an OSGi service (it can be turned off but in our environment, it's on)
  3. it hangs in applicationContext.getBean where we are trying to get a prototype bean when a new configuration is available.
and the updateConfig method has applicationContext.getBean in it.

so what happened is when indexConfigMgr is being created, it acquired a lock to create bean and call Config Admin to get configuration which also requires acquiring a lock.  Meanwhile, in another thead, the Config Admin is also trying to call the update method on indexConfigMgr which it will first acquired the lock to get configuration and inside the update method, we tried to create a new bean which tries to acquire the create bean lock.  thus, a classic dead lock.

The solution is pretty straightforward, just start a new thread in the updateConfig so the bean creation will be executed in another thread. 

Creating Spring Integration Service Activator dynamically

My project connects to multiple backend services and we have created a Connector bean for each of the backend services and they all implemented the same interface.  The way we create those Connector beans is using OSGi Config Admin, i.e. we define a configuration for each backend services and Config Admin will create an instance of Connector and registered that to OSGi service registry. 
Let's say the Connector interface has two methods,

Services getAllService()
Service getService(String id)

what we wanted to do is to set a router that would
  1. route to all available Connector for the getAllService call and have an aggregator to aggregate the results from all Connector. 
  2. route to a specific Connector for getService (assuming we can tell which Connector to use by inspecting the id)
The problem is I cannot define service activator or gateway using XML as it's configuration driven.  I've looked into Spring Integration source and determine it's too complicated to try to create a gateway or a service activator programatically.

Inspired by this post, we decided to build our own dynamic service activator.   The router will route to channels with a certain name pattern, <backend service name>.<method name>, e.g. backendA.getAllService.  Then we create a bean, DynamicServiceActivator, that will create channels with matching names and call the Connector bean.  Details about how to create a channel and send a reply is already covered in the above mentioned post, I'm not repeating here.  Since we'll need one instance of DynamicServiceActivator per Connector bean, we get a list of Connector using Gemini's list

<osgi:list interface="Connector">
  <osgi:listener bind-method="addConnector" unbind-method="removeConnector" ref="connectorMgr" />

<bean id="dynamicServiceActivator" class="DynamicServiceActivator" scope="prototype"/>

and in the addConnector method, all we have to do is applicationContext.getBean("dynamicServiceActivator") and set the Connector to the activator.

there's one little piece is missing from the above picture.  Since we are creating a channel per method, service activator is suppose to call one method, when we set the connector to the dynamic service activator, it'll have to inspect all method names and create channels for each one of them.  Two ways we can control the method to be exposed, either define in the application context xml, e.g.

<bean id="dynamicServiceActivator" class="DynamicServiceActivator" scope="prototype">
  <property name="methods">
      <entry><key>getAllService</key><value>SpEL expression</value></entry>
      <entry><key>getService</key><value>SpEL expression</value></entry>

or annotate the Connector class, so we can find the appropriate method signature to call.

Sequence Diagram

We are using UML to communicate designs and sometime you'll want to have it drawn out nicely (yes, i do need to work on my penmanship.) and i stumbled upon It's a really useful web site that we can just write all the interaction and it will generate a nice sequence diagram. It has all kinds of plugin, confluence, eclipse, intellij, maven so you can use it in your favorite editor. Then I came across PlantUML,, which the syntax for the sequence diagram is highly deived from but it supports a whole lot more!