Sunday, March 24, 2013

Raise condition when creating bean in OSGi managed property callback

Our application had been suffering with mysterious problem that some of the OSGi services are not registered and just by restarting the server, you'll have a 50-50 chance that it'll all start up.  It has been bothering us for a few months.  One of my team's engineer finally spotted two things.

  1. the bundle not registering the service registered some other services.
  2. application context of that bundle is not registered as an OSGi service (it can be turned off but in our environment, it's on)
  3. it hangs in applicationContext.getBean where we are trying to get a prototype bean when a new configuration is available.
and the updateConfig method has applicationContext.getBean in it.

so what happened is when indexConfigMgr is being created, it acquired a lock to create bean and call Config Admin to get configuration which also requires acquiring a lock.  Meanwhile, in another thead, the Config Admin is also trying to call the update method on indexConfigMgr which it will first acquired the lock to get configuration and inside the update method, we tried to create a new bean which tries to acquire the create bean lock.  thus, a classic dead lock.

The solution is pretty straightforward, just start a new thread in the updateConfig so the bean creation will be executed in another thread.