jump to navigation

The Perils of a clustered environment July 30, 2007

Posted by Phill in General J2EE.

Firstly, I apologise for not posting up here much recently! It got quite busy last week, hence the lack of updates. Still, it did highlight one issue for me: developing applications on a single app server which are going to be deployed on multiple app servers.

We developed an application for a client which was going to be run on multiple application servers (four app servers sharing the same database). It’s not in a cluster or anything complicated like that – the setup is basically just a load balancer in front of four identical application servers, each running a copy of our application. Nothing complex!

Unfortunately,  we only tested the application on one application server.  When it was deployed on  two different application servers, we began to experience a problem: our application needs to connect to another third-party application, and to do so it needs an authentication token. What was happening was one app server authenticated, got the auth token, and then stored it in memory until it expired (in this case, 24 hours). Unfortunately, this meant that until the other server wasn’t able to authenticate against it (because was still using an old authentication token), meaning that no-one was able to use the application on that server until it re-requested the authentication token. And once it did, the other application server would then not be able to authenticate … etc, <i>ad infinitum</i>.

The solution was to store the authentication key in the database, and all of the application instances would share the same authentication token. Whenever one application requested a new auth token, all applications would use it.

I think the main thing I’m going to remember about this particular problem is (and I’m sure that other people have pointed this out at great length): running an application on multiple servers will introduce levels of complexity which should be borne in mind from the outset! In this particular case it was a fairly small fix, but next time it might not be so easy.

Here endeth the lesson…

%d bloggers like this: