Linux failover server in the old days

I received an email from Livejournal reminding me of my 18 anniversary. Apart from another reminder of my age :-). I had even forgotten that Livejournal, a popular blogging platform in those days, still existed.

Most of the posts were about Perl and my work, but the last blog post on how to failover an oracle database brought a smile. It was one of the projects where I had a lot of fun and had to dig into Linux and networking pretty deep and had to figure out a how oracle database was working without any documentation to make this work. This failover system was used in a few US states then.

I joined EF Johnson Technologies (EFJ), which provided land mobile radio services (the radio used by police, military …) to clients. Some of our clients were Wisconsin, Hawaii, and some stations of the US military that used EFJ radio equipment.

I had worked on Perl and had done my fair share of sysadmin, but this project was different.

A project I worked on was to create a failover of the network management system, which ran Perl on Apache webserver and an oracle database. The requirement was that there were two Linux servers located in different locations. Each server had two ethernet connections, and each port was connected to a different router so that if one line goes down, there is a connection from the other line.

We were able to achieve failover using the following open source tools

Heartbeat — for communication between servers.

Quagga routing to make each server act as a router so that a floating IP is visible to both networks using the OSPF protocol

DRBD : sync writes between two servers

custom linux install script to create lvm partitions like the way we wanted.

modifying routers to configure OSPF and our own custom algorithm to solve split-brain situations and create scripts to install Linux and oracle.