Our homemade trick for reconnecting Couchbase with Node.js

Couchbase is a distributed NoSQL database, where you can store your data with JSON documents and access these documents with view query or N1ql search. Index, combine, and transform your documents with JavaScript. It’s like Couchdb’s database, but Couchbase provides the distribution feathers and it’s really easy to replicate on multiple servers.

Couchbase Easy Integration

Couchbase provides lots of software development kits (SDKs) in different programming languages, including one in Node.js, so JavaScript development integrates Couchbase into the product very easily. For example:

const couchbase = require('couchbase');
const cluster = new couchbase.Cluster('couchbase://127.0.01');
const bucket = cluster.openBucket('example-bucket');
bucket.get('lorem', (err, res) => {
  // You will get you data here
});

It’s easy to use, but some reasons, like a lack of internet, kill our Couchbase server. Your app doesn’t notice that because Couchbase’s Node.js SDK doesn’t provide a good method to receive things like disconnect events. You never know if your database server is down until your users start to recieve messages like `cannot perform operations on a shutdown bucket｀. Since Couchbase doesn’t provide any solutions, what do we do?

Our solution

Preparation

We wrap Couchbase into a single instance, which means we only keep one cluster instance in memory. We write a simple couchbase.js file like this:

let cluster = null;
function couchbase(options) {
  if (cluster) return cluster;
  return new CouchCluster(options);
}
class CouchCluster extends EventEmitter {
  constructor(cnstr, options) {
    super();
    this.buckets = {};
    this.cluster = new Couchbase.Cluster(cnstr, options);
  }
}

And we implement some simple functions that couchbase original cluster had.

 openBucket(name, pass) {
    assert(name);
    if (this.buckets[name]) return this.buckets[name];

    return this._openBuket.apply(this, arguments);
  }
 _openBuket(name, pass) {
    debug('connecting to bucket %s', name);
    this.buckets[name] = this.cluster.openBucket(name, pass);
    this.buckets[name].on('connect', () => {
      debug('connected to bucket %s', name);
      this.emit('connect');
    });
    this.buckets[name].on('error', this._onError.bind(this, name));
    return this.buckets[name];
  }
 reconnectAll() {
    for (let i in this.buckets) {
      debug('reconnect bucket %s', i);
      this.reconnect(i);
    }
  }
  reconnect(name) {
    this.buckets[name].connected = true;
  }

The openBucket function will recall the original one, but it will be cached in memory and every time you call this function you’ll get the same bucket instance. And we also have a reconnectAll function that allows us to reconnect all the buckets we have in our memory. So how do we solve the reconnect problem?

Solutions

There are a couple solutions: a) catching the error and reconnecting the server and b) pinging the database server on regular intervals. Let’s give them a look.

• Catch errors and reconnect the server

Our app is built with an express framework, so we have a global error handler that will catch all errors thrown in each middleware. When users call up our API, but the connection between our app and the Couchbase server is broken, Couchbase’s SDK will throw a cannot perform operations on a shutdown bucket error. Our error handler will catch the error and reconnect all buckets’ connections automatically.

For example:

 app.use((err, req, res, next) => {
    // Handle cannot perform operations on a shutdown bucket.
    // Handler Couchbase Error code 16
    // Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES,
    // or via `detailed_errcodes` in the connection string) and/or enable logging to get more information
    if (err) {
      if (err.message === 'cannot perform operations on a shutdown bucket' ||
        (err instanceof couchbase.Error && err.code === 16)) {
        cluster.reconnectAll();
      }
    }
    next(err);
  });

This solution is not perfect, because the app doesn’t know the connection broke. It’s noticed when the user sees an error message. It’s not very friendly to our users, but it costs less in resources.

• Ping the database server on an interval

We build a loop to ping the Couchbase server in like 10 seconds, but use the existing bucket connections so we don’t need to create a new connection every time. It goes like this:

let sti = setInterval(() => {
   bucket.get('lorem', (err, res) => {
     if (err.message === 'cannot perform operations on a shutdown bucket' ||
        (err instanceof couchbase.Error && err.code === 16)) {
        cluster.reconnectAll();
      }
     clearInterval(sti);
  });
});

This solution makes more sense than the first one. The user won’t feel any difference and our app will self-check and reconnect automatically, so this solution is good but still cost a lots of CPU and memory when you have several buckets. So it’s still not the best.

• What we choose

We at Wiredcraft use the first solution proposed but we have a separate service that will have a heart rate call to our app about every 5 seconds to check the health of our services. Our combined approach, using the health check service and our app’s express framework, so it’s just like the second solution but costs much less memory.

Conclusion

The reconnection between the service and database is necessary. For some reason, database server downtime doesn’t alert your app, which then ticks off your users. Couchbase’s SDK for Node.js doesn’t provide a very good way to handle this problem, so we needed figure this mess out by ourselves.

Keep any eye out! This little tool for reconnecting will be open sourced very soon.

Wanna see more? Check out these links for reference:

• http://www.couchbase.com/nosql-databases/couchbase-server
• http://developer.couchbase.com/documentation/server/4.1/sdks/node-2.0/introduction.html
• https://github.com/xeodou/couchbase-reconnect

Let us know what you think about our solutions for reconnecting Couchbase on Twitter or by shooting us an email.