teenage mutant ninja hero coders

CouchDB is a nearly zero-configuration multi-master document oriented database. It is a awsome product build by an awsome team.

So far I have been using CouchDB like we would have used any other modern Document Datastore: in a centraized fashion. One Server at our premises. For backup purposes we replicated on a second couchdb instance running on our backup server.

Hosting about 300 GB of data a small 2.6 GHz Server with consumer-grade disks we started seeing preformance issues. Also we see latency issues since we are hosting some application at Amazon EC2 “in the cloud” which
results an an addiotional 40 ms delay for all queries to our locally hosted server.

So this is the right time to use some more of CouchDBs capabilities and spin up additional instances on demand at Amazon EC2. I assume you have already set up an Amazon EC2 account and are comfortable with the general concepts.

There are some tutorials out there which threat EC2 like a regular hosting provider. This is a seriously misguided approach. If you don’t use EC2 in a way that you always can loose one or two instances, you are using it wrong. If you are not spinning up servers in a way that it takes the same time to set up one instance than it takes to set up 10 instances you are using it wrong.

To use EC2 as it meant to be used, we need automation. We will use puppet in this example.

I assume that you have installed a “puppetmaster” on a machine called puppet.example.com. I also assume the puppet configuration on the puppetmaster is at /etc/puppet. On BSD ist might be located instead at /usr/local/etc/puppet. Place the following content at /etc/puppet/files/etc/couchdb/local.ini:

[couchdb]
database_dir = /mnt/couchdb
view_index_dir = /mnt/couchdb

[httpd]
bind_address = 0.0.0.0

[couch_httpd_auth]
require_valid_user=true

[admins]
admin = sekrit

This ensures that only clients which authenticate as user “admin” with the password “sekrit” are allowed to access the server. You might want to change “sekrit” to something more suble.

Add /etc/puppet/fileserver.conf to make sure the local.ini file can be moved the clients:

[files]
  path /etc/puppet/files
  allow *

Then add /etc/puppet/manifests/site.pp to allow automatic installation and configuration:

class couchserver {
  package { "couchdb": ensure => latest }
  package { "python-couchdb": ensure => installed }
  group { "couchdb": ensure => present }
  user { "couchdb": ensure => present, groups => "couchdb",
    comment => "CouchDB Administrator",
    home => "/mnt/couchdb" }
  file { "/etc/couchdb": ensure => directory,
    owner  => couchdb, group  => couchdb,
    mode   => 755 }
  file { "/mnt/couchdb": ensure => directory,
    owner  => couchdb, group  => couchdb,
    mode   => 700 }
  file {"local.ini":
    mode => 774,
    owner => couchdb, group => couchdb,
    path => "/etc/couchdb/local.ini",
    source =>
      "puppet://puppet.exmple.com/files/etc/couchdb/local.ini"
  }
  service { couchdb:
    ensure    => running,
    subscribe => [Package[couchdb],
                  File["local.ini"],
                  File["/mnt/couchdb"]]
}}

node "PLACEHOLDER" {
    include couchserver
}

Now we have to create a Amazon “security group” to firewall our CouchDB servers. Since I like the belt and suspenders way of doing things we not only will use HTTP-Auth in CouchDB but also firewall rules. You have to have the EC2 commandline tools installed. I assume your comapny has a public IP range at 17.18.19.0/24.

$ ec2-add-group couchserver -d 'couchdb server'
$ ec2-authorize couchserver -P tcp -p 5984 -s 17.18.19.0/24

Next step is starting a EC2 instance. We use a Small Ubuntu 9.10 AMI since it comes with a decent version of CouchDB. We then log in and install Puppet.

$ ec2-run-instances ami-a62a01d2 --key YOUR_EC2_SSH_KEY \
  --instance-type m1.small --region eu-west-1 \
  --group default --group couchdb
INSTANCE       i-ec985e9b   ...
$ sleep 120
# get the id from the output of ec2-run-instances
$ ec2-describe-instances i-ec985e9b
INSTANCE       i-ec985e9b      79.125.56.43   10.227.94.80
# get the ip from the output of ec2-describe-instances
$ ssh -i ~/.ssh/YOUR_EC2_SSH_KEY ubuntu@79.125.56.43
# on the EC2 instance:
$ sudo apt-get update -y
$ sudo apt-get install -y puppet
$ puppetd --test --server puppet.example.com

This will result in a Error message about certificates. The puppet client requested a certificate and you have to sign this certificate at the puppet server. There is still some room for automatation. Log into the puppetmaster and list the signature requests with puppetca -l. You’ see the name of your newly created instance. Sign that name by using puppetca -s:

root@puppet:~# puppetca -l
ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# puppetca -s ip-10-20-30-40.compute.internal
Signed ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# perl -npe 's//ip-10-20-30-40.compute.internal/;' \
   -i.bak /etc/puppet/manifests/site.pp

The last line automatically edits /etc/puppet/manifests/site.pp to contian configuration information for the new instance. That’s all there is to do on the puppet master.

Now back on the new instance you can make puppet configure your CouchDB by typing puppetd --test --server puppet.example.com. This should install CouchDB and configure it to use the “big” 140 GB disk of your instance and to require password authentication.

You can test if CouchDB is up, running and secured by using cURL:

$ curl http://127.0.0.1:5984
{"error":"unauthorized","reason":"Authentication required."}

This is the point in time where we can start replication from our internal, behind-the-firewall CouchDB to the new box running at Amazon. Since there are some issues regarding commandline tools and authentication I created a patched version of python-couchdb at GitHub. Download it from here to a machine in your internal network, untar it and change in the couchddb-python directory. Then initiate replication:

$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://couchdb.internal.example.com:5984 \
  --target=http://admin:sekrit@79.125.56.43:5984/ --push \
  --continuous

After this ran, set up permanent two-way replication between the two Servers:

$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://couchdb.internal.example.com:5984 \
  --target=http://admin:sekrit@79.125.56.43:5984/ --push
$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://admin:sekrit@79.125.56.43:5984/ \
  --target=http://couchdb.internal.example.com:5984 \
  --continuous

Basicaly that’s it. We are still missing a few bit’s and pices to get full automation, but we are nearly there. and for a cluster you probably want more than one CouchDB instance running at Amazon.

This entry was posted on Sunday, December 20th, 2009 at 22:38 and is filed under couchdb. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.