Running a CouchDB cluster on Amazon EC2
Sunday, December 20th, 2009CouchDB is a nearly zero-configuration multi-master document oriented database. It is a awsome product build by an awsome team.
So far I have been using CouchDB like we would have used any other modern Document Datastore: in a centraized fashion. One Server at our premises. For backup purposes we replicated on a second couchdb instance running on our backup server.
Hosting about 300 GB of data a small 2.6 GHz Server with consumer-grade disks we started seeing preformance issues. Also we see latency issues since we are hosting some application at Amazon EC2 “in the cloud” which
results an an addiotional 40 ms delay for all queries to our locally hosted server.
So this is the right time to use some more of CouchDBs capabilities and spin up additional instances on demand at Amazon EC2. I assume you have already set up an Amazon EC2 account and are comfortable with the general concepts.
There are some tutorials out there which threat EC2 like a regular hosting provider. This is a seriously misguided approach. If you don’t use EC2 in a way that you always can loose one or two instances, you are using it wrong. If you are not spinning up servers in a way that it takes the same time to set up one instance than it takes to set up 10 instances you are using it wrong.
To use EC2 as it meant to be used, we need automation. We will use puppet in this example.
I assume that you have installed a “puppetmaster” on a machine called puppet.example.com. I also assume the puppet configuration on the puppetmaster is at /etc/puppet. On BSD ist might be located instead at /usr/local/etc/puppet. Place the following content at /etc/puppet/files/etc/couchdb/local.ini:
[couchdb] database_dir = /mnt/couchdb view_index_dir = /mnt/couchdb [httpd] bind_address = 0.0.0.0 [couch_httpd_auth] require_valid_user=true [admins] admin = sekrit
This ensures that only clients which authenticate as user “admin” with the password “sekrit” are allowed to access the server. You might want to change “sekrit” to something more suble.
Add /etc/puppet/fileserver.conf to make sure the local.ini file can be moved the clients:
[files]
path /etc/puppet/files
allow *
Then add /etc/puppet/manifests/site.pp to allow automatic installation and configuration:
class couchserver {
package { "couchdb": ensure => latest }
package { "python-couchdb": ensure => installed }
group { "couchdb": ensure => present }
user { "couchdb": ensure => present, groups => "couchdb",
comment => "CouchDB Administrator",
home => "/mnt/couchdb" }
file { "/etc/couchdb": ensure => directory,
owner => couchdb, group => couchdb,
mode => 755 }
file { "/mnt/couchdb": ensure => directory,
owner => couchdb, group => couchdb,
mode => 700 }
file {"local.ini":
mode => 774,
owner => couchdb, group => couchdb,
path => "/etc/couchdb/local.ini",
source =>
"puppet://puppet.exmple.com/files/etc/couchdb/local.ini"
}
service { couchdb:
ensure => running,
subscribe => [Package[couchdb],
File["local.ini"],
File["/mnt/couchdb"]]
}}
node "PLACEHOLDER" {
include couchserver
}
Now we have to create a Amazon “security group” to firewall our CouchDB servers. Since I like the belt and suspenders way of doing things we not only will use HTTP-Auth in CouchDB but also firewall rules. You have to have the EC2 commandline tools installed. I assume your comapny has a public IP range at 17.18.19.0/24.
$ ec2-add-group couchserver -d 'couchdb server'
$ ec2-authorize couchserver -P tcp -p 5984 -s 17.18.19.0/24
Next step is starting a EC2 instance. We use a Small Ubuntu 9.10 AMI since it comes with a decent version of CouchDB. We then log in and install Puppet.
$ ec2-run-instances ami-a62a01d2 --key YOUR_EC2_SSH_KEY \
--instance-type m1.small --region eu-west-1 \
--group default --group couchdb
INSTANCE i-ec985e9b ...
$ sleep 120
# get the id from the output of ec2-run-instances
$ ec2-describe-instances i-ec985e9b
INSTANCE i-ec985e9b 79.125.56.43 10.227.94.80
# get the ip from the output of ec2-describe-instances
$ ssh -i ~/.ssh/YOUR_EC2_SSH_KEY ubuntu@79.125.56.43
# on the EC2 instance:
$ sudo apt-get update -y
$ sudo apt-get install -y puppet
$ puppetd --test --server puppet.example.com
This will result in a Error message about certificates. The puppet client requested a certificate and you have to sign this certificate at the puppet server. There is still some room for automatation. Log into the puppetmaster and list the signature requests with puppetca -l. You’ see the name of your newly created instance. Sign that name by using puppetca -s:
root@puppet:~# puppetca -l
ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# puppetca -s ip-10-20-30-40.compute.internal
Signed ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# perl -npe 's//ip-10-20-30-40.compute.internal/;' \
-i.bak /etc/puppet/manifests/site.pp
The last line automatically edits /etc/puppet/manifests/site.pp to contian configuration information for the new instance. That’s all there is to do on the puppet master.
Now back on the new instance you can make puppet configure your CouchDB by typing puppetd --test --server puppet.example.com. This should install CouchDB and configure it to use the “big” 140 GB disk of your instance and to require password authentication.
You can test if CouchDB is up, running and secured by using cURL:
$ curl http://127.0.0.1:5984
{"error":"unauthorized","reason":"Authentication required."}
This is the point in time where we can start replication from our internal, behind-the-firewall CouchDB to the new box running at Amazon. Since there are some issues regarding commandline tools and authentication I created a patched version of python-couchdb at GitHub. Download it from here to a machine in your internal network, untar it and change in the couchddb-python directory. Then initiate replication:
$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
--source=http://couchdb.internal.example.com:5984 \
--target=http://admin:sekrit@79.125.56.43:5984/ --push \
--continuous
After this ran, set up permanent two-way replication between the two Servers:
$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
--source=http://couchdb.internal.example.com:5984 \
--target=http://admin:sekrit@79.125.56.43:5984/ --push
$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
--source=http://admin:sekrit@79.125.56.43:5984/ \
--target=http://couchdb.internal.example.com:5984 \
--continuous
Basicaly that’s it. We are still missing a few bit’s and pices to get full automation, but we are nearly there. and for a cluster you probably want more than one CouchDB instance running at Amazon.
To have a shiny application you need domain specific search. E.g. if our call center wants to enter a new order, they might not have the customer number ready. So they need a snappy way to get the customer number based on name, city or whatever.
some might contain a ZIP code, some may geographic coordinates attached, some relate to a certain packet or pallet (called “movable unit” in huLOG-spreak) some relate to a certain shipment (which is a group of movable units). Some have file attachments, e.g. Images of the packages, signatures proofing delivery or pictures of the number plates of the trucks.
CouchDB 0.9 came around and it started to look usable for serious use. We hired 
If I import the same file with events twice, the events will have the same IDs during both imports and thus the second import will overwrite the first one: exactly what I want.
After URL-encoding we end up with 
Now we can query the view and with the startkey and endkey parameters. Here we get again into trouble for choosing the endkey. The hack we used for the IDs did work because there was a timestamp of well known format in the ID so we construct a “slightly bigger” value. Shipments are numeric. So what is bigger than 128996? 128997 is obviously bigger but would get us documents for two shipments (128996 and 128997). Since the shipments are numeric, a trick like “128996Z” does not work out.
