They are terrific for building things and while it is an urban legend that “it’s cheaper to scrap them then to send them back to Asia” (hint: the ships have to drive back to Asia anyway) new containers are produced in China or Korea and used containers are cheap and easy to get in Europe and the US.

Things you can’t do with shipping Containers

All these great features come with a bunch of restrictions. Containers come with 8 Corner Casings which are the only points where a Shipping Container is meant to be anchored. This makes a container can only rest on it’s 4 bottom Corners and can only support loads from above on it’s 4 top corners.

Everything else on a container is not meant to carry any significant load.

This meant you can only stack containers in configurations where all for bottom corner casings of the upper container are supported by four top corner casings of lower containers.

This results in many desirable stacking configurations being impossible from a stability standpoint. See for examples on the right.

Nearly all configurations where the upper container is rotated by 90 degrees are impossible. Putting 20″ on 40″ containers is in most cases impossible.

Putting a 40″ on a single 20″ container is impossible. Most overhang configurations are impossible.

Now you know the reason why there is so little creativity in professional Container Stacking.

What works is putting a 40″ Container on two 20″ containers, but there is little reason to do that.

The Roof Problem

While the floors of a Container are meant to carry loads from more than a ton per square meter the roofs are little more than weather protection than a few dozen kg per square meter. See this video of me seesawing my toes and the effect on the roof.

Walking on the Roof for a single person is slightly uncomfortable and for a group it is something in-between scary and dangerous. The high quality steel sheeting of the roof will not break but the whole thing is very shaky.

Also the roofs of pre-owned containers are usually not flat anymore. Being the most fragile part of the structure they have a tendency to bent and get into a wobbly state.

So if you plan to walk regularly on them you need to put additional flooring on top of the containers to distribute the load and stiffen the floor. We used two layers of 1 cm plywood which is fine, but still the walking experience is not totally solid.

If you plan to put heavier loads on the roofs you probably need to discuss this with a structural engineer. An approach we where contemplating was to add an additional layer of containers on top and remove roof and walls from them to keep only the ultra rigid container floor. Puma City this this for their balcony

Steel on Steel

You might or might not be aware, that steel on steel usually results in an excellent plate bearing. Meaning that Corner Casings on top of each other result in astonishing easy movability of the upper container. As in “Slam the door and it falls nearly down”. Even a pice of carton between the corner casings helps somewhat. But on the long run you need to fix your containers onto each other.

So it’t imperative that you fix the upper container to the lower containers. The best way is to use Twistlocks as the Pros do. Marine grade Twistlocks can secure a stack of 4 fully loaded containers on a rolling ship without additional help. (Usually contianers are stacked 6-8 levels high, which results in the need for additional securing. We bought our Twistlocks at Willi Wader Group.

You might be tempted to save the expense of twistlocks (15-30 € per lock) and instead screw or weld the containers together. But keep in might that the twistlocks are mass produced to rigid quality standards – while it is hard to assess the integrity of your home grown connection approach.

Doors

Doors of shipping containers are hard to open by design. Even on a brand new container you have to operate two handles with considerable strength at once and at different speeds to open and close a container. And you can not open the left dor unless the right door is already open (although you might bee able to change that with a welding touch, a knife and in 20 minutes).

On a pre-owned container being 10 years or so old it is usually much harder to operate the doors because the frame of the door and the container itself is always bent to a certain degree. These doors can not replace a normal door if you are planning to pass it several times a day.

If you receive more than one container sort them by door quality. If you are able to choose the containers before purchase, always check the doors.

Structural integrity

Shipping Containers are unibody constructions. This means the whole hull is used to contribute to load distribution.

In case you didn’t suspect: this means bad things™ happen, if you cut substantial holes in the hull.

If you spot something with big openings it is no shipping container. Or it is a shipping container which has extensively upgraded structural components – which is possible but defeats the purpose of “standardized, cheap, reusable”.

Even if you leave the door open the container is less sturdy than one wit a closed door. Most containers seem to be able to withstand removing the back wall opposite to the door I wouldn’t like to have any load on such a container.

We opted for leaving about 30 cm on the top and about 15 cm at the sides to keep some reinforcement. But in our application we are quite sure that there are only minimal shear forces. I would not feel comfortable in removing the back walls in any container which would have to carry substantial load.

If you remove one of the longer walls your container is toast. Or at least it has as much integrity as toast. We removed a complete wall of a container and found out that the container lost all stability.

While all containers conform to rigid standards (ocean carriers hate delays due to broken containers) not all containers are created equal. Some are a little less unibody by having additional reinforcement along the top. Use this type for cutting holes in the sides – they even can keep a somewhat stable roof when a whole wall is removed.

What feels like a relative comfortable solution is removing about 20% of the side wall at the right and at the left end while leaving the upper 30 cm of the wall intact. The result is a still very stable container.

Where to get containers and what to get

You can get containers at shipping companies and ad specialized container providers. The only ISO containers we where able to get at decent prices where 20″ 40″ and 40″ “high cube” containers. Keep in mind that the “modular space” container type is very different from shipping containers and usualy much less sturdy.

We bought containers still sea-worthy and with next CSC inspection scheduled for 2011 or 2012. The Containers where commissioned 1994-1997 and considered “C” quality (on a ABC scale) by the shipping company. Because of the doors and possible holes (in C quality containers or in containers not sea worthy) you should inspect the containers before buying if possible.

The floors differ much between containers and usually are contaminated by pesticides, plan to remove them if humans will spend serious time inside. Some of our containers where freshly painted on the inside and it took several weeks of venting to get the smell to a bearable level.

How to move the containers

Usually the party selling you the containers has the equipment for hauling them to your side. The first issue is getting them of the tuck and the second issue is getting them where you want them.

We rented a 8 ton fork lift for all the handling which was adequate but not over powered for the task. When carrying two containers at once (nominal weight 5 tons) the machine had considerable effort. While considering my self an seasoned fork lift driver (30 years of experience) I had to learn that a huge forklift with a huge container is something very different than the kind of forklift you drive in warehouses. A Container has lot’s of momentum and you see absolutely nothing in the direction you are driving at. For a 20″ container assume every corner with less than 8 m space requires artistic driving capabilities. We had to get around a 6.5 m corner and through a 4 m door. We handled that by carrying the containers between the big 8 t forklift and one of our own 1.5 t warehouse-forklifts.

When renting a forklift be sure to get one which can spread it’s forks wide enough to fit into the carrying lashes at the belly of a container. They are 2.5 m or so apart.

Big forklifts can only operate on relatively even ground. Alternatively you can use a crane. The main problem with a crane and especially 40″ containers is that you can not introduce big horizontal forces by the attachment cables running from all for corners to the hook. Since you probably don’t have a spreader available to distribute the load you either need a very tall crane or two cranes. Your crane operator probably can tell you more.

Placement

Containers are build to stand on their four lower corner casings. If you don’t have a flat sturdy surface to put them on you should build a foundation at the four corners. Directly insert twistlocks into the foundation during pouring for perfect stability.

Containers are not insulated against heat, cold, noise and vibrations. If you insulate them on the outside you loose the feature of the weatherproof shell and if you insulate them on the inside your rooms get very small. It’s also hard to make the connection between two containers waterproof. Therefore it seems very popular to put the containers into another building to provide heating and shelter.

Further Reading

More information on the real world issues of container construction can be found at 10 Things to Consider in Using Shipping Containers for Your Next Project. Preston Koerner has good points although we where able to get the containers much cheaper than his estimate of 2000-4000 U$. We payed less than 1000 € per 20″ (including delivery).

So far I have been using CouchDB like we would have used any other modern Document Datastore: in a centraized fashion. One Server at our premises. For backup purposes we replicated on a second couchdb instance running on our backup server.

Hosting about 300 GB of data a small 2.6 GHz Server with consumer-grade disks we started seeing preformance issues. Also we see latency issues since we are hosting some application at Amazon EC2 “in the cloud” which
results an an addiotional 40 ms delay for all queries to our locally hosted server.

So this is the right time to use some more of CouchDBs capabilities and spin up additional instances on demand at Amazon EC2. I assume you have already set up an Amazon EC2 account and are comfortable with the general concepts.

There are some tutorials out there which threat EC2 like a regular hosting provider. This is a seriously misguided approach. If you don’t use EC2 in a way that you always can loose one or two instances, you are using it wrong. If you are not spinning up servers in a way that it takes the same time to set up one instance than it takes to set up 10 instances you are using it wrong.

To use EC2 as it meant to be used, we need automation. We will use puppet in this example.

I assume that you have installed a “puppetmaster” on a machine called puppet.example.com. I also assume the puppet configuration on the puppetmaster is at /etc/puppet. On BSD ist might be located instead at /usr/local/etc/puppet. Place the following content at /etc/puppet/files/etc/couchdb/local.ini:

[couchdb]
database_dir = /mnt/couchdb
view_index_dir = /mnt/couchdb

[httpd]
bind_address = 0.0.0.0

[couch_httpd_auth]
require_valid_user=true

[admins]
admin = sekrit

This ensures that only clients which authenticate as user “admin” with the password “sekrit” are allowed to access the server. You might want to change “sekrit” to something more suble.

Add /etc/puppet/fileserver.conf to make sure the local.ini file can be moved the clients:

[files]
  path /etc/puppet/files
  allow *

Then add /etc/puppet/manifests/site.pp to allow automatic installation and configuration:

class couchserver {
  package { "couchdb": ensure => latest }
  package { "python-couchdb": ensure => installed }
  group { "couchdb": ensure => present }
  user { "couchdb": ensure => present, groups => "couchdb",
    comment => "CouchDB Administrator",
    home => "/mnt/couchdb" }
  file { "/etc/couchdb": ensure => directory,
    owner  => couchdb, group  => couchdb,
    mode   => 755 }
  file { "/mnt/couchdb": ensure => directory,
    owner  => couchdb, group  => couchdb,
    mode   => 700 }
  file {"local.ini":
    mode => 774,
    owner => couchdb, group => couchdb,
    path => "/etc/couchdb/local.ini",
    source =>
      "puppet://puppet.exmple.com/files/etc/couchdb/local.ini"
  }
  service { couchdb:
    ensure    => running,
    subscribe => [Package[couchdb],
                  File["local.ini"],
                  File["/mnt/couchdb"]]
}}

node "PLACEHOLDER" {
    include couchserver
}

Now we have to create a Amazon “security group” to firewall our CouchDB servers. Since I like the belt and suspenders way of doing things we not only will use HTTP-Auth in CouchDB but also firewall rules. You have to have the EC2 commandline tools installed. I assume your comapny has a public IP range at 17.18.19.0/24.

$ ec2-add-group couchserver -d 'couchdb server'
$ ec2-authorize couchserver -P tcp -p 5984 -s 17.18.19.0/24

Next step is starting a EC2 instance. We use a Small Ubuntu 9.10 AMI since it comes with a decent version of CouchDB. We then log in and install Puppet.

$ ec2-run-instances ami-a62a01d2 --key YOUR_EC2_SSH_KEY \
  --instance-type m1.small --region eu-west-1 \
  --group default --group couchdb
INSTANCE       i-ec985e9b   ...
$ sleep 120
# get the id from the output of ec2-run-instances
$ ec2-describe-instances i-ec985e9b
INSTANCE       i-ec985e9b      79.125.56.43   10.227.94.80
# get the ip from the output of ec2-describe-instances
$ ssh -i ~/.ssh/YOUR_EC2_SSH_KEY ubuntu@79.125.56.43
# on the EC2 instance:
$ sudo apt-get update -y
$ sudo apt-get install -y puppet
$ puppetd --test --server puppet.example.com

This will result in a Error message about certificates. The puppet client requested a certificate and you have to sign this certificate at the puppet server. There is still some room for automatation. Log into the puppetmaster and list the signature requests with puppetca -l. You’ see the name of your newly created instance. Sign that name by using puppetca -s:

root@puppet:~# puppetca -l
ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# puppetca -s ip-10-20-30-40.compute.internal
Signed ip-10-20-30-40.eu-west-1.compute.internal
root@puppet:~# perl -npe 's//ip-10-20-30-40.compute.internal/;' \
   -i.bak /etc/puppet/manifests/site.pp

The last line automatically edits /etc/puppet/manifests/site.pp to contian configuration information for the new instance. That’s all there is to do on the puppet master.

Now back on the new instance you can make puppet configure your CouchDB by typing puppetd --test --server puppet.example.com. This should install CouchDB and configure it to use the “big” 140 GB disk of your instance and to require password authentication.

You can test if CouchDB is up, running and secured by using cURL:

$ curl http://127.0.0.1:5984
{"error":"unauthorized","reason":"Authentication required."}

This is the point in time where we can start replication from our internal, behind-the-firewall CouchDB to the new box running at Amazon. Since there are some issues regarding commandline tools and authentication I created a patched version of python-couchdb at GitHub. Download it from here to a machine in your internal network, untar it and change in the couchddb-python directory. Then initiate replication:

$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://couchdb.internal.example.com:5984 \
  --target=http://admin:sekrit@79.125.56.43:5984/ --push \
  --continuous

After this ran, set up permanent two-way replication between the two Servers:

$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://couchdb.internal.example.com:5984 \
  --target=http://admin:sekrit@79.125.56.43:5984/ --push
$ PYTHONPATH=. python ./couchdb/tools/manual_replication.py \
  --source=http://admin:sekrit@79.125.56.43:5984/ \
  --target=http://couchdb.internal.example.com:5984 \
  --continuous

Basicaly that’s it. We are still missing a few bit’s and pices to get full automation, but we are nearly there. and for a cluster you probably want more than one CouchDB instance running at Amazon.

For Django there are similar Applications like django-db-log but to my Knowledge there is no hosted solution for Django. Hosted solutions are nice, because some types of errors might prevent you from saving information in the local database.

But there is django-hoptoad which brings basic hoptoad functionality to Django.

To use it, got to a href=”http://hoptoadapp.com”>hoptoadapp.com, sign up and create a Project. Go to “Edit Project and tou should see something like “Current API key: 3c81d132d2f28749eab2043bb4c987a5″.
There seems an other type of “auth_token” which is not per project but per user. Don’t use this.

If you use the Lighthouse Bugtracker, you can add it to your Hotoad configuration at this point to integrate the two Applications.

Now download django-hoptoad form it’s homepage and install it on your server.

Then add something like this to your settings.py:

HOPTOAD_NOTIFY_WHILE_DEBUG = True
HOPTOAD_API_KEY = '6c2...b3'
HOPTOAD_NOTIFY_404 = True
HOPTOAD_NOTIFY_403 = True
HOPTOAD_IGNORE_AGENTS = ['Googlebot', 'Yahoo! Slurp']
MIDDLEWARE_CLASSES = list(MIDDLEWARE_CLASSES).append( \
    'hoptoad.middleware.HoptoadNotifierMiddleware')

That’s all!

You now can see your issues aggregated by cause, create Lighthouse-Tickets from them and have peace of mind that your exceptions aren’t lost, even when your servers hard disks are full.

I’ll setup a VPN between a BSD router with a static IP Address and the Fritz!Box with a dynamically changing IP. Sometimes this is called a “road warrior” setup.

Get a Dyndns.org name for your Fritzbox and configure it.

First you need a configuration file for the Fritzbox. Replace A.B.C.D with the IP of your gateway. Also replace the key with something more secret. phase2localid needs to describe the local net of the Fritzbox. accesslist needs to be the remote (BSD) network. Put your Dyndns Name into localid { fqdn }.

/*
 * C:\fritzbox_kame.cfg
 * Thu Sep 24 23:36:34 CEST 2009
 */
vpncfg {
        connections {
                enabled = yes;
                conn_type = conntype_lan;
                name = "BSD";
                always_renew = no;
                reject_not_encrypted = no;
                dont_filter_netbios = yes;
                localip = 0.0.0.0;
                local_virtualip = 0.0.0.0;
                remoteip = A.B.C.D;
                remote_virtualip = 0.0.0.0;
                localid {
                        fqdn = "example.ath.cx";
                }
                remoteid {
                        fqdn = "A.B.C.D";
                }
                mode = phase1_mode_aggressive;
                phase1ss = "all/all/all";
                keytype = connkeytype_pre_shared;
                key = "sekritt";
                cert_do_server_auth = no;
                use_nat_t = no;
                use_xauth = no;
                use_cfgmode = no;
                phase2localid {
                        ipnet {
                                ipaddr = 172.30.20.0;
                                mask = 255.255.255.0;
                        }
                }
                phase2ss = "esp-3des-sha/ah-no/comp-no/pfs";
                accesslist = "permit ip any 192.168.0.0 255.255.0.0";
        }
        ike_forward_rules = "udp 0.0.0.0:500 0.0.0.0:500",
                            "udp 0.0.0.0:4500 0.0.0.0:4500";
}

Now you have to set up your Unix Box. I used FreeBSD but to my understanding it’s the same with OpenBSD and probably also with some Linux Variants and NetBSD. You need to install ipsec-tools and racoon. They might come in two packages or in one or might be already installed. On my FreeBSD box I added something like this in /etc/rc.conf:

racoon_enable="YES"
racoon_flags="-l /var/log/racoon.log"
racoon_create_dirs="YES"

Next thing is to save your IPsec “shared secret” (Password) somewhere:

echo "example.ath.cx sekritt" > /usr/local/etc/racoon/psk.txt

The last Part missing is /usr/local/etc/racoon/racoon.conf:

# racoon.conf
path pre_shared_key "/usr/local/etc/racoon/psk.txt" ;
log info; # notify info debug;
padding {
        maximum_length 20;
        randomize off;
        strict_check off;
        exclusive_tail off;
      }
listen { isakmp A.B.C.D [500]; # add your public IP here }
timer {
        counter 5;
        interval 20 sec;
        persend 1;
        phase1 30 sec;
        phase2 15 sec;
      }
remote anonymous { # we don't know the peers IP during phase 1
        exchange_mode main, aggressive;
        nonce_size 16;
        lifetime time 140 min;   # sec,min,hour
        initial_contact on;
        proposal_check obey;    # obey, strict or claim
        support_proxy on;
        ike_frag on;
        weak_phase1_check on;
        # important for automatically configuring
        # the Security Policy Database (SPD)
        generate_policy on;
        passive on;
        # Fritz!box
        proposal {
                encryption_algorithm aes;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group 2 ;
        }
      }
 # local net - Fritz net
sainfo address 192.168.0.0/16 any address 172.30.20.0/24 any {
	pfs_group 2;
	lifetime time 8 hour;
	encryption_algorithm aes, 3des, des;
	authentication_algorithm hmac_sha256, hmac_sha1, hmac_md5 ;
	compression_algorithm deflate;
}

If you use pf for Packet Filtering you need something like this in /etc/pf.conf:

pass on $ext_if proto udp from any port 500 to A.B.C.D port 500 keep state
pass on $ext_if proto udp from A.B.C.D port 500 to any keep state
pass quick on $ext_if proto { esp ah ipencap } from any to A.B.C.D
pass quick on $ext_if proto { esp ah ipencap } from A.B.C.D to any

This Configuration still has issues but works.

Unfortunately since 6.3 or so, FreeBSD comes with the hptrr binary blob driver from Highpoint. The hptrr driver breaks detection of the SuperMicro AOC-SAT2-MV8 controllers (which are handled by the ata driver).

Unfortunately the hptrr is compiled in into recent GENERIC cernels and can not be disabled. To disable it you have to edit the kernel configuration and comment out the “device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx” line and rebuild the kernel.

You also have to add the line ‘hptrr_load=”NO”‘ to /boot/loader.conf.

Very annoying.

We did experiment with lot’s of LIKE queries to our In our legacy ERP database system. This didn’t feel good, had some SQL injection vulnerabilities and required lot’s of full table scans.

Looking for alternatives we decided to use CouchDB. There is some work on full text indexing for CouchDB you can build something much more simple yourself.

Once a day we copy all customer data from the legacy system into CouchDB. Then we use the map function to emit a line for each word in each data field of each document. It looks like this:

function(doc) {
    function output(value) {
        // Split into search terms
        if(value && (value != "-") && (value.length > 2)) {
            emit(value, 1);
            for(var word in value.split(" ")) {
                if(word && (word != "-") && (word.length > 2)) {
                    emit(word, 1);
                }
            }
        }
    }
    output(doc.kundennr);
    output(doc.name1);
    output(doc.name2);
    output(doc.ort);
    output(doc.land + "-" + doc.plz);
}

This basically generates a view (index) containing every word and the document it occurs in.

You now can use that for a prefix based search in a function like this:

from couchdb.client import *

def finde_kundendaten(searchstring):
    server = Server('http://couchdb.local.hudora.biz:5984/')
    db = server['kunden']
    rows = []
    while len(rows) < 1 and len(searchstring) > 2:
        rows = db.view('suche/alle_felder', startkey=searchstring, limit=25)
        if rows:
            break
        searchsting = searchstring[:-1]
    return [(x.id, x.key) for x in rows]

>>> finde_kundendaten("Sport Dornseif") # no exact match in the DB
[(u'51320', u'Sport Alm SysIntersport'),
 (u'27094', u'Sport Freizeit'),
 (u'31071', u'SPORT FREIZEIT  TREFF'),
...]

Nifty!

While writing something about using CouchDB I came across the issue of “slice indexes” (called startkey and endkey in CouchDB lingo).

I found no exact definition of startkey and endkey anywhere in the documentation. Testing reveals that access on _all_docs and on views documents are retuned in the interval

[startkey, endkey] = (startkey <= k <= endkey).

I don’t know if this was a conscious design decision. But I like to promote a slightly different interpretation (and thus API change):

[startkey, endkey[ = (startkey <= k < endkey).

Both approaches are valid and used in the real world. Ruby uses the inclusive ("right-closed" in math speak) first approach:

>> l = [1,2,3,4]
>> l.slice(1,2)
=> [2, 3]

Python uses the exclusive (”right-open” in math speak) second approach:

>>> l = [1,2,3,4]
>>> l[1:2]
[2]

For array indices both work fine and which one to prefer is mostly an issue of habit. In spoken language both approaches are used: “Have the Software done until saturday” probably means right-open to the client and right-closed to the coder.

But if you are working with keys that are more than array indexes, then right-open is much easier to handle. That is because you have to *guess* the biggest value you want to get. The Wiki at http://wiki.apache.org/couchdb/View_collation contains an example of that problem:

It is suggested that you use
startkey=”_design/”&endkey=”_design/ZZZZZZZZZ”
or
startkey=”_design/”&endkey=”_design/\u9999″
to get a list of all design documents

This breaks if a design document is named “ZZZZZZZZZTop” or “\9999Iñtërnâtiônàlizætiøn”. Such names might be unlikely but we are computer scientists; “unlikely” is a bad approach to software engineering.

The think what we really want to ask CouchDB is to “get all documents with keys starting with ‘_design/’”.

This is basically impossible to do with right-closed intervals. We could use startkey=”_design/”&endkey=”_design0″ (’0′ is the ASCII character after ‘/’) and this will work fine … until there is actually a document with the key “_design0″ in the system. Unlikely, but …

To make selection by intervals reliable currently clients have to guess the last key (the ZZZZ approach) or use the fist key not to include (the _design0 approach) and then post process the result to remove the last element returned if it exactly matches the given endkey value.

If couchdb would change to a right-open interval approach post processing would go away in most cases. See http://blogs.23.nu/c0re/2008/12/building-a-track-and-trace-application-with-couchdb/ for two real world examples.

At least for string keys and float keys changing the meaning to [startkey, endkey[ would allow selections like

* “all strings starting with ‘abc’”
* all numbers between 10.5 and 11

It also would hopefully break not to much existing code. Since the notion of endkey seems to be already considered “fishy” (see the ZZZZZ approach) most code seems to try to avoid that issue. For example ’startkey=”_design/”&endkey=”_design/ZZZZZZZZZ”‘ still would work unless you have a design document being named exactly “ZZZZZZZZZ”.

In my compay we use a logistics application called huLOG. Part of huLOGs (award winning) functionality is to aggregate track and trace events from about two dozen sources. The events are schema free in there nature: some might contain a ZIP code, some may geographic coordinates attached, some relate to a certain packet or pallet (called “movable unit” in huLOG-spreak) some relate to a certain shipment (which is a group of movable units). Some have file attachments, e.g. Images of the packages, signatures proofing delivery or pictures of the number plates of the trucks.

My first approach was an SQL database. Later I coupled it with a self-designed Document store called DoDoStorage. For background on that project see here, here and here.

In summer 2007 we experienced severe performance problems with DoDoStorage. I started a rewrite in Erlang and came across Damien Katz and his then obscure CouchDB project. It was just in the transition from XML to JSON and Damion assured me that “it will not be production ready for an other two years”.

So we decided to solve the DoDoStorage speed issues with more hardware for the time being.

In Fall 2008 the landscape for CouchDB hat changed: it was now the hot thing in database technology and everybody’s darling. CouchDB 0.9 came around and it started to look usable for serious use. We hired Jan Lenhardt, one of the CouchDB core team to work with us on moving huLOG from PostgreSQL and DoDoStorage to CouchDB.

In December we started migrating services and data over to a CouchDB based system. While CouchDB has its wards we are very happy with it so far.

Data Model

As stated above tracking data comes in many different flavors ad colors. It might reference an MUI (”movable unit ID”), a shipment or both. Let’s concentrate on events referencing a MUI. Some typical tracking events might look like this:

{
   "_id": "01420000000378-20061005T064500.000000",
   "mui": "01420000000378",
   "shipment": 572,
   "message": "Processing, 0132-Vlotho, Route 0132, Code 101",
   "code": "410",
   "timestamp": "20061005T064500.000000",
   "facility": "DPD Depot 0132",
   "ort": "Vlotho (DE)",
   "plz": "32657"
},
{
   "_id": "01420000000378-20061005T085400.000000",
   "mui": "01420000000378",
   "shipment": 572,
   "message": "proof of delivery",
   "code": "421",
   "timestamp": "20061005T085400.000000",
   "plz": "32634",
   "_attachments": {
       "POD.pdf": {
           "stub": true,
           "content_type": "application/pdf",
           "length": 136446
       }
   }
}

One thing about my choice of keys (the _id field). I have choosen a “meaningful” ID over traditional “random” UUIDs. The IDs we use consist of MUI-Timestamp The Timestamp is the time the event was generated (e.g. the pallet was loaded). The good news is that this automatically keeps my database free of duplicates. If I import the same file with events twice, the events will have the same IDs during both imports and thus the second import will overwrite the first one: exactly what I want.

The bad news is that while physically there can’t really happen two events at once due to clock drift etc. I might get two different events for a MUI with exactly the same timestamp. This would result in the same key being generated for both events and thus the second event would overwrite the first one. With the relatively sparse populated space of timestamps I expect this to happen once in 10.000 events or so.

For huLOG it is acceptable to use one in 10.000 events – especially if it is the earlier one. We only get about 95% of the events we should get due to problems in the track and trace infrastructure of the freight companies. As long as the all important “has been handed to the customer” messages arrive, our system has to be able to handle missing messages.

Data Access by MUI

With that we just need a few view functions to work with the data. Most obvious we want to get all documents for a certain MUI.

This is easy since our document IDs already contain the MUI. We only have to get a list of all IDs starting with MUI. The CouchDB Document API already provides that functionality:

$ curl -s 'http://localhost:5984/hulog_events/_all_docs
  ?startkey=%22094147562251-0%22
  &endkey=%2209445147562253-9%22
  &include_docs=true' 

{"total_rows":184708,"offset":184705,"rows":[
{"id":"09445147562251-20081114T165600.000000",
 "key":"09445147562251-20081114T165600.000000",
 "value":{"rev":"463392510"},
 "doc": {..., "message":"Einrollung, 0147, Route 0280, Code 461 105",
        "mui":"09445147562251", "shipment":124524}},
{"id":"09445147562253-20081114T165700.000000",
 "key":"09445147562253-20081114T165700.000000",
 "value":{"rev":"2436256439"},
 "doc":  {..., "message":"Einrollung, 0280, Route 0234, Code 461 105",
          "mui":"09445147562253", "shipment":124524}}
]}

Some things to note: startkey and endkey must be set to valid JSON objects. We want a string containing the mui so we have to put “quotes” arround it. After URL-encoding we end up with %22094147562251%22.

The returned documents are within the interval [startkey:endkey]. If we choose a startkey which is guaranteed same or smaller than the key we want to retrieve. That’s easy: 09445147562251 is smaller than any string which has more characters and starts with 09445147562251. Endkey is somewhat more tricky. What is the biggest value? 09445147562252? This might catch one record to much, because there actually might be a key 09445147562252.

A common idiom used in the Erlang community is to use something like 09445147562251Z as the endkey. But then what is if there actually is a key 09445147562251Zabc? So use 09445147562251ZZZZZ, but …

An other suggestion is using a “high unicode character” like \u9999. But at least there is no highest unicode code point and you are well of in considering the sorting rules for obscure unicode characters as indetermistic. See the CouchDB Wiki for further information.

In our case all this is no problem. The MUI is followed by a ISO 8601 timestamp. So
startkey=094147562251-0, endkey=09445147562253-9 should work fine until the end of year 8999.

Data Access by Shipment

That was easy. Now we want to access data based on something which is not encoded in the document ID. Say the “shipment” number. For that we need a map function which gives us shipment numbers.

map: function(doc) {
    if(doc.shipment) {
        emit(doc.shipment, null);
    }
}

We permanently save this function in the server to allow CouchDB to play its clever optimization tricks. See the CouchDB wiki for more information on the HTTP View API. You could use the Futon GUI at http://localhost:5984/_utils/ for that or curl:

 curl -X PUT -H 'Content-Type: applicatioe": "javascript",
  "views": {"all":{"map":"function(doc){if(doc.shipment)
                          {emit(doc.shipment, null);}}"}}}
  ' http://localhost:5984/hulog_events/_design%2fsendung

{"ok":true,"id":"_design/sendung","rev":"23237918"}

Now we can query the view and with the startkey and endkey parameters. Here we get again into trouble for choosing the endkey. The hack we used for the IDs did work because there was a timestamp of well known format in the ID so we construct a “slightly bigger” value. Shipments are numeric. So what is bigger than 128996? 128997 is obviously bigger but would get us documents for two shipments (128996 and 128997). Since the shipments are numeric, a trick like “128996Z” does not work out.

But numeric comparing of numeric values in JavaScript work nicely with different numeric types. Shipments are integers. If we use a float key we nicely can place it to be bigger than 128996 but smaller than 128997. We use 128996.1 as our endkey.

$ curl 'http://localhost:5984/hulog_events/_view/shipment/all
  ?startkey=128996&endkey=128996.1'

{"total_rows":184707,"offset":184674,"rows":[
{"id":"09445122481336-20081223T105212.491567","key":128996,"value":null},
{"id":"09445122481336-20081223T155200.000000","key":128996,"value":null},
{"id":"09445122481336-20081223T202600.000000","key":128996,"value":null},
{"id":"09445122481337-20081223T105216.377305","key":128996,"value":null},
{"id":"09445122481337-20081223T155300.000000","key":128996,"value":null},
{"id":"09445122481337-20081223T202500.000000","key":128996,"value":null}
]}

You can also add &include_docs=true to the query – this will get you the complete documents, not only the IDs.

At this point we have a scalable, elegant datastore for our Track & trace related events ad files.

Usage

Currently we are running CouchDB with subset of our tracking archives. This subset is about a quarter million documents of wich 20% or so have attachments resulting in a database size of about 5 GB. No complains so far.

By experimenting and reading the source I came up with thisdescription in pseudo Python:

def mapstep(alldata):
    # the map is applied to every document
    # and the result is collected in two lists of rows
    k_rows = []
    v_rows = []
    for _id, doc in alldata:
       k, v = mapfun(doc) # actually mapfunc uses emit() not return()
       k_rows.append([k, _id])
       v_rows.append(v)
    return k_rows, v_rows

def reducestep(keys, values):
    # now several reduce steps follow. For this example
    # we randomly chose two
    # all even elements
    tmp1 = reducefun(k_rows[::2], v_rows u[::2], False)
    # all uneven elements
    tmp2 = reducefun(k_rows[1::2], v_rows u[1::2], False) 

    # finally several rereduce steps follow.
    # For this example we use only one.
    return reducefun(None, [tmp1, tmp2], True)

result = reducestep(mapstep(alldocs()))

If you call the view with group=true the map step stays the same, but the server applies grouping and calls the reduce step for each group. It looks like this:

def reduce_with_grouping(keys, values):
    gdict = {}
    # create dictionary mapping values to keys
    for k, v in zip(keys, values):
        gdict.setdefault(k, []).append(v)
    ret = []
    for k, values in gdict.items():
        ret.append([k, reducestep(k*len(values), values])
    return ret

result = reduce_with_grouping(mapstep(alldocs()))

If you experiment with views keep in mind that the the Futon Web-Client silently adds group=true to your views and that group=true is ignored if you don’t provide a reduce function.

I tried to see how much beating CouchDB can take. So I installed it on a modest box (1.8GHz, 512 MB RAM, Debian) and started at 12:00h pouring data in it. At about 22:00h I asked for the computation of a simple view while still dumping data into it. At that Time it contained about 400.000 Documents of with about 10 % contained an Attatchment. DB size was on Disk was about 8 GB.

And an hour later (now) I can’t reach the box anymore.

While there are obscure mail applications which are able to read message/partial modern software ignores it, because it is a huge security issue.

This in turn means, I can’t open longer documents mailed by the MP 161 to me.

Because I like to scan large documents, I hate this quirk of the MP 161. So I wrote a Python tool called departicularifier which connects to an IMAP server, downloads the mails send by the fax, reassembles and decodes the messages and dumps the files on your harddisk.

$ python departicularifier.py --help
Usage: departicularifier.py --user=you@example.com [options].
Try departicularifier.py --help for details.

Extract files from message/partial MIME messages generated by NRG MP 161
scanfax appliances.

Options:
  --version            show program's version number and exit
  -h, --help           show this help message and exit
  --server=SERVER      hostname of the IMAPS server where the messages are
                       stored (default: "mail.hudora.biz")
  --user=USER          User name for logging into the server (default: "none")
  --password=PASSWORD  Pasword used to login (default: ask for password)
  --folder=FOLDER      Folder to scann for messages (default: "INBOX")
  --sender=SENDER      Sender whose messages to process (default: "scanner@")
  --dir=DIR            Destination directory (default: ".")

For you it need probably some fiddeling with the parameters. A run shoul look like this:

$ python decoder.py --user=m.dornseif@example.com
connecting to 'mail.hudora.biz' as 'm.dornseif@example.com'.
Password:
[long wait]
writing ./20080528132721094.pdf
writing ./20080528130042306.pdf
writing ./20080528131755217.pdf

You can download departicularifier.py here.

A typical situation in ERP systems is, that you have to use SQL tables as an “interface” between applications.
Let’s say you want to submit an order. You write an “order header” containing the customer number etc and orderlines how much to ship of what.

order = {'customer_no': 12345,
         'delivery_date': '2001-01-01',
         'orderlines': [{'item_no': '10105', 'quantity': 80},
                        {'item_no': '14600', 'quantity': 12}]
        }

In SQL you use two tables and add some primary keys:

order_head:

id   | customer_no | delivery_date | processed
-----+-------------+---------------------------
5123 | 12345       | 2001-01-01    | False

orderline:

orderid | item_no | quantity
--------+---------+----------
5123    | 10105   | 80
5123    | 14600   | 12

Since oldschool databases usually don’t have auto incrementing keys, you usually have to write something like this to insert an order:

next_id = query('SELECT MAX(id) FROM order_head')
query('INSERT INTO order_head (id, customer_no, delivery_date, processed)
       VALUES(%d, '12345', '2001-01-01', False)" % next_id)
query('INSERT INTO orderline (orderid | item_no | quantity)
       VALUES(%d, '10105', 80)" % next_id)
query('INSERT INTO orderline (orderid | item_no | quantity)
       VALUES(%d, '14600', 12)" % next_id)

This has some issues: what if some programm starts reading after you have written order_head and the first orderline, but not the second one? The ustomer would not get the goods from the second orderline. Transaction in modern SQL databases can solve this, but if you don’t have transactions you can solfe this problem in otehr ways. Let’s assume the code in the ERP for processing this interface looks like this:

for order in query('SELECT * FROM order_head'):
    for orderlines in query("SELECT * FROM orderline
                             WHERE orderid='%d'" % order.id):
        # do something with order and orderline
    query("UPDATE orderline SET processed=True
           WHERE orderid='%d'" % order.id)

This means that orderlines are never read unless there is also an header with the respective ID. So if we write the orderlines first we are save from them beeing read if tey are not ready yet. So by reordering our code we can make it already much more robust:

next_id = query('SELECT MAX(id) FROM order_head')
query('INSERT INTO orderline (orderid | item_no | quantity)
       VALUES(%d, '10105', 80)" % next_id)
query('INSERT INTO orderline (orderid | item_no | quantity)
       VALUES(%d, '14600', 12)" % next_id)
query('INSERT INTO order_head (id, customer_no, delivery_date, processed)
       VALUES(%d, '12345', '2001-01-01', False)" % next_id)

This already avoids the race condition “reading while writing”. But what if two processes try to write orders at once? Then next_id = query('SELECT MAX(id) FROM order_head') would result in both processes getting the same next_id and this mixing up two orders into a single one.

The easiest solution is to ensure that only one process can ever write orders. This avoids lot’s of trouble. But it is suprisingly hard to enforce that. So by having code which reduces the risk of dual write race conditions we can make our program much more robust.

One approach would be to use SQL functionality like this to insert the head?

query('INSERT INTO order_head (id, customer_no, delivery_date, processed)
       VALUES(SELECT MAX(id) FROM order_head, '12345', '2001-01-01', False)")

This would ensure that no id is used twice but how would we get the ID we generated for our order? SELECT MAX(id) FROM order_head would not help since some other program might have inserted another record in the meantime.

We are here in an situation where we can’t make the code reliable without resorting to some tricks. In this case the processed field is our solution: Teh application reading the database ignores all records where processed == True. So as long as we set processed to True we can use the other fields for whatever we like. In this case we use the date field to temporary store an unique id identifying the record.

So let’s generate a nice 10 character random value:

token = hex((int(time.time() * 10000)
            ^ (os.getpid() << 16)
            ^ thread.get_ident() << 8)
           % 0xFFFFFFFFFF).rstrip('L')[2:]

If this random value is guaranteed to be unique it’s easy to write a robust insertation code.

def insert_order():
    token = hex((int(time.time() * 10000)
        ^ (os.getpid() << 16)
        ^ thread.get_ident() < 1:
    next_id = query('SELECT MAX(id) FROM order_head')
    query('INSERT INTO order_head (id, customer_no, delivery_date, processed)
       VALUES(%d, '12345', %r, False)" % (next_id, token))
    rowcount = query('SELECT COUNT(*) FROM order_head WHERE id = %d' % next_id)
    if rowcount > 1:
       # the race condition has hit - remove our entry and retry
        query('DELETE FROM order_head WHERE delivery_date = %r' % token)
        time.sleep(random.randint()/100.0)
        insert_order()
    else:
        query('INSERT INTO orderline (orderid | item_no | quantity)
               VALUES(%d, '10105', 80)" % next_id)
        query('INSERT INTO orderline (orderid | item_no | quantity)
               VALUES(%d, '14600', 12)" % next_id)
        # fix up the header
        query('UPDATE order_head SET delivery_date='2001-01-01',
               processed=False WHERE delivery_date=%r' % token)

While this code is much more complex than the original, it can guarantee that no race conditions occur as long as token is unique.

It can be controlled by HTTP although it uses GET requests wich breaks idempotence. Still: it works:

curl http://admin:admin@192.168.0.225/SWITCH.CGI?s1=0 > /dev/null

One nice thing about the Switch is the fact that it supports DHCP.

The Software Stack running on the Switch seems to be based on QNX and acts quite reasonable. (See the manual for an overview). E.g. you can use curl to switch ports on and off. E.g.

curl --data 'P4=0' -i http://admin:admin@192.168.100.100/config/home_f.html

to switch Port 4 off.

One thing I don’t like about the device is the fact that it doesn’t support DHCP but needs a fixed IP. On the Pro side it comes with an additional RS-232 port for configuration and operation.

The Switch is on sale for about 400 Euro which results in a cost per port of about 100 Euro.

LoadModule authz_host_module libexec/apache22/mod_authz_host.so
LoadModule authz_user_module libexec/apache22/mod_authz_user.so
LoadModule authnz_ldap_module libexec/apache22/mod_authnz_ldap.so
LoadModule authz_default_module libexec/apache22/mod_authz_default.so
LoadModule auth_basic_module libexec/apache22/mod_auth_basic.so
LoadModule auth_digest_module libexec/apache22/mod_auth_digest.so
LoadModule ldap_module libexec/apache22/mod_ldap.so

  AuthName "HUDORA Internal"
  AuthType Basic
  AuthBasicProvider ldap
  AuthLDAPURL ldap://zimbra.example.com/?mail
  Require valid-user

This should alow you to login as yourmail@zimbra.example.com with your zimbra password.

Enjoy!

For monitoring 4 disks connected to a 3ware 9500S-12 i changed /usr/local/etc/munin/plugin-conf.d/plugins.conf to read:

[hddtemp_smartctl]
user root
env.smartctl /usr/local/sbin/smartctl
env.drives ad0 da0 da1 da2 da3
env.type_da0 3ware,1
env.type_da1 3ware,1
env.type_da2 3ware,2
env.type_da3 3ware,3
env.dev_da0 twa0
env.dev_da1 twa0
env.dev_da2 twa0
env.dev_da3 twa0

Finally you have to install /usr/ports/sysutils/smartmontools.

That’s it

It turned out JasperReports had all I needed. While Jasper is promoted as reporting solution it can be used as a simple PDF generator. Together with the iReport visual report designer we had the complete Toolchain. USers could use iReport to “paint” the desired PDF output using placeholders for all the variable data to be filled in. iReport would save this design in a .jrxml file. The jasper library would then compile that file to a .jasper file for performance reasons.

For generation the actual report you would feed that .jasper and a “datasource” to the JasperReports library.. A datasource could be a SQL/JDBC query or an xml-file. This datasource would than be used to fill in the placeholders you left in the .jrxml and generate a PDF.

All nice and smooth. The only problem is: Jasper Reports is written in Java and our application stack is written mostly in Python. I also avoid programming Java whenever possible. I feel incredibly clumsy when having to code in Java.

Jython to the rescue! Jython allows you to access Java libraries and still write Python code. this is done by having Python running inside a Java VM. Wit some help we where able to generate PDFs from jython. But the whole beast was still running inside the java VM.

So to call it from our regular Python code running inside mod_python on the Apache web server we had to write the XML to disk, fork a process, fire up the Java VM with Jython and the call the Jasper library to generate the PDF from the XML and the jrxml report source code.

Worked but took about 7 seconds per document. This was OK for low volume production but for some application we needed much shorter turn-arround times. It turned out, starting up the java VM was the biggest time consumer. So we created a long running Jython process which received PDF rendering jobs via a home grown UDP Protocol. It was messi. Sending absolute filesystem paths via UDP was probably not a good idea but it worked. PDF Generation times where down to 1.5 seconds or so. Using ths structure we have been running 21 months or so and generated about 500.000 PDFs.

But it was still a mess. Especially it meant that every server which had to generate PDFs also needed Java installed. And Java on FreeBSD Servers is very limited fun. So we came up with a servlet based approach: There would be a webserver; upon POSTing a XML datasource and a JRXML file to a specific URL you would get back a rendered PDF. As a Java based Webserver we use Jetty which is driving a Jython servlet calling the JasperReports library. Works like a charm.

After using it for 9 Months or so Im just in the process of publishing this Jetty based version real soon now(TM). It will be available at http://cybernetics.hudora.biz/projects/wiki/pyJasper

This is based on Ben Fry’s zipdecode and PLZ.tab from OpenGeoDB. I used a simple script to preprocess the data.

Built with Processing.

If the Java Applet doesn’t work, try this page. Klick into the applet and tipe a few numbers, e.g. “42″

host# ezjail-admin create -f djangoserver extranet a.b.c.6
host# /usr/local/etc/rc.d/ezjail.sh start extranet

workstation# ssh a.b.c.6
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
	The Regents of the University of California.  All rights reserved.

FreeBSD 7.0-RELEASE (GENERIC) #0: Sun Feb 24 10:35:36 UTC 2008

This is a preconfigured virtual machine (jail) for running Django applications.
Avoid manual configuration!
[md@extranet ~]$ su
extranet# cd /usr/local/www
extranet# svn co https://example.com/.../projects/extranet/trunk extranet
extranet# ln -s /usr/local/www/extranet /usr/local/www/djangoprojectdir
extranet# sh /usr/local/etc/rc.d/apache22 start

delete aolsystemmsgs

# MAC       TelNr SIP-Username   Monitor Caller-IDs       Realname
0004132620A2 4130 pudel          4623,4629,4677,4630      Carsten Pudel
000413262E20 4110 bibo           4132,4116,4124,4131,4138 Birgit Bonrath
0004132620a7 4111 edornseif      4122,4137,4901,4611,4133 Evelyn Dornseif
00041323B063 4112 mdornseif      4110,4132,4116,4124      Dr. Dornseif

MAC is the MAC-Address (can be found under the phone below a barcode), TelNr is the desired extension where this phone should be reachable, SIP-Username is the sip user name you wish to use for this phone. Monitor Caller-IDs is the list of other phone numbers (up to 8) which should be monitored via the special buttons on the right of a Snom telephone.

The Python script snom.py reads the configuration shown above from a file called snom.txt. It then generates a lot of new configuration files:

To make Snom Auto configuration (”Mass Deployment“) work you need to update your Phones to Firmeware V7. You also need a webserver beeing able to deliver the directory ./snom/. I assume the Webserver is available at the IP-Address 172.28.1.2.

Finally you need some additional files:

snom/snom3x0.htm – master configuration file

<?xml version="1.0" encoding="utf-8" ?>
<setting-files>
  <file url="http://172.28.1.2/snom/general.xml" />
  <file url="http://172.28.1.2/snom/phonebook.xml" />
  <file url="http://provisioning.snom.com/config/web_lang.xml" />
  <file url="http://provisioning.snom.com/config/gui_lang.xml" />
</setting-files>

snom/general.xml – configuration for all you phones

<?xml version="1.0" encoding="utf-8" ?>
<settings>
 <phone-settings>
  <challenge_response perm="&">off</challenge_response>
  <filter_registrar perm="&">off</filter_registrar>
  <guess_number perm="&">off</guess_number>
  <user_phone perm="&">off</user_phone>
  <silence_compression perm="&">off</silence_compression>
  <date_us_format perm="&">off</date_us_format>
  <time_24_format perm="&">on</time_24_format>
  <user_outbound idx="1" perm="$">172.28.1.2</user_outbound>
  <user_host idx="1" perm="$">172.28.1.2</user_host>
  <user_pname idx="1" perm="$"></user_pname>
  <user_active idx="1" perm="$">on</user_active>
  <user_sipusername_as_line idx="1" perm="$">on</user_sipusername_as_line>
  <ntp_server perm="$">172.28.1.2</ntp_server>
  <timezone perm="$">GER+1</timezone>
  <web_language perm="$">Deutsch</web_language>
  <language perm="$">Deutsch</language>
  <tone_scheme perm="$">GER</tone_scheme>
 </phone-settings>
</settings>

Nov configure your Phones to get the Configuration from your Webserver and you are done. Ether use DHCP or edit the phones configuration at Advanced -> Update -> Setting URL:

Save, reboot and you should be done.

Now all Phones

* Are configured automatically
* get nice printed labels
* get a up-to date phonebook

while you yourself have only to edit a single file – snom.txt. This was tested with “Asterisk 1.2.24-BRIstuffed-0.3.0-PRE-1y-k”.

1. Install unixodbc

2. Install IBM iSeriesAccess

3. edit /etc/odbcinst.ini

[ODBC]
Trace		= No
TraceFile	= /tmp/sql.log
ForceTrace	= No
Pooling		= Yes

[iSeries Access ODBC Driver]
Description	= iSeries Access for Linux ODBC Driver
Driver		= /opt/ibm/iSeriesAccess/lib/libcwbodbc.so
Driver64	    = /opt/ibm/iSeriesAccess/lib64/libcwbodbc.so
Setup		= /opt/ibm/iSeriesAccess/lib/libcwbodbcs.so
Setup64		= /opt/ibm/iSeriesAccess/lib64/libcwbodbcs.so
UsageCount	= 1
CPTimeout	=
CPReuse		=
Threading	= 2
DontDLClose	= 1
FileUsage	= 1
# TRACE           = 31

4. edit /etc/odbc.ini

; see http://publib.boulder.ibm.com/infocenter/
            iseries/v5r3/?topic=/rzaik/connectkeywords.htm
; ConnectionType=0 = Read/Write | 1 = Read/Call | 2 = Read-only

[AS400]
;CCSID=1208 ; utf-8 unicode
; 273 germany
CCSID			= 819 ; latin-1
UNICODESQL 		= 1
Description		= Production AS/400 Database
Driver			= iSeries Access ODBC Driver
System			= as400.example.com
UserID			= ODBC_RW
Password		= Hj12SDf7gZ
DefaultLibraries	= SMKDIFP ; you have to change this
AllowDataCompression	= 1
AllowUnsupportedChar	= 1
ForceTranslation	= 0
Trace			= 0
TraceFile               = /tmp/odbc.trace
Servername		= as400.example.com

You now can type isql AS400 to access your machine.

The Hokuyo URG04 LIDAR now works as expected. This is the code I used to read and display the data.

See here for possible applications.

So far the Results are not exactly what I was expecting.

… perhaps a cluttered kitchen table is a bad testing range.

The beauty of this solution is that it also can be used to identify where these goods are put.

By carefully placing barcode labels on your storage racks you can scan at which location a pallet is being put. Care must be taken so neither the pallet nor the forks hide the barcode. You probably have to experiment with the placement and angle of barcode reader.

Then every location can have it’s unique ID consisting of Row of shelfes (red in the image above), Column (all locations above each other; blue in the image above) and Level (all location besides each other; yellow in the image above).

This scheme allows you to label allocations with Row-Column-Level, e.g. 01-02-03 for the first Row, second Column, third Level. Usually labels identifying the storage locations are places on the shelfing above level 1.

An easy way to make Pallets identifiable is to put Barcode stickers on the Pallets. If you place them on the middle block they are easy to fix there by glue or clamps and easy to read automatically.

One good location to place a barcode reader is at the fork/mast assembly (detail):

Mount the scanner looking about 30 degrees downward. At the mast assembly the barcode scanner is relatively well protected against damage an can be affixed without to much hassle. The main obstacle is getting power and data lines from the moving fork/mast assembly to the chassis.

After mounting the scanner it should permanently project a scan line on the forks about 80 cm away from the mast.

Now when the fork lift picks up a pallet it should automatically read the barcode during entering the pallet. See below for an illustration:

Sun knowingly ships broken Java stuff to its paying customers.

Another fact of a developer’s life is that there are always more things to work on than time to do so. And everything is of the utmost priority to someone. And so here again, we find ourselves working on multiple things at the same time, making slow-yet-incremental progress on each of them so at least we can say about them all with a clear conscience and a forced smile, “As it turns out, I’m working on that right now.” (Subversion Blog)

Good preformance! As always, there is little software i can recoment more than Zimbra.

/Applications/iChat.app/Contents/MacOS/iChat -errorLogLevel 7

Very often I hold the camera in the wrong Position. QuickTime helps.

1. Create the users in Zimbra. Enable “External POP3 access” for these users.

2. Log into the Managment Site of your Google Apps and change the passwords of all users to something you know.

3.
Log into the Zimbra Account using the Zimbra managment interface. Create a folder “Gmail”. Navigate to Options -> POP Accounts and configure access to Gmail.

3. Select the folder Gmail and than select “Get POP Mail”. In my experience this is very unreliable. It took ca. 20 attempts to get all mails and resulted in many duplicates.

* cannot be easily defined so that all stakeholders agree on the problem to solve;
* require complex judgements about the level of abstraction at which to define the problem;
* have no clear stopping rules;
* have better or worse solutions, not right and wrong ones;
* have no objective measure of success;
* require iteration-every trial counts;
* have no given alternative solutions-these must be discovered;
* often have strong moral, political or professional dimensions”

(Simon Buckingham Shum)

1. Find the Python configuration file site.py (e.g. locate /site.py)

2. sudo perl -npe 's/encoding = "ascii"/encoding = "utf-8"/g;' -i.bak /opt/local/lib/python2.5/site.py

This ^^^^ is more readable than that.

I had feared worse.

Remove all pylint/logilab traces in /opt/local/lib/python2.5/site-packages/easy-install.pth. Then try:

$ sudo rm -Rf /opt/local/lib/python2.5/site-packages/pylint*
$ sudo rm -Rf /opt/local/lib/python2.5/site-packages/logilab*
$ wget ftp://ftp.logilab.fr/pub/pylint/pylint-0.13.2.tar.gz
$ wget ftp://ftp.logilab.fr/pub/common/logilab-common-0.25.1.tar.gz
$ wget ftp://ftp.logilab.fr/pub/astng/logilab-astng-0.17.1.tar.gz
$ tar logilab-common-0.25.1.tar.gz
$ tar xzvf logilab-common-0.25.1.tar.gz
$ cd logilab-common-0.25.1
$ python manage.py test mypl2
$ sudo python setup.py install
$ cd ..
$ tar xzvf logilab-astng-0.17.1.tar.gz
$ cd logilab-astng-0.17.1
$ sudo python setup.py install
$ cd ..
$ tar xzvf pylint-0.13.2.tar.gz
$ cd pylint-0.13.2
$ sudo python setup.py install
$ cd ..

Erlang (BEAM) emulator version 5.5.2

Eshell V5.5.2  (abort with ^G)
1> application:start(odbc).
ok
2> {ok,Ref} = odbc:connect("DSN=HD400",[{trace_driver,on}]).
{ok,}
3> odbc:sql_query(Ref, "SELECT * FROM ISB00").
{selected,["IBFNR",
           "IBBWSL",
           "IBLGNR",
           "IBARTN",
           "IBMNGB",
           "IBINFO",
           "IBFCOD",
           "IBDFSL",
           "IBSTAT"],
          [{"01",
            135,
            100,
            "85030               ",
            12.0000,
            "275                 ",
            "          ",
            "          ",
            "X"},
...
4> odbc:sql_query(Ref, "UPDATE SMKDIFP.IET00 SET ETDRET=2 WHERE ETSANR=355384").
{updated,1}

\o/

$ sudo sh -c 'port install py25-bsddb ; port install py25-bz2 ; port install py25-hashlib ; port install py25-httplib2 ; port install py25-ipython ; port install py25-pil ; port install py25-psycopg2 ; port install py25-readline ; port install py25-simplejson ; port install py25-sqlalchemy ; port install py25-zlib ; port install py25-checker ; port install py25-mysql'

Now make the MacPorts Python the default:

$ sudo ln -s /opt/local/bin/python2.5 /opt/local/bin/python
$ sudo ln -s /opt/local/bin/pydoc2.5 /opt/local/bin/pydoc
$ sudo ln -s /opt/local/bin/easy_install-2.5 /opt/local/bin/easy_install

Check we are now using the right Python:

$ which python
/opt/local/bin/python

Install various additional software packages:
$ sudo easy_install http://bicyclerepair.sourceforge.net/bzr/bicyclerepair-nightly.tar.gz
$ sudo easy_install docutils
$ sudo easy_install pyro
$ sudo easy_install huTools
$ sudo easy_install dodostorage
$ sudo easy_install huBarcode
$ svn co -r6040 http://code.djangoproject.com/svn/django/trunk/ django
$ cd django
$ sudo python setup.py install
$ cd ..
$ sudo rm -Rf django

]]>
			http://blogs.23.nu/c0re/2007/12/antville-16716/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16714/
		http://blogs.23.nu/c0re/2007/12/antville-16714/#comments
		Sun, 09 Dec 2007 18:43:50 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16714/
		
			… bug.

“You build it, we break it”.
]]>
			http://blogs.23.nu/c0re/2007/12/antville-16714/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16713/
		http://blogs.23.nu/c0re/2007/12/antville-16713/#comments
		Sun, 09 Dec 2007 12:07:51 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16713/
		
			“A much more effective knob to twiddle when it comes to time to market is scope.  Cut scope.  Cut it early and cut it again if the project is drifting off schedule.  It is amazing how much scope can really be done away with and still have a release that completely satisfies commercial needs and makes customers happy.” (Why Small Software Teams Grow Large And Other Software Development Conundrums)

Create software, not code!
“What does “done” mean to you?  I find that my definition of “done” continues to expand. When I was still pretty green, I would say “It’s done” when I had finished coding.” (Wide Awake Developers)
“It is this reason that languages like Ruby, which is implemented mainly in C, provide less maximum power than languages like Smalltalk, which is implemented mainly in … Smalltalk.” (The significance of the meta-circular interpreter)
“For years I’d been a bit confused about the standard “don’t program defensively” Erlang advice, but just recently I saw a definition of what that really means.” (defensive programming)
“Second, the cost of global consistency is global serialization. If you assume a certain volume of traffic you must support, the cost of a globally consistent solution will be a multiple of the cost of a latency-tolerant solution. That’s because global consistency can only be achieved by making a single master system of record. When you try to reach large scales, that master system of record is going to be hideously expensive.” (Wide Awake Developers)
Investing in your own abilities. That?s incredibly important, but I don?t need to tell you that. If you exercise with programming katas, or learn a new programming language once a year, or pick up a book like The Reasoned Schemer and actually go through the exercises, then you are already in the top 1% of software developers for personal skills improvement. (Sorry, certifications don?t count. They are the classic case of doing the wrong thing for the wrong reason!)” (But Y would I want to do a thing like this?)
“A closure is a function that captures the bindings of free variables in its lexical context.” (A Definition of Closures)
“We have an example of the ?Telephone Test:? when code very closely resembles how you would explain your solution over the telephone, we often say it is ?very high level.? The usual case is that such code expresses a lot more what and a lot less how.” (Why Why Functional Programming Matters Matters)
“The first person to write about these issues, as far as I know, was Fred Brooks in the Mythical Man Month. He wrote that programmers seemed to generate about the same amount of code per day regardless of the language. When I first read this in my early twenties, it was a big surprise to me and seemed to have huge implications. It meant that (a) the only way to get software written faster was to use a more succinct language, and (b) someone who took the trouble to do this could leave competitors who didn’t in the dust.” (Succinctness is Power)
“We can often ignore the time and space performance penalties, but the cost of comprehending any abstraction is real and slowly increases as we add abstractions in our code” (Abstraction and Variation)
Programs must be written for people to read, and only incidentally for machines to execute.

(Abelson & Sussman, Structure and Interpretation of Computer Programs via Writing programs for people to read)
“Is it possible to create a large product with a small team? How do you do it? Well, the first realization is that what we want to do is create software, not code. Coding takes time. Typing takes time. Testing and fixing bugs takes time. Don’t code. Create software. Big difference.” (Create software, not code)
“I’ve been coding in Python for years, but lately I?ve been using it a lot, and can’t get enough of it. It’s so clean, and as Fredrick?s code shows, it?s got some extremely powerful capabilities that are also extremely easy to use. It?s obviously smoking fast as well. Some folks who commented on some of my previous blog entries seem to equate programs written using dynamic languages like Python with unmaintainable one-off solutions. If that’s your experience with such languages, blame the programmer, not the language.” (Steve Vinoski)
]]>
			http://blogs.23.nu/c0re/2007/12/antville-16713/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16710/
		http://blogs.23.nu/c0re/2007/12/antville-16710/#comments
		Sun, 09 Dec 2007 07:47:27 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16710/
		
			I encourage my coworkers to lock their coats in their locker. Security, after all, is everyone’s business. But often gentle encouragement is not enough. Sometimes, more.. persuasive methods are necessary.

This is called the noble art of goating.
Goating techniques vary from insidious and subtle to invasive, borderline vandalism. I prefer the milder forms:
* Turning the coat inside out.

* Knoting the arms of the coat.

* Hanging the coat in an other office.

* Putting the car keys in an other coat.

* Take out the Cellular and remove the battery.
Much hilarity ensued, and more importantly, crucial lessons are learned about security by all.
If this is funny, why shouldn’t the above be funny too?
]]>
			http://blogs.23.nu/c0re/2007/12/antville-16710/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16703/
		http://blogs.23.nu/c0re/2007/12/antville-16703/#comments
		Sat, 08 Dec 2007 07:38:01 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16703/
		
			“I love throwing away old code” (effbot)

]]>
			http://blogs.23.nu/c0re/2007/12/antville-16703/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16684/
		http://blogs.23.nu/c0re/2007/12/antville-16684/#comments
		Fri, 07 Dec 2007 05:49:07 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16684/
		
			
Erlang (BEAM) emulator version 5.5.5 [source] [async-threads:0] [hipe]

Eshell V5.5.5  (abort with ^G)
(mypl_produktion@airvent)1> mypl_provpipeline:update_pipeline({versandtermin, "931631", "2007-10-14"}).
Floating point exception (core dumped)

]]>
			http://blogs.23.nu/c0re/2007/12/antville-16684/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16682/
		http://blogs.23.nu/c0re/2007/12/antville-16682/#comments
		Fri, 07 Dec 2007 05:26:38 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16682/
		
			

]]>
			http://blogs.23.nu/c0re/2007/12/antville-16682/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16678/
		http://blogs.23.nu/c0re/2007/12/antville-16678/#comments
		Fri, 07 Dec 2007 05:16:16 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16678/
		
			I have been complainingabout ActiveMQ for some time. Just an other example.



ERROR RecoveryListenerAdapter        - Message id ID:lichtblick.local-53968-1187158776157-5:16123:-1:1:2 could not be recovered from the data store!


Such a thing should never ever happen ….
]]>
			http://blogs.23.nu/c0re/2007/12/antville-16678/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16663/
		http://blogs.23.nu/c0re/2007/12/antville-16663/#comments
		Wed, 05 Dec 2007 18:19:35 +0000
		mdornseif
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16663/
		
			

]]>
			http://blogs.23.nu/c0re/2007/12/antville-16663/feed/
		0
		
		
		
		http://blogs.23.nu/c0re/2007/12/antville-16660/
		http://blogs.23.nu/c0re/2007/12/antville-16660/#comments
		Wed, 05 Dec 2007 08:02:13 +0000
		cklein
				

		http://3.blogs.23.nu/c0re/2007/12/antville-16660/
		
			
WOPR:~ chris$ traceroute mailbox.informatik.uni-bonn.de
traceroute: Warning: mailbox.informatik.uni-bonn.de has multiple addresses; using 131.220.10.6
traceroute to mailbox.informatik.uni-bonn.de (131.220.10.6), 64 hops max, 40 byte packets
 1  untergrund (192.168.1.1)  2.959 ms  2.220 ms  3.345 ms
 2  ip-62-143-76-1.1311a-cud12k-01.ish.de (62.143.76.1)  24.975 ms  27.897 ms  12.234 ms
 3  172.16.178.33 (172.16.178.33)  13.206 ms  21.509 ms  22.604 ms
 4  172.16.178.1 (172.16.178.1)  14.644 ms  12.357 ms  15.493 ms
 5  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  12.126 ms  11.744 ms  11.814 ms
 6  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  12.283 ms  17.856 ms  11.603 ms
 7  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  12.859 ms  12.138 ms  24.029 ms
 8  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  13.486 ms  13.515 ms  13.303 ms
 9  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  15.488 ms  21.870 ms  13.750 ms
10  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  16.188 ms  13.879 ms  23.698 ms
11  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  30.599 ms  20.972 ms  13.382 ms
12  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  15.153 ms  15.136 ms  27.570 ms
13  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  14.111 ms  16.131 ms  17.484 ms
14  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  50.386 ms *  13.142 ms
15  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  14.421 ms  16.341 ms  15.112 ms
16  * ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  15.662 ms  18.333 ms
17  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  15.846 ms  14.929 ms  17.119 ms
18  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  29.123 ms  17.760 ms  15.568 ms
19  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  16.497 ms  19.547 ms  16.626 ms
20  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  15.707 ms  25.336 ms  17.115 ms
21  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  19.426 ms  17.781 ms  26.571 ms
22  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  18.726 ms  19.612 ms  14.941 ms
23  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  17.203 ms  15.792 ms  16.885 ms
24  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  19.307 ms  20.128 ms  16.314 ms
25  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  18.996 ms  17.179 ms  17.521 ms
26  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  19.144 ms  21.354 ms  19.589 ms
27  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  23.307 ms  18.998 ms  19.436 ms
28  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  25.242 ms  19.498 ms *
29  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  20.401 ms  18.100 ms  20.142 ms
30  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  20.928 ms  19.119 ms  19.471 ms
31  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  77.465 ms  18.578 ms  21.112 ms
32  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  27.145 ms  209.941 ms  20.840 ms
33  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  18.599 ms  22.922 ms  24.180 ms
34  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  31.935 ms  25.824 ms  19.293 ms
35  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  98.172 ms  23.434 ms  24.214 ms
36  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  60.520 ms  138.006 ms  24.547 ms
37  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  26.328 ms  76.100 ms  32.585 ms
38  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  38.920 ms  21.373 ms *
39  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  28.277 ms  23.404 ms  23.241 ms
40  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  43.978 ms  66.590 ms  24.521 ms
41  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  156.096 ms  224.505 ms  112.153 ms
42  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  46.065 ms  25.534 ms  21.754 ms
43  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  24.619 ms  23.950 ms  26.204 ms
44  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  49.543 ms  22.085 ms  24.431 ms
45  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  23.796 ms  35.980 ms  23.186 ms
46  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  23.151 ms  22.807 ms  43.882 ms
47  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  24.141 ms  27.384 ms  23.296 ms
48  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  42.035 ms  27.507 ms  24.218 ms
49  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  28.289 ms  107.582 ms  32.636 ms
50  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  26.489 ms  24.413 ms  28.512 ms
51  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  25.050 ms  30.975 ms  24.277 ms
52  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  49.317 ms  111.538 ms *
53  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  27.210 ms  29.153 ms  29.203 ms
54  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  32.085 ms  25.569 ms  26.230 ms
55  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  33.927 ms  28.220 ms  30.343 ms
56  * ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  26.491 ms  30.228 ms
57  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  41.982 ms  28.645 ms  26.162 ms
58  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  55.247 ms  37.147 ms  27.451 ms
59  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  27.899 ms  30.886 ms  28.641 ms
60  * * ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  30.947 ms
61  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  31.050 ms  31.697 ms  29.419 ms
62  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  29.071 ms  27.042 ms  27.894 ms
63  ph-13noc-jm160r-02-so-0-1-0.kerpen.ish.de (80.69.96.73)  33.857 ms  31.840 ms  28.685 ms
64  ph-1311g-jm160r-01-so-0-2-0.huerth.ish.de (80.69.96.74)  29.392 ms  31.419 ms  28.360 ms
WOPR:~ chris$

After figuring out that the WMS that came with our ERP Software Stack from SoftM AG was a) bad software and b) never really integrated into our processes we decided to write our own replacement for it.

We started about 18 Months ago one experimental testbed and two prototypes using different techniques in Python. In October 2007 we started the final implementation.

The Software uses a custom built database optimized for warehousing, called kernelE. kernelE ist written using the Erlang/OTP Software framework. kernelE is depicted in gray in the diagram above. The size of kernelE is about 5500 LoC Erlang Code of which 10% are auto generated interface code.

kernelE is used by a Python application build with the Django framework. This Python Application is also providing the buiseness logic and the interfacing to the legacy AS/400 based ERP System. The Python application contains slightly more than 1000 LoC and depends heavily on Metaprogramming techniques like higher Order functions and Metaclasses.

It also sits on top of a 8100 LoC interface library for communicating with the AS/400 and 400 LoC for logistics calculations. These Libraries where developed during the building of the various prototypes.

All in all I’m pretty amazed how well it all worked out. Some more Information on it in german can be found here, here, here and here.