Summary

The most critical OS subsystem for the performance and stability of a database like Cassandra is disk.

Tobert wrote an excellent go utility called effio that harnesses the power of fio and d3.js for io benchmarking and charting (respectively).

His readme doc is a bit light so I thought I'd do a deeper dive on how to actually run effio. I did this on an [ancient] ubunto 12.04 box, so I ended up having to build a lot of things from source. You may be able to pull some of the dependencies right from the packages if you're on a more modern OS.

Dependencies

Install golang from a tarball, version 1.0.3--which is what sits in the 12.04 repos--is to old to build effio:

wget https://godeb.s3.amazonaws.com/godeb-amd64.tar.gz
./godeb install

Install fio from source. Again the debian repos for precise were too old. That version doesn't support the json output flag which Al leverages in effio. I went back and forth between a few repos and ended up building master from kernel.dk:

git clone git://git.kernel.dk/fio.git
cd fio/
make

Make this accessible from your shell by adding a syslink:

sudo ln -s <full path to fio binary> /usr/bin/fio

Pull, build, and run effio

git clone https://github.com/tobert/effio.git

cd effio/

go build

Set up your devices.json. This is what tells effio what disk to use and how to catalogue it. The catalog stuff is important for the UI.

Here's the config from the machine I was using:

$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1      10321208 1982388   7814532  21% /
udev             3806788       8   3806780   1% /dev
tmpfs            1525892     196   1525696   1% /run
none                5120       0      5120   0% /run/lock
none             3814728       0   3814728   0% /run/shm
/dev/xvdb      433455904  203012 411234588   1% /mnt/ephemeral

capacity:

$ sudo blockdev --getsize64 /dev/xvdb
450934865920

block size:

$ sudo blockdev --getpbsz /dev/xvdb
512

Based on the info above, my devices.json looks as follows:

[
  {
    "name": "ec2-ctool-box",
    "notes": "single disk",
    "ignore": false,
    "device": "/dev/xvdb",
    "mountpoint": "/mnt/ephemeral",
    "filesystem": "ext4",
    "brand": "aws",
    "series": "m3 ephemeral",
    "datasheet": "",
    "capacity": 450934865920,
    "rotational": true,
    "transport": "SATA",
    "hba": "AHCI",
    "media": "MLC",
    "blocksize": 512,
    "rpm": 0
  }
]

./effio run takes a name, a device configuration, and a fio configuration file and runs the stress test on the disk. For this example, I'm using the latency_full default config that tobert ships with the tool. Seems to have a bit of everything.

sudo rm -rf suites && sudo ./effio run -name sebTest -dev ./conf/machines/devices.json -fio ./conf/fio/latency_full/

This generates a directory called suites where the fio config and results are saved. It's going to take a while.

I wanted to monitor progress and pulled up htop:
regular htop

Doesn't seem like there's much going on huh. Took me a second to realize that most of the CPU activity was probably waiting on disk!

Here's htop with detailed CPU time turned on:

detailed htop

Confirm by checking the io wait column in dstat -rvn 10

dstat

Clearly we're hitting disk pretty hard. After the benchmark is complete, we can host and play with the d3.js visualizatons.

This sets up the data in the format that effio needs to make pretty d3 charts. I did have to fix one issue to get this to work:

mkdir public/data

./effio summarize-all -path=suites

Finally host the data on port 9000.

./effio serve

You can hit http://hostname:9000 and start exploring your results!

effio

Enjoy!