Summary
The most critical OS subsystem for the performance and stability of a database like Cassandra is disk.
Tobert wrote an excellent go utility called effio
that harnesses the power of fio
and d3.js
for io benchmarking and charting (respectively).
His readme doc is a bit light so I thought I'd do a deeper dive on how to actually run effio. I did this on an [ancient] ubunto 12.04 box, so I ended up having to build a lot of things from source. You may be able to pull some of the dependencies right from the packages if you're on a more modern OS.
Dependencies
Install golang from a tarball, version 1.0.3--which is what sits in the 12.04 repos--is to old to build effio
:
wget https://godeb.s3.amazonaws.com/godeb-amd64.tar.gz
./godeb install
Install fio from source. Again the debian repos for precise were too old. That version doesn't support the json output flag which Al leverages in effio. I went back and forth between a few repos and ended up building master from kernel.dk:
git clone git://git.kernel.dk/fio.git
cd fio/
make
Make this accessible from your shell by adding a syslink:
sudo ln -s <full path to fio binary> /usr/bin/fio
Pull, build, and run effio
git clone https://github.com/tobert/effio.git
cd effio/
go build
Set up your devices.json. This is what tells effio
what disk to use and how to catalogue it. The catalog stuff is important for the UI.
Here's the config from the machine I was using:
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 10321208 1982388 7814532 21% /
udev 3806788 8 3806780 1% /dev
tmpfs 1525892 196 1525696 1% /run
none 5120 0 5120 0% /run/lock
none 3814728 0 3814728 0% /run/shm
/dev/xvdb 433455904 203012 411234588 1% /mnt/ephemeral
capacity:
$ sudo blockdev --getsize64 /dev/xvdb
450934865920
block size:
$ sudo blockdev --getpbsz /dev/xvdb
512
Based on the info above, my devices.json looks as follows:
[
{
"name": "ec2-ctool-box",
"notes": "single disk",
"ignore": false,
"device": "/dev/xvdb",
"mountpoint": "/mnt/ephemeral",
"filesystem": "ext4",
"brand": "aws",
"series": "m3 ephemeral",
"datasheet": "",
"capacity": 450934865920,
"rotational": true,
"transport": "SATA",
"hba": "AHCI",
"media": "MLC",
"blocksize": 512,
"rpm": 0
}
]
./effio run
takes a name, a device configuration, and a fio
configuration file and runs the stress test on the disk. For this example, I'm using the latency_full
default config that tobert ships with the tool. Seems to have a bit of everything.
sudo rm -rf suites && sudo ./effio run -name sebTest -dev ./conf/machines/devices.json -fio ./conf/fio/latency_full/
This generates a directory called suites
where the fio config and results are saved. It's going to take a while.
I wanted to monitor progress and pulled up htop:
Doesn't seem like there's much going on huh. Took me a second to realize that most of the CPU activity was probably waiting on disk!
Here's htop with detailed CPU time turned on:
Confirm by checking the io wait column in dstat -rvn 10
Clearly we're hitting disk pretty hard. After the benchmark is complete, we can host and play with the d3.js visualizatons.
This sets up the data in the format that effio
needs to make pretty d3 charts. I did have to fix one issue to get this to work:
mkdir public/data
./effio summarize-all -path=suites
Finally host the data on port 9000.
./effio serve
You can hit http://hostname:9000 and start exploring your results!
Enjoy!