This is a topic currently on my radar as well. I'll dump my findings here in the hopes that they might point you in the right direction.
The likely route I will attempt first will be PhantomJS. It's a headless WebKit browser, so it should have excellent DOM, JSON, HTML5, and CSS selectors support (it works with jQuery and qUnit, for example).
I chose this because it is used by knockout.js itself, which I discovered in the knockout.js repository, where there was a .travis.yml file and this comment:
I don't have any proof that this is going to work, but was encouraged by its use in knockout.js core. I also found this runner script for knockout/phantom that looks like a great launch point.
I've also found a few examples using Mocha and PhantomJS via node.js, including this lib extending grunt to run mocha inside Phantom, and this script showing how to run mocha inside PhantomJS. So that part is certain, at the very least.
Another solution noted in the knockoutjs archives, is to use knockout-node and JsDOM to create a workable DOM, but at first glance, this seemed too nebulous and likely to result in implementing your own test environment.
There is a slidedeck suggesting zombie.js would work with knockout/node/etc. But I can't find anything offering hard evidence, so I didn't like this route either.