What is the recommended method of testing a JSON client-server api?

Question

Edit #2: Does anyone have a good method of testing the "middle" of a client-server application where we can intercept requests and responses, fake the client or server as needed, and which provides self-documentation of the api?

Cucumber might be a good solution in many cases, but it's not quite what I'm looking for. And this middle layer should be client/server implementation agnostic. (e.g., black-box).

---

Our client-server model is a ruby-on-rails server with a Flex client, using a RESTish interface with JSON as the data format. So anything the client posts to the server is usually a single JSON parameter. The server does it's thing and responds with a pure JSON model.

We have standard rails testing on the server and we're working on getting proper FlexUnit tests completed on the client (it's a moving target). However, there's a debate in my team about the effectiveness of the current testing model, since every change on the server seems to break part of the API. This is telling me that there is both a problem with API communication (between team members, self-documentation in code, etc..), and a lack of proper API sanity testing.

So I've been questioning whether we need to have a mock client for testing the server at a pure JSON level (without all the other complexities of a rich client), and possibly a mock-server for doing the same thing with the rich client. This would serve two purposes, to document the API and to provide more thorough testing of the API itself.

The reason there's a debate is that the rails guy claims that the rails integration testing is sufficient for testing all the server requests, and the middle-ground testing environment would simply be redundant.

So the question here is, given our situation, how should be go about self-documenting the API, and how should we test the API itself?

EDIT:

We have routes like /foo/444/bar.js, but the parameters can be virtually any complex JSON string depending on the action, e.g.:

json={
  "foo":{
    "x":1, 
    "y":2
   }, 
  "bar":[1,2,3,4,5]
}

but besides manually-edited API docs, there's no self-documentation. The rails controller often just deserializes and applies changes directly to the model. Would be nice to have common tests to tell us when it's changed, and what's expected.

Collard

Answer 1

I just started looking at this web functional testing tool called Maxq and I think it has the potential to solve your problem, Maxq acts as a proxy server between your web client and server application. It sits on top of Junit so that means you could do proper unit testing for your API by asserting the behavior and responses of calls to your server app. It basically captures and records all the requests you make from a web client and the responses you get back from a server, it also has the ability to generate test scripts of your request which you could use to play back and test on any server.

You should try it out http://maxq.tigris.org/

Electrician

This seems like what I'm looking for, but it's hard to tell. Their documentation is a bit sketchy. – Collard

Answer 2

You can think of it as two different projects.. if you had two project, you would've writing two separate test suites right?

You should start by establishing the API between the server and the client- as if you wont have any communication between the teams after you start implementhing.

Then you build the client that consume the API and a server that produce the API (or the tests first, if you TDD).

For testing, one team need a mock-server to supply fake API responses to test the client, and the other team need to test the produced data of the server (i.e, the second team is using ails integration testing like your rails guy claims)

Verbiage

Thanks for the response, but your solution results in duplication of API calls. What I mean is that the server integration testing effectively deals with their view of the API and the client testing has a different version. Keeping them consistent is proving a pain, which is why it would be nice to drive both client and server from a unified self-documenting suite. Apologies if I'm not describing this very well. – Collard

But you need a status quo between your two teams. API changes should be as minimal as possible and documented. Mocking the server for client testing might be avoidable if the client team gets its own 'stable'-server version. And yes: write some tests for the API (e.g. with QUnit) or a separate flex application, if you abstract the REST API in the client again. – Corydalis

The api is new, as is the project. So things are constantly changing. – Collard

there should be no duplication- in your exmaple, the server tests check that the given some request the server generate "bar: [1,2,3,4,5]", and in your client test you check that given "bar: [1,2,3,4,5]" the client works as it should – Verbiage

as a side note, if the api is constantly changing, it's even better to write the tests first- it's less costly to change tests then to change working code. if the working code is "so dumb" in a sense that it's just moving data- make it a common code (e.g, inherited_resources) – Verbiage

Answer 3

I would recommend Cucumber. It allows you to write specific tests for your application by emulating a browser. This way you can send requests and validate the JSON response easily.

Salcido

+1 for creativity. I can't confirm if it's exactly what I need from a quick look, but it is interesting. – Collard

It's not only creative, but it actually works. We're testing a REST API with XML and JSON this way. You just have to specify in your steps correctly. In the case of jason, parse the 'response' json to a Hash and validate that the hash contains what you find it should contain. Works really well and it exercises your entire Rails stack, so it's quite thorough test as well. – Salcido

-1 IMO this isn't what cucumber is best used for. api's aren't user facing. controller tests were designed specifically to test this purpose and the only value that cucumber ads is the step definition syntax. – Nix

Answer 4

One way you can do this is controller tests using rspec (you can also use test::unit)

describe PersonApiController 
  # GET /person/1.json
  it "should respond with a person" do 
     person = Person.create(:name => "scott")
     get :show, :id => person.id, :format => 'json'
     response.should be_success
     response.body.should have_selector('name', :content => person.name)
  end
end 

Nix