Compliance test for OpenID providers

Question

What automated standards-conformance tests are there for OpenID providers?

I'm making changes to the implementation of an OpenID provider, to bring it from version 1.1 of the standard to version 2.0.

Before releasing the code, I want to be sure that it conforms to the specifications of the standard. For testing web standards compliance, the W3C has validator tools. What tools exist so that I can point an automated tester at my OpenID provider and get a report of compliance with the standard?

Markman

the hardest part to automate would be the expiration checks and the fact you need a browser input to test so you need to write a selenium script to do the login process. I sort of did one with Heroku but that was to test my OpenID client. You can probably retrofit it to do your own testing. – Pooka

Answer 1

In case anyone stumbles across this thread, OpenID Connect now has an official conformance test suite that is part of the certification process:

https://openid.net/certification/testing/

Enjoy!

Endgame

Answer 2

You could look at http://test-id.net/ which has a set of tests written in .net.

Pitchdark

This is an OSIS-sponsored set of interoperability, security, and spec implementation tests that work against any RP or OP -- regardless of whether it's built on .NET or another platform. – Therapist

Answer 3

There are no conformance tests (at least officially approved) that I know of - even for 1.1. Certainly its something that would be very high value. Same goes for oAuth - they're both complex protocols and sometimes event the spec doesn't cover everything.

Probably the only thing you can do right now is thorough unit testing coverage locally.

Vertebra

This isn't accurate [any more]. test-id.org has some tests sponsored by OSIS and the OpenID board. – Therapist

@Andrew and now test-id.org returns "service unavailable" – Chamorro

Answer 4

There's this thing for 1.1-only: http://openidenabled.com/resources/openid-test/diagnose-server/

We never upgraded it for 2.0. Once or twice a year someone comes along and says "hey, we should have better testing tools," but as far as I (and others, judging from the responses here) know, none of those efforts has bore fruit yet.

Edited to add: another related project is at http://code.google.com/p/openid-test/

Metanephros

Looks good. Is it free software? If so, where is the source code? – Markman

I think we released the code at some point, but I'm not sure where it is now. And, frankly, while I usually discourage people from starting from scratch, it might be a good idea in this case. Part of the reason that server never got upgraded was because it relied heavily on internal implementation details of the v1 python OpenID library, and so it wasn't maintainable as the library evolved and upgraded. – Metanephros

Answer 5

OSIS have interoperability tests, feature tests for IPs and feature tests for RPs

However these aren't automated, everyone gets together at one of the RSA conferences and checks they all work with each other.

Wheelock

Thanks. It's only a report of existing tests, and it doesn't give any indication of how someone developing an OpenID provider could apply the same tests. – Markman

Answer 6

As of now, and to the best of my knowledge there are no tools that can report compliance with standard. If you really want to make sure that your code is 2.0 compliant, you should hire some independent consultants to review your unit tests for each function of openID 2.0. They should also do thier own testing of course. The consultants must be experienced with auditing in general like PCI DSS, etc. They are experienced to go over the spec and test your app libraries and database.

Bucksaw

Your advice seems more suitable if my question was about something like ISO 9001 compliance, but it's not. An open, fairly-brief standard specification of a machine-to-machine protocol, like OpenID, is a target for automated tests, like the W3C Validator tools. – Markman