How do I test error conditions in HTML5 pages with cucumber?

Question

I am testing web application behavior with Cucumber (using Selenium and Watir under the hood). My web app has HTML5 pages and makes use of the new "required" attribute. If I have a data entry form with a required field, and I submit that form with the field blank, I would like to verify that the web app correctly responds with an error condition.

Unfortunately, when testing using an HTML5 web browser, the error message that pops up on a failed field validation does not appear to be accessible to Cucumber through the web driver. In any case, the form is not submitted and the page remains on the data entry form.

On the other hand, when testing headless or with a non-HTML5-compliant browser, the form may submit, allowing my web app to trap the error and send the user back to the form page with an error message.

In the second case, I can easily test for the existence of the error message since it's part of the HTML delivered in the page. My problem is that I can't see how to write a single test scenario that will validate the error condition for both headless and HTML5 browser situations.

It may be that this is impossible with the current state of Selenium and Watir web drivers. But if anyone has any idea how I can verify the HTML5 "required" error popup message, that would be a big help.

EDIT 2012-06-02:

Here is a sample page with a sampling of HTML5 browser warnings.

There is a required text and select, a text box showing internal hint text, and a text box with regex input validation. The page doesn't have any CSS or javascript to confuse the issue, it's just vanilla HTML5. See the w3schools page about HTML5 field attributes for a complete breakdown. The form submits to a simple CGI script that just echoes the form input, assuming the form succeeds. A submit failure will remain on the sample form page.

Dibbuk

Is there an online example that uses the 'required' attribute that we can look at? – Flytrap

I've created a small one. It's not exhaustive, but should provide enough to play around with. See EDIT to original post above. – Dibbuk

The more I think about it, testing the HTML5 error is not an issue - that would be testing whether the browser works as designed. The problem is in testing the server validation when the HTML is written with HTML5 forms. I don't want different HTML code in test environment. It would be best if Selenium and/or Watir:Webdriver could enable/disable HTML5 responses just as they can javascript. – Dibbuk

Just so you know, I wouldn't trust W3Schools w3fools.com – Berndt

Answer 1

I haven't worked with the HTML5 required attribute before. But from the looks of it, that required attribute just alerts the browser that that form field must be filled out (i.e. the DOM doesn't change).

It seems to me that it would be reasonable to just assert that that required attribute is present in the HTML of the appropriate form fields. That test should pass for BOTH HTML5 browsers and non-HTML5 browsers.

Trying to assert anything more than that seems to me like you'd be testing the functionality of the browser.

Eraste

the only thing i can do when i have to validate a element like is: p $browser.element(:css => "input:invalid").exists? p $browser.element(:css => "input:required").exists? if anyone has something better or how we can get the text of the popup html5 – Hyperopia

Answer 2

Other than validating that the HTML created is correct to enable the browser validation, I'm not sure how much you can do that doesn't amount to testing the browser and not your code.

Using watir or watir-webdriver you could use .type to validate that the input has the proper type (e.g. email) set, which is one thing that controls the browser validation. The other is the presence of the required attribute which is a little tricker Potentially .attribute_value("required") might work, but normally that returns the value of an attribute, so not sure how that method would respond to a boolean attribute. Other alternatives might be to look at .attribute_list and

Seems also like a good reason here for Watir to add a .required? method to input elements that would allow you to easily check if that attribute has been set. So I asked for that feature https://github.com/watir/watir-webdriver/issues/189

Pregnant

According to Webdriver spec boolean attribute returns true if it's value is true and nil if it's value is false – Convertite

Answer 3

You should have CSS selectors in place to target the particular field and look for an error identifier. If it is visible or not. A detailed step definition needs to be there.

Spoofery

Answer 4

One solution would be to not use Cucumber to test the error behaviour but instead test that you have configured the fields.

So in Cuke terms you might have something like

Given I am filling in my form
Then I should see that my name is required

and then write something that looks for the required option on the html tag for the name field.

Anymore than that is testing the browser not your application.

Tinney

this is pretty much the same answer as Chuck van der Linden's. Sorry about that. – Tinney