page object model: why not include assertions in page methods?

Question

First-time poster. I've been working in UI automation for many years, but was only recently introduced to/instructed to work with the Page Object Model. Most of it is common sense and includes techniques I've been using already, but there's a particular fine point which I haven't been able to justify in my own mind, despite searching extensively for a well-reasoned explanation. I'm hoping someone here might enlighten me, as this question has caused some consternation as I try to integrate the POM with my own best practices.

From http://code.google.com/p/selenium/wiki/PageObjects:

The code presented above shows an important point: the tests, not the PageObjects, should be responsible for making assertions about the state of a page.... Of course, as with every guideline there are exceptions...

From http://seleniumhq.org/docs/06_test_design_considerations.html#chapter06-reference:

There is a lot of flexibility in how the page objects may be designed, but there are a few basic rules for getting the desired maintainability of your test code. Page objects themselves should never be make verifications or assertions. This is part of your test and should always be within the test’s code, never in an page object. The page object will contain the representation of the page, and the services the page provides via methods but no code related to what is being tested should be within the page object.

There is one, single, verification which can, and should, be within the page object and that is to verify that the page, and possibly critical elements on the page, were loaded correctly. This verification should be done while instantiating the page object.

Both of these "guidelines" allow for potential exceptions, but I couldn't disagree more with the basic premise. I'm accustomed to doing a considerable amount of verification within "page methods", and I think the presence of verification there is a powerful technique for finding issues in a variety of contexts (i.e., verification occurs every time the method is called) rather than only occurring in the limited context of particular tests.

For example, let's imagine that when you login to your AUT, some text appears that says "logged in as USER". It's appropriate to have a single test validate this specifically, but why wouldn't you want to verify it every time login is called? This artifact is not directly related to whether the page "loaded correctly" or not, and it's not related to "what is being tested" in general, so according to the POM guidelines above, it clearly SHOULDN'T be in a page method... but it seems to me that it clearly SHOULD be there, to maximize the power of automation by verifying important artifacts as often as possible, with as little forethought as possible. Putting verification code in page methods multiplies the power of automation by allowing you to get a lot of verification "for free", without having to worry about it in your tests, and such frequent verification in different contexts often finds issues which you would NOT find if the verification were limited to, say, a single test for that artifact.

In other words, I tend to distinguish between test-specific verification and "general" verification, and I think it's perfectly appropriate/desirable for the latter to be included - extensively - in page methods. This promotes thinner tests and thicker page objects, which generally increases test maintainability by reusing more code - despite the opposite contention in these guidelines. Am I missing the point? What's the real rationale for NOT wanting verification in page methods? Is the situation I've described actually one of the 'exceptions' described in these guidelines, and therefore actually NOT inconsistent with the POM? Thanks in advance for your thoughts. -jn-

Sinistrorse

Answer 1

As a guideline, assertions should be done in tests and not in page objects. Of course, there are times when this isn't a pragmatic approach, but those times are infrequent enough for the above guideline to be right. Here are the reasons why I dislike having assertions in page objects:

1. It is quite frustrating to read a test that just calls verify methods where assertions are buried elsewhere in page objects. Where possible, it should be obvious what a test is asserting; this is best achieved when assertions are directly in a test. By hiding the assertions somewhere outside of a test, the intent of the test is not so clear.

2. Assertions in browser tests can be expensive - they can really slow your tests down. When you have hundreds or thousands of tests, minutes/hours can be added to your test execution time; this is A Bad Thing. If you move the assertions to just the tests that care about those particular assertions you'll find that you'll have much quicker tests and you will still catch the relevant defects. The question included the following:

   Putting verification code in page methods multiplies the power of automation by allow you to get a lot of verification "for free"

   Well, "Freedom Isn't Free" :) What you're actually multiplying is your test execution time.

3. Having assertions all over the place violates another good guideline; "One Assertion Per Test" ( http://blog.jayfields.com/2007/06/testing-one-assertion-per-test.html ). I don't stick religiously to it, but I try to follow the principle. Where possible, a test should be interested in one thing only.

4. The value of tests is reduced because one bug will cause loads of tests to fail thus preventing them from testing what they should be testing.

   For example, let's imagine that when you login to your AUT, some text appears that says "logged in as USER". It's appropriate to have a single test validate this specifically, but why wouldn't you want to verify it every time login is called?

   If you have the assertion in the page object class and the expected text changes, all tests that log in will fail. If instead the assertion is in the test then only one test will fail - the one that specifically tests for the correct message - leaving all the other tests to continue running to find other bugs. You don't need 5,000 tests to tell you that the login message is wrong; 1 test will do ;)

5. Having a class do more than one thing violates 'S' in SOLID, ie: 'Single Responsibility Principle' (SRP). A class should be responsible for one thing, and one thing only. In this instance a page-object class should be responsible for modelling a page (or section thereof) and nothing more. If it does any more than that (eg: including assertions) then you're violating SRP.

Lithesome

What you're actually multiplying is your test execution time.. Let assume that page load time = 2 seconds. One basic assertion is about 10 ms (in Firefox). Doing one assertion after each page load adds less than 1% to test execution time and more than 1% to my ability to quickly find out bugs. My opinion is that tests should fail fast – Ritualize

One assertion per test is good for unit testing but not so much for system testing. As I said in previous comment test action (e.g. page load) takes 2 seconds, assertion takes 10 ms. It's MUCH faster to do N assertions after test action than to write N tests and make 1 assertion after each of them – Ritualize

Regarding point 5: don't forget about youtube.com/… – Ritualize

A Youtube video is not evidence of anything except yours and the speaker's idiosyncratic opinion. Of course you should write classes in a SOLID manner for your test code, especially if your test code is complicated, just as you would for your production code. Code is code. – Maxantia

Of course you should keep to one assertion per test, otherwise tests will fail for ambiguous reasons which will waste further time trying to figure out exactly how a test failed. – Maxantia

@RicardoGladwell I think assertion methods should be wrtten inside page objects. Thus this assertion method can be invoked from different pieces of code. – Ritualize

@RicardoGladwell one assertion per test is good for unit tests. But I think it's not good for browser tests where page load time = 2 seconds and one assertion = 50 ms. – Ritualize

So... 10ms or 50 ms for an assertion? It makes a difference when you're running that assertion thousands of times when you only need to run it once :P – Lithesome

Where do you put assertion method if you want to use it multiple times and this method is not a one-liner? – Ritualize

I would start by questioning why I want to assert the same thing more than once... – Lithesome

Answer 2

I too have struggled at times with this recommendation. I believe the reason behind this guideline is to keep your page objects reusable, and putting asserts inside your page objects could possibly limit their ability to be reused by a large number of unrelated tests. That said, I have put certain verification methods on my page objects like testing the caption for a header - in my experience, that is a better way to encapsulate test logic for elements of a page that don't change.

Another note - I have seen MVC applications that have domain models reused as page objects. When done correctly, this can significantly reduce redundant code in your testing library. With this pattern, the view models have no reference to a testing framework, so obviously, you could not put any asserts in them.

Dudden

>putting asserts inside your page objects could possibly limit their ability to be reused by a large number of unrelated tests – Sinistrorse

Thanks Matt. That makes sense, but I can design my page methods to be widely reusable AND do some useful verification at the same time - in fact that's my goal. This seems like a weak reason to strongly recommend that it be avoided... I wonder if anyone out there can provide a compelling argument for avoidance? – Sinistrorse

I don't think there are any hard rules you need to follow when creating page objects. The goal is to make things reusable, and if you can do that then you are on the right track. – Dudden

I agree. Unfortunately, the POM seems to be viewed with a certain reverence by my colleagues, and I've effectively been asked to stop 'blaspheming it' and just follow it for the time being, because whoever thought it up probably understood it better than I do. In my view, the widespread acceptance of the POM as 'gospel', including guidelines like this one which have no obvious foundation, is leading me to write suboptimal code. I've tried & failed to make this point with my colleagues... I just don't buy the dogma, and I hesitate to accept recommendations without a solid rationale. – Sinistrorse

I agree with you. We can have validations at page level. But when you say "That makes sense, but I can design my page methods to be widely reusable AND do some useful verification at the same time" - I think it depends on the AUT and design. I don't want my EndToEnd test failing without doing majority of steps just because one text is missing after logging in.. – Supporting

It w/b interesting to know how closely folks follow the POM in general; I suspect it's targeted toward less-experienced testers and perhaps applies less to experienced testers who can manage their own risks. I've heard numerous people talk about how it's a well-understood and accepted model, and particularly parroting this bit about where verification does/doesn't belong, but my sense is that they just haven't thought outside the box in this regard. – Sinistrorse

AJ: agreed; one should generally use verifications rather than assertions to prevent unwarranted test failure - regardless of AUT and design, I'd suggest. The guideline, however, clearly states that "Page objects themselves should never be make verifications or assertions." Would the ideal guideline omit the restriction on verifications but emphasize the restriction on assertions, or is there something more fundamental behind the restriction? I know it's not intended as a restriction so much as a guideline, but in my case it actually is. – Sinistrorse

@Jon Hmm.. I see your point. For me its just a guideline and not a hard rule. I can't see why people ask you follow that literally. I would suggest you put this question across to the selenium group in #selenium IRC channel. Who knows, may be they will correct it in wiki :) – Supporting

@AJ: That sounds like great advice, but I'm not familiar with IRC channels... could you be more specific? I tried a few IRC search sites and they couldn't find that channel. Thanks! – Sinistrorse

@JonNelson You can either post it in selenium-developer google groups or connect to webchat.freenode.net . Give yourself a nickname and type #selenium in channel and connect. You can post your question or give a link to this question there.. – Supporting

Answer 3

Your page object shouldn't perform an assertion because then the page object has to know about your test framework (unless you're using built-in language assertions). But your page needs to know it's state to locate elements and perform actions.

The key is in the statement "Of course, as with every guideline there are exceptions..."

Your page should throw exceptions, not perform assertions. That way your test can catch the assertion and bail or act accordingly. For instance.

page = ProfilePage.open
try 
  page.ChangePassword(old, new)
catch notLoggedIn
  page.Login(user, pass)

assert page.contains "your password has been updated"

In this limited example you'd have to check again (and again) so it might not be the best way, but you get the idea. You could also just check state (twice)

if page.hasLoginDialog
  page.Login

if page.hasLoginDialog //(again!)
  assert.fail("can't login")

You could also just check that you have a profile page

try 
  page = site.OpenProfilePage
catch notOnProfilePage

or has the elements you need try profilepage.changePassword(old,new) catch elementNotFound

or without throwing an exception

page = site.OpenProfilePage
if ! page instanceof ProfilePage

or with complex checking

assert page.looksLikeAProfilePage

It's not how you do it that matters. You want to keep logic in your tests to a minimum but you don't want your page objects to be tied to your test framework -- after all, you might use the same objects for scraping or data generation -- or with a different test framework that has it's own assertions.

If you feel a need you can push your assertions out of your test case into test helper methods.

page = site.GoToProfilePage
validate.looksLikeProfilePage(page)

which a great opportuntity for a mixin if your language supports them, so you can have your clean page objects -- and mixin your sanity checks.

Bakki

Answer 4

This perplexes me when I see same assertion could be used across multiple test methods. For example, writing assertion specific method -

public PaymentPage verifyOrderAmount(BigDecimal orderAmount) {      
   Assertion.assertEquals(lblOrderAmount.getText(), orderAmount, "Order Amount is wrong on Payment details page"); 
   return this; 
}

Now I can reuse it in all tests I need. Instead of repeating same assertion statement in multiple tests dealing with multiple scenarios. Needless to say I can chain multiple assertions in a method depending on test i.e -

 .verifyOrderAmount(itemPrice)
 .verifyBankAmount(discountedItemPrice)
 .verifyCouponCode(flatDiscountCode.getCouponCode()) 

When page object is supposed to represent the services offered by page then, is not assertion point also a service provided by Page?

Surplusage

You have a good example of a reusable validation assertion. However, it would go better with our test helpers, which allow us to reuse handy methods from the actual testing side of things. – Arreola

@bgoad I so very wish if you could elaborate on your comment. – Surplusage

please see https://mcmap.net/q/391839/-unit-test-helper-methods for some suggestions on how to use test helpers in Java. We use them extensively in Ruby in our unit tests. – Arreola

You could design custom Hamcrest matchers to be able to reuse same checks accross different pages but still have checking separated from PageObject code. – Geist

@Surplusage - What are you checking in multiple pages which requires your PageObjects to have some assertions ? Have you thought of alternate approaches to avoid it ? – Jorie

Answer 5

@Matt reusing domain models in page object might save you time but isn't that a Test Smell, test logic be well clear of domain model (depending on what you are trying to achieve).

Back to the Original Question, If you really must do assertions in the Page Object, why not use selenium loadablecomponent<> where you can use the isLoaded() method or include your custom assertion in the loadablecomponent<> class. This will keep your page object free of assertions.But you get to do assertions in the loadable component. See link below...

https://github.com/SeleniumHQ/selenium/wiki/LoadableComponent

_The Dreamer

Upthrow

Answer 6

I couldn't agree more with the author.

Adding assertions in test methods helps you to 'fail early'. By assertions, I mean checking if a certain page is loaded after clicking button, etc. (the so called general assertions).

I really don't believe it increases the execution time so much. UI automation is slow by default, adding some few-millisecond checks do not make that much of a difference, but could ease your troubleshooting, report an early failure and make your code more re-usable.

However, it also depends on the type of UI tests. For instance, if you are implementing end-to-end tests with mostly positive paths, it makes sense to make a check inside the test method that clicking a button actually results in opening a page. However, if you are writing bunch of negative scenarios, that's not always the case.

Graaf