On 04/11/2013 06:37 AM, Stephen D Green wrote:

I also do a lot of work with testers writing tests for HTML pages using the
very well-known integration test tool Selenium IDE (for Firefox) and that
makes heavy use of XPath even though the target is HTML. Here we end
up with many tests which include XPaths which break every time a
significant change is made to the HTML. So we have to keep mending the
XPath expressions (even though we try to write them with //, etc, to avoid
this). So I know the problem. It doesn't stop people favouring Selenium
IDE for website integration tests though. It's well worth the hassle to many.
I guess it might be different if we were using XPath for unit tests but I
don't know about that. I guess modern unit tests get associated with TDD
and Agile, so maybe having separate test assertions (written in prose)
would be a 'no no' for many unit test writers as it might be seen as going
against Agile mantras. You'd have to update the assertions less frequently
than the unit tests but having to update something other than the tests
and the XML / code would be just too much for many Agile teams. So
if you wanted to inut test your XML you'd probably be stuck with having to
keep revising a load of XPath expressions. It goes with the territory.

The problem with Selenium tests is not the tool: it's the markup.  HTML is not semantic markup, which means: it's not meaningful.  We can extract meaning from it, but its not easy because the underlying application logic (which is usually what we want to test) is not what people are thinking about when they are crafting the markup.   They're thinking about how to get pixels to line up on four different browser platforms.

Selenium IDE is great for testing layout.  Not as good for testing behavior, but in some cases it's all we have.  We've had good success by working with the layout developers to get them to include meaningful identifiers in the markup *so that it can be tested*.

If you want to make sure the price calculation is displayed to the user properly on a checkout screen, it's much more stable and testable if there is an id="price" attribute on the price display box.  Otherwise, if you end up having to write XPath like //div[@class="blue-box"]/ul/li[3]/span to select the price, you've lost the game.

No amount of tooling support is going to allow you to write stable tests if the ground is constantly shifting underneath you.

-Mike