XPath Writing Techniques and Best Practices

      These days most of the QEs uses Selenium for automating their UI Pages. With the latest front-end technologies, there are lot of dynamic elements which makes it difficult to develop the consistent automation code. One of the best ways to handle such elements is by writing robust xpath. This article describes some basics and best practices of writing your own xpath and Xpath axes to overcome such difficulties.

Terminology used in XPath:

          XPath is a syntax for defining parts of an XML document. It uses the path expressions to navigate in XML documents. It is like a small programming language, it has functions, expressions, wild card characters etc. XML documents consist of trees of nodes. The topmost element of the tree is called the root element followed by descendant nodes.

Let’s look into the terminology used to identify these nodes: 

• Each element has one parent except root
• Each element may have zero or one or more child elements referred as child
• Nodes can have same parent called siblings. Among siblings, the nodes which are to the immediate left are referred as preceding-sibling and other left nodes as preceding. The nodes to the immediate right are referred as following-sibling and other right nodes are referred as following.
• A nodes parent, parent’s parent is referred as Ancestor
• A nodes children, children’s children is referred as Descendant

  

The below image will give more insight about the above terminology. This is a sample hierarchy in a company. Here I am considering Manager2 as node for reference.

Manager2 node has

• Manager3 as following sibling node
• Manager1 as Preceding sibling node

For Manager2,

• Director1 is the parent node
• CTO is the ancestor node (Parent of Parent).

For Manager2,

• Lead is the Child node
• QE1 and QE2 are Descendant nodes

Basic Operators in XPath:

Operator	Meaning
[]	predicate
/	starts search selection from root element in document
//	starts search selection anywhere in document
|	Used for combining more than one xpath
and	Boolean and
or	Boolean or
@	Denotes attributes
*	Represents any node of the document
contains	Used for denoting the regex expressions in Xpath

               Start-with           starting text of the attribute that is used to locate an 
                                       element       

As described above, any relative xpath would start with // and absolute xpath would start with /.

XPath Axes:

Below examples describes the usage of those XPath axe. I am using stubhub.com as a reference site.

1.    Child Axes:

     The child axis defines the children of the context node.

Syntax: child::*

     The first step is to find the root node of the context node and then forming xpath with the   help of child keyword.
     
     In the below example, I am trying to write the xpath for location text. Here location text is a <div> tag and it is under <a> tag. So, I have written xpath for <a> tag first and then used key word child to get the <div> tag.    
       

      xpath: //a[@class=’geo-display’]/child::div[@class=’geo-location-text’]

2.     Parent Axes:

    The parent axis contains only a maximum of one node. The parent node may be either the root node or an element node.

     The root node has no parent; therefore, when the context node is the root node, the parent axis is empty. For all other element nodes, the parent axis contains one node.

     If we consider the same example, I can write xpath for location <a> tag by writing xpath for

     <div > tag and then go from there by using parent key word.

     xpath: // div[@class=’geo-location-text’] /parent:: a[@class=’geo-display’]

3.    Following Axes:

     selects everything in the document after the closing tag of the current node.

     It will identify the immediate node which start after the current node.

    In the below example I have identified Help Link in the menu by identifying the logo first. Logo is <a> tag and I have written xpath for identifying it. Then I used “following” key word as the help link is the tag after the closing tag of <a>.

      Xpath: //a[@id='logo']/following::span[text()='Help']

4.     Preceding Axes:

    The preceding axis contains all nodes in the same document as the context node that are before the context node in document order.

     I will take the above example to explain this. I can use the preceding to get the logo from help menu option.

      Xpath: //span[text()='Help']/preceding::a[@id='logo']

5.     Ancestor:

     To find an element on the basis of the parent element we can use ancestor attribute of XPath. It Selects all ancestors (parent, grandparent, etc.) of the current node.

    This example is to identify the menu banner. I identified the Logo which is <a> tag. Then I used ancestor keyword as menu <div> is ancestor tag of <a>.

     Xpath: //a[@id='logo']/ancestor::div[@id='global-header-container']

6.    Descendant:

     Descendant lets you select all descendants [e.g., Children and Grandchildren] of the current node.

The Example here is to identify the “sports” link from the “find events” Menu option. I identified the “Find events” link first and then navigated to “Sports” link by using descendant keyword.

      Xpath: //li[@id='header-explore-menu']/descendant::a[@id='sports-ticket']

Best Practices of writing XPath:

         If we use relative or improper xpath, we will have several issues when UI changes. In order to avoid such problems, it’s necessary to follow some best practices in the XPath Statements before using them in Selenium Automation.

 Following are the few best practices while writing a XPath:

1. id attribute should be used if available
2. Combination of attributes such as name, type, class, value etc makes XPath more specific
   
   Eg: //span[contains(@class, 'category-ui-testing-automation') and @name='post']

3. Should be as small as possible
4. Use the Relative XPath instead of Absolute XPath Statements
5. A locator should not get effected by changes to its parents
6. Always avoid using indexes in Xpath.
7. Verify the xpath using Selenium IDE commands
8. Use XPath functions in XPath wherever necessary to better identification
9. Identification of objects with same attribute values
10. Don't identify the images using src or alt attributes
11. Handle multiple Xpaths for a single object by using ‘|’ operator

Features of Selenium 4: Chromium DevTools

    Selenium 4 alpha versions came up with much awaited native support for Chrome Dev Protocol through “DevTools” interface. This helps us getting Chrome Development properties such as Application Cache, Fetch, Network, Performance, Profiler, Resource Timing, Security and Target CDP domains etc.

    Chrome DevTools is a set of web developer tools built directly into the Google Chrome browser. DevTools can help you edit pages on-the-fly and diagnose problems quickly, which ultimately helps you build better websites, faster.

Initializing Devtools session:

You can get this devtools object from driver object. Once you receive the object, we must create a session.

@BeforeMethod

public void before(){        System.setProperty("webdriver.chrome.driver","pathoftheexefile");

    WebDriver driver=new ChromeDriver();

    driver = new ChromeDriver();

    devtools = driver.getDevTools();

    devtools.createSession();

}

Verifying the performance metrics of a page:

Most modern browsers have excellent tools that help measure the runtime performance of websites. Metrics like frame rates, paint and layout times present an aggregated state of the website’s runtime performance from these logs. Automating this process can also help us integrate these metrics into the dashboards that we use to monitor the general health of our web properties. Below code shows, how we can use devtools to get the metrics through automation.

public void getPerfMetrics(){         devTools.send(Performance.setTimeDomain(TimeDomain.ThreadTicks));

    devTools.send(enable());

    driver.get("<Ur browser url>");

    List<Metric> metrics = devTools.send(getMetrics());   

    devTools.send(disable());

}

Emulate Geo Location:

Certain applications have different features and functionalities across different locations. Automating such application is difficult using Selenium as we cannot emulate the geo location in the browser. But with the help of Devtools, we can emulate it. Below code snippet demonstrates that.

@Test

public void setGeoLocationTest(){

   driver.get("<your site url>");

   Map<String, Object> params = new HashMap<>();

   params.put("latitude", 50.2334);

   params.put("longitude", 0.2334);

   params.put("accuracy", 1);

   driver.executeCdpCommand("Emulation.setGeolocationOverride", params);

}

First, you need to find the latitude and longitude of your location. Then using executeCdpCommand, we can override the geo location.

Testing network throttling Conditions:

Testing the application in different network conditions(i.e 2G, 3G, 4G etc) is always challenging. But we can easily do it using devtools as part of our automation itself.

private static void enableNetworkOffline() {

  devTools.send(Network.enable(of(100000000), empty(), empty()));

  devTools.send(emulateNetworkConditions(true, 100, 1000, 2000,

                of(ConnectionType.cellular3g));

  driver.get(“<your browser>”);

}

The above example is to emulate 3G speed in your browser. You can also add 2G, 4G etc in the same way. We can also emulate ‘offline’ condition by just sending ‘false’ as the first parameter in emulateNetworkConditions().

Loading Insecure websites:

When you have a website which has SSL certificates, it is difficult to load and proceed further in the automation. We can use this dev tools to ignore that SSL certificate errors and load the website. Code snippet is as below.

@Test

public void loadInsecureWebsite() {

   chromeDevTools.send(Security.enable());

   chromeDevTools.send(Security.setIgnoreCertificateErrors(true));

   driver.get("https://expired.badssl.com/");

}

Verifying Console messages:

We can also verify the console logs using devtools. The below snippet verifies the given message from the console logs.

private static void consoleLogs(String message) {

   devTools.send(Console.enable());

   //add listener to verify the console message

   devTools.addListener(Console.messageAdded(),    consoleMessageFromDevTools -> assertEquals(true, consoleMessageFromDevTools.getText().equals(message)));

}

Automating email validations through Selenium and disposable email service

There are some aplications which sends Welcome emails/User registration emails to activate accounts etc. Also there are some systems which sends marketing emails. For testing them, manually verifying these emails is always tedious and time consuming. What if we have a way to automate them just through Selenium. Sounds Interesting? Lets see how it works.

We all know that Selenium can only interact with HTML elements. So, to verify the mails, we need to open our email in browser. Luckily we have a free disposable email service which opens up our mail in a browser called Endtest Mailbox. All we need to do is, use any mailid with the extension '@endtest.io'.

The username can be anything you choose.
For instance, you can choose something like ram12234@endtest.io.

You can access the Inbox for that email address in browser just by adding the email parameter in the URL, like this:
https://endtest.io/mailbox?email=ram12234@endtest.io

Please note that these emails will be deleted after 2 hours.

To demonstarte this I am using thissite . When you enter an email id and click on check, a test email is sent to the given email id immediately. Lets consider this as our application and we need to verify if it sends an email correctly to the given email address!

driver.get("http://ismyemailworking.com");
driver.manage().window().maximize();
driver.findElement(By.id("verify_email")).sendKeys("ram12234@endtest.io");
driver.findElement(By.id("content_cob_check")).click();

The above code will send a test mail to your inbox. Now we can open the inbox in browser through selenium with the link https://endtest.io/mailbox?email=ram12234@endtest.io.

driver.get("https://endtest.io/mailbox?email=klaus123@endtest.io");
driver.findElement(By.id("email_item")).click();

After this, we can add all the assertions such as Subject,  email body etc.

There are also few alternatives for Endtest mailbox such as mailnator, guerrillamail, Nada

Note:
If your emails are confidential, this solution is not appropriate. Because these free mailboxes are public and can be accessed by anybody.

In our next post, we will discuss about 'how we to verify the mails received in your Gmail Account or outlook'. 

Fake it until you make it - Stubby4j

While I was searching for a Contract Testing tool, I came across this interesting Stubby4j. Stubby4j is highly flexible and configurable tool for service virtualization or mock API server. It is actually a HTTP server that uses embedded Jetty which allows stubbing of external systems with ease for integration, contract & behavior testing.

In today's microservices era, it is very common that an API you depend on doesn't exist or isn't complete.  To stay productive in those situations, we can use these mock API servers which constantly produces the desired response. We can also use them while testing edge cases as the real APIs don't reliably produce these scenarios.

Key Features of Stubby4j:

• Emulate external webservice in a SANDBOX for your application to consume over HTTP(S)
• HTTP request verification and HTTP response stubbing
• Regex support for dynamic matching on URI, query params, headers, POST payload (ie:. mod_rewrite in Apache)
• Dynamic token replacement in stubbed response, by leveraging regex capturing groups as token values during HTTP request verification
• Record & Replay. The HTTP response is recorded on the first call, having the subsequent calls play back the recorded HTTP response, without actually connecting to the external server
• Dynamic flows. Multiple stubbed responses on the same stubbed URI to test multiple application flows
• Fault injection, where after X good responses on the same URI you get a bad one
• Serve binary files as stubbed response content (images, PDFs. etc.)
• Embed stubby4j to create a web service SANDBOX for your integration test suite

Quick Start:

1. Download the latest stubby4j version (the JAR archive).

2. Create the following local YAML file:

 -  request:
      method: GET
      url: /hello-world
 
   response:
      status: 200
      headers:
         content-type: application/json
      body: Hello World!

3. Execute the downloaded stubby JAR using command

java -jar stubby4j-x.x.xx.jar -d <PATH_TO_YOUR_CREATED_LOCAL_YAML_FILE>

4. Navigate to http://localhost:8882/hello-world to get the stubbed response “Hello World!”

5. Navigate to stubby4j admin portal at http://localhost:8889/status to see what has been stubbed & other useful data

Apart from using it is as a jar file, we can also use Stubby4j as a docker container.

Running Stubby4j as Docker Container:

Dockerfile example:

FROM <any-image-that-includes-java>
RUN wget http://central.maven.org/maven2/io/github/azagniotov/stubby4j/6.0.1/stubby4j-6.0.1.jar -O /usr/local/stubby4j.jar
EXPOSE 8889 8882
CMD java -jar /usr/local/stubby4j.jar -d no.yaml --location "0.0.0.0" --watch 

By default http://localhost:8882 is the stubs portal, http://localhost:8889 is the admin portal. They should be exposed externally.

The command for running a docker image

docker run -p 127.0.0.1:8889:8889 127.0.0.1:8882:8882 <your-image>

How QE can use Stubby4j:

• Specifiable mock responses to simulate page conditions without real data.
• Ability to test polling mechanisms by stubbing a sequence of responses for the same URI
• Easily swappable data config files to run different data sets and responses.
• All-in-one stub server to handle mock data with less need to upkeep code for test generation

There are several tools available for Service Virtualization or mock API server. But Stubby4j has the following advantages over other tools.

1. You can simulate responses from real servers
2. Simulates support for different types of HTTP authentication.
3. Enables delayed responses for performance and stability testing
4. Allows invocation of service call for services which are not ready yet or in-accessible due to various reasons.

Apart from it's advantages, stubby4j also has below limitations.

1. You can neither change the name of the mapping file nor break it in multiple files.
2. Data that you capture from request using regular expression can be used either in response mapping or response body (if response body is specified in separate file)
3. There is no way to generate dynamic dates, random numbers etc. The data which is captured from the request can only be used in response.
4. If your simulator application grows, .yaml is not maintainable

There is a tool called Stubmatic which adresses many of these limitations. 

For more information on stubby4j, please visit https://github.com/azagniotov/stubby4j

You can also look at below alternatives for Stubby4j:

• Wiremock
• Betamax
• REST-driver 
• MockServer 
• Moco 
• Mountebank  

Zalenium – A Dockerized Selenium Grid

Over the years Selenium users have used Selenium Grid extensively to reduce the execution time, to perform cross browser testing, to perform multiple OS testing. Although grid provides us all these benefits, they come with their own cost. If you look at the issues that we are facing now with grid are:

                           

• Have a stable grid to run UI tests with Selenium
• Maintain it over time (keep up with new browser, Selenium and drivers versions)
• For longer executions Java process will run out of memory.
• Restarting the nodes in case of issues
• Scaling up for more tests is tedious.

Zalenium provides the solution for above problems. It is a docker-selenium based solution which can start a Selenium Grid in seconds, a grid that scales up and down dynamically to run your tests in Firefox and Chrome. It works out of the box in docker and Kubernetes. Zalenium also provides the video recordings of your tests and live preview of your test execution.

Zalenium Setup:

1. Make sure that the latest Docker is up and running. (check here for instructions to install  and run)

2. Open command prompt/ terminal and run the below commands to pull the Docker images from Docker hub.

• docker pull elgalu/selenium
• docker pull dosel/zalenium

3. start the Zalenium-selenium-grid by executing the below command

•     Mac:

 docker run --rm -ti --name zalenium -p 4444:4444 -p 5555:5555 \  
 -v /var/run/docker.sock:/var/run/docker.sock \  
 -v /tmp/videos:/home/seluser/videos \  
 dosel/zalenium start  

•      Windows:

 docker run --rm -ti --name zalenium -p 4444:4444 -p 5555:5555 -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start  

4. Zalenium grid is ready now for the execution. Initially zalenium creates one chrome and one firefox container. If you need more containers, we have to specify them as parameters like below while launching our grid.

 docker run --rm -ti --name zalenium -p 4444:4444 -p 5555:5555 -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start --desiredContainers 4  

5. Now write a selenium test using remote webdriver. Look at the below script for reference.

 @Test  
 public void zaleniumDemoTest()  
 {  
 DesiredCapabilities capability = DesiredCapabilities.chrome();  
 WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);  
 driver.get("https://<www.yoursite.com>/");  
 driver.manage().window().maximize();  
 Thread.sleep(3000);  
 driver.quit();  
 }  

In above script, we need to specify the IP and port address of the machine where Zalenium is running as URL.

6. If you run the test now, it will execute in zalenium-grid.

Live Preview:

As discussed above, Zalenium provides live preview of your execution.  You can watch your test execution live once your tests are triggered. You can access the live preview from the below link:

http://localhost:4444/grid/admin/live#

To Auto-refresh, add ?refresh=numberOfSeconds to refresh the view automatically.

E.g. http://localhost:4444/grid/admin/live?refresh=20 will refresh the page every 20 seconds.

DashBoard:

• Zalenium also provides very useful dashboard for monitoring the test results. From this dashboard you can replay those recorded videos. Below is the link for the dashboard.

              http://localhost:4444/dashboard/#

         

• You can replace localhost for the IP/machine name where Zalenium is running

• Click on Cleanup to remove all videos and logs from the local drive and the dashboard.Also reset the dashboard via http://localhost:4444/dashboard/cleanup?action=doReset or cleanup via http://localhost:4444/dashboard/cleanup?action=doCleanup

To stop the docker containers spun through Zalenium, simply exit Zalenium grid using the below command

 docker stop zalenium.  

Integrating with Cloud Solutions:

Zalenium usually creates containers only with Chrome and firefox. If you want to execute your tests in any other browser or any other platform, you can still use other cloud based platforms such as Saucelabs, Browser stack. Zalenium enables you to integrate these platforms with the existing zalenium-grid.

Below command helps to integrate with Saucelabs:

 export SAUCE_USERNAME=[your Sauce Labs username]  
 export SAUCE_ACCESS_KEY=[your Sauce Labs access key]  
 docker run --rm -ti --name zalenium -p 4444:4444 -p 5555:5555 \  
     -e SAUCE_USERNAME -e SAUCE_ACCESS_KEY \  
     -v /tmp/videos:/home/seluser/videos \  
     -v /var/run/docker.sock:/var/run/docker.sock \  
     dosel/zalenium start --sauceLabsEnabled true  

You can visit Zalenium Site for more details.