How I automated Tableau dashboard testing

Tableau is a traditional BI tool. It is a collection of different views or visualization where each view showcases a different kind of data at the same time. It allows users to get a holistic view of all the data on one screen.

The crux of automating any Tableau dashboard test scenario is to extract data from UI and compare that with the data source (Eg: Database). Extracting data from Tableau dashboards using traditional UI automation tools such as Selenium or Cypress is very difficult because these dashboards are displayed as <canvas> elements in UI. And making them work with complex xpaths is not a feasible solution. On the other hand we have a few paid tools like Kinesis and BI Validator which can serve the purpose.

But I wanted a reliable, light weight and completely free alternative for all the above options. While searching for it, I stumbled upon Tableau’s Rest APIs. In this post we are going to discuss about extracting the data from these dashboards using Tableau’s REST APIs. These APIs give you simple access to the functionality behind Tableau data sources, projects, workbooks, site users, sites, flows,and more. In this approach we would need a little help from our Tableau development team.

Before going into the REST APIs, we should understand the internal architecture of the tableau dashboards. Each Tableau dashboard uses a workbook. Each workbook contains multiple sheets and views. A single dashboard component such as a bar chart/ pie chart is a combination of one or more views. So when you are trying to extract the data from a chart, you have to drill till the views.

Now lets get into the approach. Everything starts with the Sign-In API. Any Tableau Server REST API requires that you send a credentials token with each request. To get a token, you call Sign-In API and pass credentials of a valid user, either a Personal Access Token (PAT) or a username and password, along with the content URL (subpath) of the site you are signing in to. 

Once you have access to Tableau dashboard, you can set up your own PAT token. Check here for more details on this. Note the PAT token and substitute it in the below API.

Endpoint:

POST http://<my-server>/api/3.15/auth/signin

Request Body:	

{
  "credentials": {
	"personalAccessTokenName": "<MY_PAT_TOKEN_NAME>",
	"personalAccessTokenSecret": "<MY_PAT_TOKEN_SECRET>",
	"site": {
		"contentUrl": "<CONTENT_URL>"
	}
   }
}

Parameters:

MY_PAT_TOKEN_NAME,MY_PAT_TOKEN_SECRET are the PAT token details that you create in the Tableau dashboard UI. By default CONTENT_URL is empty string ("") unless your Tableau team customizes the URL.

If you execute the API with all the required details, you will receive the response like below:

{
    "credentials": {
        "site": {
            "id": "7668ddfd-sdfds-sdfds87-sdfdsds",
            "contentUrl": "https://mydhashboatd.com"
        },
        "user": {
            "id": "6746-sddf78sdfds-sad5763"
        },
        "token": "hjhjhdDSDghGHGghGH233jHhhjHJKKJ",
        "estimatedTimeToExpiration": "234:18:27"
    }
}

From the above response, we need site id, token for all the subsequent APIs.

At this point, you will need some information from your Tableau developers such as workbook_id. From the workbook_id, you can get all the views that are available using the below API. Make sure you pass in header “X-Tableau-Auth” with the token that received from above request.

GET:
http://my-server/api/<api-version>/sites/<site-id>/workbooks/<workbook_id>/views

Parameter Values:

api-version The version of the API to use, such as 3.15. 

site-id The ID of the site that we noted above.

Workbook_id The ID that you receive from your development team.

Request Body

None

Once you receive all the views, you need to work with your Tableau development team to identify the relation/mapping between each view and the corresponding dashboard component. For example a single pie chart may have more than one view. 

After identifying the views, you can get the data from each view using the below API. Make sure you pass in header “X-Tableau-Auth” with the token.

GET     http://my-server/api/<api-version>/sites/<site-id>/views/<view-id>/data

Parameter Values

api-version The version of the API to use, such as 3.15. 

site-id The ID of the site that contains the group.

view-id The ID of the view whose details are requested.

Request Body

None

Header

X-Tableau-Auth

This way you can easily retrieve the data from each dashboard view. This is just a single approach. But these REST APIs provide multiple options for us to receive the data otherwise could be very difficult to extract via UI. 

Accessibility scanning using Selenium in C#

While searching for the axe-core engine integration with Selenium C#, I found this very useful library. It is very easy to use as well. You just have to add this nuget package into your project and you will have variety of options to scan the page in a single line of code. The most useful option is, generating the scanned a11y voilation results as email-able html reports. Below is the sample code to scan the page and generate the results as html reports in your desired path.

IWebDriver webDriver = new ChromeDriver();
//navigate to the page to scan
HtmlReport.CreateAxeHtmlReport(webDriver, path);

For more details check the below documentation:

https://troywalshprof.github.io/SeleniumAxeDotnet/#/?id=getting-started

Pairwise Testing Technique overview with implementation using PICT tool

Pairwise testing is a Combinatorial Testing technique also known as ‘All-Pairs Testing’.  While testing a system with multiple combinations, it is very hard to test all the combinations. Pairwise testing is a permutations and combinations based method, in which to test a system or an application, for each pair of input parameters of a system, all possible discrete combinations of the parameters are tested. This is a smart way of reducing the test combinations thus reducing the testing time without compromising the test coverage.

Why do we need it?

Lets take an example of Marketing email delivery API of a e-learning company. Consider that API takes Age, Gender, Geography and Last Purchased Course Category of user as parameters and decides the email content with Course suggestions etc. to the user.

Let’s list down the variables involved here:

1.  Age

     i)    10-20

     ii)   20-30

     iii)  30-40

     iv)  40-50

2.  Gender

     i)   Male

     ii)  Female

3.  Geography

     i)    EMEA

     ii)   NA

     iii)  LATAM

     iv)  APAC

4.  Last Purchased Course Category

     i)    Marketing.

    ii)    IT and Software.

    iii)   Personal Development.

    iv)   Business.

    v)    Music.

Now to test this API by using the conventional or exhaustive testing approach, we will have

 = 4*2*4*5 = 160 combinations.

In the current agile environment, it is hard for the testers to test all these combinations due to the tight timelines. Pairwise testing technique will help us significantly reducing these combinations in such a way that, we will have 100% coverage and high defect yield rate.

The process of generating these combinations manually through Pairwise testing technique involves placing each catogery in such a way that each combination is repeated once. There are lot of resources available to explain this process in internet. 

In this post, we will focus on generating these combinations automatically with the help of PICT tool.

we have few tools which can help us do this automatically. 

Tools available:

• PICT – ‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp.
• IBM FoCuS – ‘Functional Coverage Unified Solution’, provided by IBM.
• ACTS – ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of the US Government.
• Hexawise
• Jenny
• Pairwise by Inductive AS
• VPTag free All-Pair Testing Tool

Check here for more tools.

Using PICT for Pairwise Testing:

Among all the tools available, PICT is the widely used open source tool for generating combinations through Pairwise technique. But it is available only in windows.

Let’s start generating the combinations for the above example using PICT.

First we need to install it. You can download the PICT from here. After downloading this .exe file, just click on it, and installation steps are self-explanatory.

To confirm the installation is successful, open Command prompt and type ‘pict’. You should see below text:

The next step is, we need to create an input model in .txt file. PICT uses this model as an input for the combinations.

The input file will look like this.

We use colon(:) to separate parameters and their real values. All the parameter values are split by comma(,).

Now open your command prompt, navigate to the folder where PictExample.txt file is located using: cd path_to_PictExample.txt

Execute the following command to get the combinations: pict PictExample.txt

The result is:

We can redirect these combinations into excel sheet by using pict PictExample.txt > PictExample-combinations.xls

So, now you can see that, 160 combinations got reduced to 22 using Pairwise testing technique. You can also observe that we created maximum test coverage through these combinations.

Excluding the conflict combinations:

Consider that Music catogery is not available in NA region. But PICT generated that combination as well. We can eliminate that combination for future generations by excluding this combination in the input file itself like below:

Note: PICT is case sensitive about parameters and options. The exclusion condition should end with semicolon (;)

If you execute pict PictExample.txt command now, you will see the below output by removing the invalid combination.

Advantages:

• Pairwise testing reduces the number of test cases. So, less time to complete the execution thus saving the time.
• Pairwise testing increases the test coverage almost up to hundred percentage.
• Pairwise testing increases the defect detection ratio.

Disadvantages:

• Pairwise testing is not beneficial if the values of the variables are inappropriate.
• In pairwise testing it is possible to miss a highly probable combination while selecting the test data.
• In pairwise testing, defect yield ratio may be reduced if a combination is missed.
• Pairwise testing is not useful if the combination of variables are not understood correctly.

XPath Writing Techniques and Best Practices

      These days most of the QEs uses Selenium for automating their UI Pages. With the latest front-end technologies, there are lot of dynamic elements which makes it difficult to develop the consistent automation code. One of the best ways to handle such elements is by writing robust xpath. This article describes some basics and best practices of writing your own xpath and Xpath axes to overcome such difficulties.

Terminology used in XPath:

          XPath is a syntax for defining parts of an XML document. It uses the path expressions to navigate in XML documents. It is like a small programming language, it has functions, expressions, wild card characters etc. XML documents consist of trees of nodes. The topmost element of the tree is called the root element followed by descendant nodes.

Let’s look into the terminology used to identify these nodes: 

• Each element has one parent except root
• Each element may have zero or one or more child elements referred as child
• Nodes can have same parent called siblings. Among siblings, the nodes which are to the immediate left are referred as preceding-sibling and other left nodes as preceding. The nodes to the immediate right are referred as following-sibling and other right nodes are referred as following.
• A nodes parent, parent’s parent is referred as Ancestor
• A nodes children, children’s children is referred as Descendant

  

The below image will give more insight about the above terminology. This is a sample hierarchy in a company. Here I am considering Manager2 as node for reference.

Manager2 node has

• Manager3 as following sibling node
• Manager1 as Preceding sibling node

For Manager2,

• Director1 is the parent node
• CTO is the ancestor node (Parent of Parent).

For Manager2,

• Lead is the Child node
• QE1 and QE2 are Descendant nodes

Basic Operators in XPath:

Operator	Meaning
[]	predicate
/	starts search selection from root element in document
//	starts search selection anywhere in document
|	Used for combining more than one xpath
and	Boolean and
or	Boolean or
@	Denotes attributes
*	Represents any node of the document
contains	Used for denoting the regex expressions in Xpath

               Start-with           starting text of the attribute that is used to locate an 
                                       element       

As described above, any relative xpath would start with // and absolute xpath would start with /.

XPath Axes:

Below examples describes the usage of those XPath axe. I am using stubhub.com as a reference site.

1.    Child Axes:

     The child axis defines the children of the context node.

Syntax: child::*

     The first step is to find the root node of the context node and then forming xpath with the   help of child keyword.
     
     In the below example, I am trying to write the xpath for location text. Here location text is a <div> tag and it is under <a> tag. So, I have written xpath for <a> tag first and then used key word child to get the <div> tag.    
       

      xpath: //a[@class=’geo-display’]/child::div[@class=’geo-location-text’]

2.     Parent Axes:

    The parent axis contains only a maximum of one node. The parent node may be either the root node or an element node.

     The root node has no parent; therefore, when the context node is the root node, the parent axis is empty. For all other element nodes, the parent axis contains one node.

     If we consider the same example, I can write xpath for location <a> tag by writing xpath for

     <div > tag and then go from there by using parent key word.

     xpath: // div[@class=’geo-location-text’] /parent:: a[@class=’geo-display’]

3.    Following Axes:

     selects everything in the document after the closing tag of the current node.

     It will identify the immediate node which start after the current node.

    In the below example I have identified Help Link in the menu by identifying the logo first. Logo is <a> tag and I have written xpath for identifying it. Then I used “following” key word as the help link is the tag after the closing tag of <a>.

      Xpath: //a[@id='logo']/following::span[text()='Help']

4.     Preceding Axes:

    The preceding axis contains all nodes in the same document as the context node that are before the context node in document order.

     I will take the above example to explain this. I can use the preceding to get the logo from help menu option.

      Xpath: //span[text()='Help']/preceding::a[@id='logo']

5.     Ancestor:

     To find an element on the basis of the parent element we can use ancestor attribute of XPath. It Selects all ancestors (parent, grandparent, etc.) of the current node.

    This example is to identify the menu banner. I identified the Logo which is <a> tag. Then I used ancestor keyword as menu <div> is ancestor tag of <a>.

     Xpath: //a[@id='logo']/ancestor::div[@id='global-header-container']

6.    Descendant:

     Descendant lets you select all descendants [e.g., Children and Grandchildren] of the current node.

The Example here is to identify the “sports” link from the “find events” Menu option. I identified the “Find events” link first and then navigated to “Sports” link by using descendant keyword.

      Xpath: //li[@id='header-explore-menu']/descendant::a[@id='sports-ticket']

Best Practices of writing XPath:

         If we use relative or improper xpath, we will have several issues when UI changes. In order to avoid such problems, it’s necessary to follow some best practices in the XPath Statements before using them in Selenium Automation.

 Following are the few best practices while writing a XPath:

1. id attribute should be used if available
2. Combination of attributes such as name, type, class, value etc makes XPath more specific
   
   Eg: //span[contains(@class, 'category-ui-testing-automation') and @name='post']

3. Should be as small as possible
4. Use the Relative XPath instead of Absolute XPath Statements
5. A locator should not get effected by changes to its parents
6. Always avoid using indexes in Xpath.
7. Verify the xpath using Selenium IDE commands
8. Use XPath functions in XPath wherever necessary to better identification
9. Identification of objects with same attribute values
10. Don't identify the images using src or alt attributes
11. Handle multiple Xpaths for a single object by using ‘|’ operator

Features of Selenium 4: Chromium DevTools

    Selenium 4 alpha versions came up with much awaited native support for Chrome Dev Protocol through “DevTools” interface. This helps us getting Chrome Development properties such as Application Cache, Fetch, Network, Performance, Profiler, Resource Timing, Security and Target CDP domains etc.

    Chrome DevTools is a set of web developer tools built directly into the Google Chrome browser. DevTools can help you edit pages on-the-fly and diagnose problems quickly, which ultimately helps you build better websites, faster.

Initializing Devtools session:

You can get this devtools object from driver object. Once you receive the object, we must create a session.

@BeforeMethod

public void before(){        System.setProperty("webdriver.chrome.driver","pathoftheexefile");

    WebDriver driver=new ChromeDriver();

    driver = new ChromeDriver();

    devtools = driver.getDevTools();

    devtools.createSession();

}

Verifying the performance metrics of a page:

Most modern browsers have excellent tools that help measure the runtime performance of websites. Metrics like frame rates, paint and layout times present an aggregated state of the website’s runtime performance from these logs. Automating this process can also help us integrate these metrics into the dashboards that we use to monitor the general health of our web properties. Below code shows, how we can use devtools to get the metrics through automation.

public void getPerfMetrics(){         devTools.send(Performance.setTimeDomain(TimeDomain.ThreadTicks));

    devTools.send(enable());

    driver.get("<Ur browser url>");

    List<Metric> metrics = devTools.send(getMetrics());   

    devTools.send(disable());

}

Emulate Geo Location:

Certain applications have different features and functionalities across different locations. Automating such application is difficult using Selenium as we cannot emulate the geo location in the browser. But with the help of Devtools, we can emulate it. Below code snippet demonstrates that.

@Test

public void setGeoLocationTest(){

   driver.get("<your site url>");

   Map<String, Object> params = new HashMap<>();

   params.put("latitude", 50.2334);

   params.put("longitude", 0.2334);

   params.put("accuracy", 1);

   driver.executeCdpCommand("Emulation.setGeolocationOverride", params);

}

First, you need to find the latitude and longitude of your location. Then using executeCdpCommand, we can override the geo location.

Testing network throttling Conditions:

Testing the application in different network conditions(i.e 2G, 3G, 4G etc) is always challenging. But we can easily do it using devtools as part of our automation itself.

private static void enableNetworkOffline() {

  devTools.send(Network.enable(of(100000000), empty(), empty()));

  devTools.send(emulateNetworkConditions(true, 100, 1000, 2000,

                of(ConnectionType.cellular3g));

  driver.get(“<your browser>”);

}

The above example is to emulate 3G speed in your browser. You can also add 2G, 4G etc in the same way. We can also emulate ‘offline’ condition by just sending ‘false’ as the first parameter in emulateNetworkConditions().

Loading Insecure websites:

When you have a website which has SSL certificates, it is difficult to load and proceed further in the automation. We can use this dev tools to ignore that SSL certificate errors and load the website. Code snippet is as below.

@Test

public void loadInsecureWebsite() {

   chromeDevTools.send(Security.enable());

   chromeDevTools.send(Security.setIgnoreCertificateErrors(true));

   driver.get("https://expired.badssl.com/");

}

Verifying Console messages:

We can also verify the console logs using devtools. The below snippet verifies the given message from the console logs.

private static void consoleLogs(String message) {

   devTools.send(Console.enable());

   //add listener to verify the console message

   devTools.addListener(Console.messageAdded(),    consoleMessageFromDevTools -> assertEquals(true, consoleMessageFromDevTools.getText().equals(message)));

}