TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use. TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration.

Introduction of TestNG

TestNG, where NG stands for “next generation” is a test automation framework inspired by JUnit (in Java) and NUnit (in C#)

A few of the features that TestNG has over JUnit 4 are:

  • Extra Before and After annotations such as Before/After Suite and Before/After Group
  • Dependency test
  • Grouping of test methods
  • Multithreaded execution
  • In-built reporting framework

Advantages of TestNG

Now let’s discover more features/advantages offered by TestNG.

  1. Multiple Before and After annotation options
  2. XML-based test configuration and test suite definition
  3. Dependent methods
  4. Groups/group of groups
  5. Dependent groups
  6. Parameterization of test methods
  7. Data-driven testing
  8. Multithreaded execution
  9. Better reporting

We will discuss these features in more detail in coming tutorials.

Installing TestNG onto Eclipse

Before we can download and start using TestNG, make sure you have Java JDK5 or above is installed on your system. Also make sure that JDK is set in the system path.

In case you just want to download the TestNG JAR, you can get it from the following URL:

TestNG Jar – http://testng.org/testng-6.8.zip

Now, let’s start with the installation process of TestNG onto Eclipse. I will try to capture all steps in the process.

1) Open your Eclipse application.

2) Go to Help | Install New Software.

Eclipse - Go to Help - Install New Software

3) Click on the Add… button next to the Work with text box.

Click on the Add

4) Enter TestNG site into the Name box and enter URL http://beust.com/eclipse into the Location box. Once done, click on the OK button.

Enter TestNG site into the Name box

5) On clicking OK, TestNG update site will get added to Eclipse. The available software window will show the tools available to download under the TestNG site.

Select TestNG and click on Next

6) Select TestNG and click on Next.

7) Eclipse will calculate the software requirements to download the selected TestNG plugin and will show the Install Details screen. Click on Next on the details screen.

8) Accept the License Information and click on Finish. This will start the download and installation of the TestNG plugin onto Eclipse.

download and installation of the TestNG

9)In case you get the following warning window, click on the OK button.

TestNG warning window

10) Once the installation is complete, Eclipse will prompt you to restart it. Click on Yes on the window prompt.

11) Once Eclipse is restarted, verify the TestNG plugin installation by going to Window | Preferences. You will see a TestNG section under the preferences window.

Verify the TestNG plugin installation

We have successfully installed the TestNG plugin into our Eclipse installation. This will help us in executing our TestNG tests or suite using Eclipse.

TestNG Annotations:

In this post, we see the list of TestNG Annotations. You could learn complete TestNG Tutorial here. Here is a quick overview of the annotations available in TestNG.

@Test: Marks a class or a method as a part of the test.

@BeforeMethod: A method which is marked with this annotation will be executed before every @test annotated method.

@AfterMethod: A method which is marked with this annotation will be executed after every @test annotated method.

@BeforeClass: A method which is marked with this annotation will be executed before first @Test method execution. It runs only once per class.

@AfterClass: A method which is marked with this annotation will be executed after all the test methods in the current class have been run

@BeforeTest: A method which is marked with this annotation will be executed before first @Test annotated method.

@AfterTest: A method which is marked with this annotation will be executed when all @Test annotated methods complete the execution of those classes which are inside <test> tag in testng.xml file.

@BeforeSuite: A method which is marked with this annotation will run only once before all tests in the suite have run

@AfterSuite: A method which is marked with this annotation will run once after execution of all tests in the suite have run

@BeforeGroups: This annotated method will run before the first test run of that specific group.

@AfterGroups: This annotated method will run after all test methods of that group completes its execution.

Some other TestNG Annotations, we need to discuss here are mentioned below:

@Parameters: This annotation is used to pass parameters to test methods.

@DataProvider: If we use @DataProvider annotation for any method that means you are using that method as a data supplier. The configuration of @DataProvider annotated method must be like it always return Object[][] which we can use in @Test annotated method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.

@Factory: Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[ ].

@Listeners: This annotation is used with test class. It helps in writing logs and results.

How To Ignore TestNG Test | TestNG Tutorial

At times we may face some situations where our code is not ready and the test cases written to test a particular method may fail. In such cases, we could use TestNG annotation @Test(enabled = false). This TestNG annotation allows us to ignore testng test.

If a test method is annotated with @Test(enabled = false), then the test case that is not ready to test is ignored.

Now, let’s see how to ignore TestNG test using a script.

Script – Test Case 1:

package softwareTestingMaterial;
import org.testng.annotations.Test;
public class TestCase1 {
	@Test
	public void printClass(){
		System.out.println("This is Test Case 1");
	}

Script – Test Case 2: I would like to ignore this test from the test execution

12345678package softwareTestingMaterial;import org.testng.annotations.Test;public class TestCase2 { @Test (enabled=false) public void printClass2(){ System.out.println(“This is Test Case 2”); }}

testNg.xml file:

1234567891011<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.TestCase1″ /> <class name=”softwareTestingMaterial.TestCase2″ /> </classes> </test></suite>

Skip TestNG Test | TestNG Tutorial

How To Skip TestNG Test:

Let’s see how to skip TestNG test deliberately. Sometimes we may face a situation where our test cases might not be ready and we need to skip those tests from running. One way of skipping a test method is by using throw new SkipException() exception.

Scenario: Skip TestNG Test, If a condition met else continue execution.

Let see a sample WebDriver test case example where I have placed SkipException() inside if condition to Intentionally skip that test.

Note: Once SkipException() thrown, remaining part of that test method will not be executed and control will goes directly to next test method execution.

Scenario 1: 

Let me show you how the below program works by commenting the “throw new SkipException()” in the if condition. 

123456789101112131415161718192021222324package softwareTestingMaterial; import org.testng.annotations.Test;import org.testng.SkipException; public class SkipTestCase {  @Test public void aSkipTest(){ String condition =”Skip Test”; if(condition.equals(“Skip Test”)){ // throw new SkipException(“Skipping – This is not ready for testing “); }else{ System.out.println(“I am in else condition”); } System.out.println(“I am out of the if else condition”); } @Test public void nonSkipTest(){ System.out.println(“No need to skip this test”); } }

In the output console, we could see “out of loop” from first test method “aSkipTest” and No need to skip this test from second test method “nonSkipTest”

 Output Console:

123456789[TestNG] Running: I am out of the if else conditionNo need to skip this test ===============================================Default suiteTotal tests run: 2, Failures: 0, Skips: 0===============================================

TestNG Groups | TestNG Tutorial

TestNG allows you to perform sophisticated groupings of test methods. Not only can you declare that methods belong to groups, but you can also specify groups that contain other groups. Then TestNG can be invoked and asked to include a certain set of groups (or regular expressions) while excluding another set.  This gives you maximum flexibility in how you partition your tests and doesn’t require you to recompile anything if you want to run two different sets of tests back to back.

Groups are specified in your testng.xml file and can be found either under the <test> or <suite> tag. Groups specified in the <suite> tag apply to all the <test> tags underneath.

Script – Test Case 1:

12345678package softwareTestingMaterial;import org.testng.annotations.Test;public class TestCase1 { @Test (groups = { “smokeTest”, “functionalTest” }) public void loginTest(){ System.out.println(“Logged in successfully”); }}

Script – Test Case 2:

12345678package softwareTestingMaterial;import org.testng.annotations.Test;public class TestCase2 { @Test (groups = { “functionalTest” }) public void composeMail(){ System.out.println(“Mail Sent”); }}

testng.xml:

123456789101112131415161718<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <groups> <run> <include name=”smokeTest” /> </run> </groups> <classes> <class name=”softwareTestingMaterial.TestCase1″ /> <class name=”softwareTestingMaterial.TestCase2″ /> </classes> </test></suite>

Console Output:

12345678[TestNG] Running: Logged in successfully ===============================================softwaretestingmaterialTotal tests run: 1, Failures: 0, Skips: 0===============================================

Group of Groups in TestNG Groups:

Groups can also include other groups. These groups are called MetaGroups. For example, you might want to define a group all that includes smokeTest and functionalTest. Let’s modify our testng.xml file as follows:

testng.xml – Group of Groups:

1234567891011121314151617181920<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <groups>     <define name=”all”>             <include name=”smokeTest”/>             <include name=”functionalTest”/>     </define>         <run> <include name=”all” /> </run>         </groups> <classes> <class name=”softwareTestingMaterial.TestCase1″ /> <class name=”softwareTestingMaterial.TestCase2″ /> </classes> </test></suite>

Console Output:

12345678910[TestNG] Running:  C:\Users\Administrator\Desktop\TestNGProject\testng.xml Logged in successfullyMail Sent ===============================================softwaretestingmaterialTotal tests run: 2, Failures: 0, Skips: 0===============================================

Groups Exclusion:

TestNG allows you to include groups as well as exclude them. You can ignore a group by using the <exclude> tag as shown below:

For example, it is quite usual to have tests that temporarily break because of a recent change, and you don’t have time to fix the breakage yet. However, you do want to have clean runs of your functional tests, so you need to deactivate these tests but keep in mind they will need to be reactivated.

123456789101112131415161718192021<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <groups>                       <run> <exclude name=”smokeTest”/>             <include name=”functionalTest”/> </run>          </groups> <classes> <class name=”softwareTestingMaterial.TestCase1″ /> <class name=”softwareTestingMaterial.TestCase2″ /> </classes> </test></suite>

Console Output:

12345678[TestNG] Running: Mail Sent ===============================================softwaretestingmaterialTotal tests run: 1, Failures: 0, Skips: 0===============================================

You can also disable tests on an individual basis by using the “enabled” property available on both @Test and @Before/After annotations.

TestNG Exception

TestNG Exception:

In this post, I will show how to use TestNG Exception (i.e., expectedExceptions) parameter along with the @Test annotation. TestNG provides an option of tracing the exception handling of code.

Firstly, we see a basic program without using TestNG Exception.

Created a java class, say, TestNGException.java and added an error condition in the method testException().

12345678910package softwareTestingMaterial;import org.testng.annotations.Test;public class TestNGException {  @Test public void testException() { System.out.println(“SoftwareTestingMaterial.com”); int i = 1 / 0; }}

testng.xml:

123456789<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”softwaretestingmaterial”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.TestNGException” /> </classes> </test> </suite>

Console Output:

12345678[TestNG] Running: SoftwareTestingMaterial.com ===============================================Default suiteTotal tests run: 1, Failures: 1, Skips: 0===============================================

As you can see from the test results, testException() method was marked as failed by TestNG during execution.

Now let’s see TestNG Exception in action.

The expected exception to validate while running the below test is mentioned using the expectedExceptions attribute value while using the @Test annotation.

12345678910package softwareTestingMaterial;import org.testng.annotations.Test;public class TestNGException { @Test(expectedExceptions = ArithmeticException.class) public void testException() { System.out.println(“SoftwareTestingMaterial.com”); int i = 1 / 0; }}

testng.xml:

123456789<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”softwaretestingmaterial”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.TestNGException” /> </classes> </test> </suite>

Console Output:

12345678[TestNG] Running: SoftwareTestingMaterial.com ===============================================Default suiteTotal tests run: 1, Failures: 0, Skips: 0===============================================

As you can see from the test results, testException() method was marked as passed by TestNG during execution.

TestNG Dependencies

Sometimes, you may need to invoke methods in a test case in a certain order. Here comes TestNG Dependencies into the picture.

TestNG allows you to specify dependencies either with annotations or in XML.

First, we see Dependencies with annotations:

TestNG allows you to specify dependencies either with:

  • Using attribute dependsOnMethods in @Test annotations, OR.
  • Using attribute dependsOnGroups in @Test annotations.
package softwareTestingMaterial; import org.testng.annotations.Test; public class DependsOnMethodsTestCase { @Test public void testCase1(){ System.out.println(“Test Case 1”); } @Test public void testCase2(){ System.out.println(“Test Case 2”); } }

testng.xml

123456789101112<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.DependsOnMethodsTestCase” /> </classes> </test></suite>

Console Output:

123456789[TestNG] Running: Test Case 1Test Case 2 ===============================================softwaretestingmaterialTotal tests run: 2, Failures: 0, Skips: 0===============================================

Now we add the dependsOnMethods attribute to the @Test Annotations and execute the same program.

See the script below.

12345678910111213141516package softwareTestingMaterial; import org.testng.annotations.Test; public class DependsOnMethodsTestCase { @Test(dependsOnMethods = {“testCase2”}) public void testCase1(){ System.out.println(“Test Case 1”); } @Test public void testCase2(){ System.out.println(“Test Case 2”); } }

Execute the same testng.xml which was placed above and see the difference in Console Output

Console Output:

123456789[TestNG] Running: Test Case 2Test Case 1 ===============================================softwaretestingmaterialTotal tests run: 2, Failures: 0, Skips: 0===============================================

package softwareTestingMaterial;

import org.testng.annotations.Test;

public class DependsOnMethodsTestCase {

@Test(groups = {“FirstGroup”})

public void testCase1(){

System.out.println(“Test Case 1”);

}

@Test(groups = {“SecondGroup”})

public void testCase2(){

System.out.println(“Test Case 2”);

}

}

testng.xml

123456789101112131415161718<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” ><suite name=”softwaretestingmaterial”> <test name=”testngTest”>  <groups>                    <dependencies>                          <group name=”FirstGroup” depends-on=”SecondGroup”></group>                    </dependencies>                </groups> <classes> <class name=”softwareTestingMaterial.DependsOnMethodsTestCase” /> </classes> </test></suite>

Console Output:

123456789[TestNG] Running: Test Case 2Test Case 1 ===============================================softwaretestingmaterialTotal tests run: 2, Failures: 0, Skips: 0===============================================

TestNG Parameterization Using XML

TestNG Parameterization Using XML:

Parameterized tests allow developers to run the same test over and over again using different values.

There are two ways to set these parameters:

  • with testng.xml
  • with Data Providers

Let’s see passing parameters with testng.xml:

With this technique, we could define the parameters in the testng.xml file and then reference those parameters in the source files.

Create a java test class, say, ParameterizedTest.java

Add test method parameterizedTest() to your test class. This method takes a string as input parameter

Add the annotation @Parameters(“browser”) to this method. The parameter would be passed a value from testng.xml, which we will see in the next step.

1234567891011121314151617package softwareTestingMaterial; import org.testng.annotations.Parameters;import org.testng.annotations.Test; public class ParameterizedTest { @Test @Parameters(“browser”) public void parameterizedTest(String browser){ if(browser.equals(“firefox”)){ System.out.println(“Open Firefox Driver”); }else if(browser.equals(“chrome”)){ System.out.println(“Open Chrome Driver”); } } }

testng.xml

1<parameter name=”browser” value=”firefox”/>

Here, name attribute represents the parameter name and value represents the value of that parameter. We could use this parameter to the test method parameterizedTest by using @Parameters(“browser”) as mentioned in the above step.

Now let’s see the complete testng.xml file

1234567891011<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd” > <suite name=”softwaretestingmaterial”> <test name=”testngTest”> <parameter name=”browser” value=”firefox”/> <classes> <class name=”softwareTestingMaterial.ParameterizedTest” /> </classes> </test> </suite>

Console Output:

12345678[TestNG] Running: Open Firefox Driver ===============================================softwaretestingmaterialTotal tests run: 1, Failures: 0, Skips: 0===============================================

TestNG will automatically try to convert the value specified in testng.xml to the type of your parameter. Here are the types supported:

  • String
  • int/Integer
  • boolean/Boolean
  • byte/Byte
  • char/Character
  • double/Double
  • float/Float
  • long/Long
  • short/Short

TestNG Parameterization Using DataProviders

Specifying parameters in testng.xml might not be sufficient if you need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc…). In this case, you can use a Data Provider to supply the values you need to test.  A Data Provider is a method on your class that returns an array of objects.  This method is annotated with @DataProvider

1234567891011121314151617181920212223242526272829303132package softwareTestingMaterial; import org.testng.annotations.DataProvider;import org.testng.annotations.Test; public class DataProviderClass {  // This method takes data as input parameters. The attribute dataprovider is mapped to “getData” @Test (dataProvider=”getData”)    // Number of columns should match the number of input parameters public void loginTest(String Uid, String Pwd){ System.out.println(“UserName is “+ Uid); System.out.println(“Password is “+ Pwd); } //If the name is not supplied, the data provider’s name automatically defaults to the method’s name. //A data provider returns an array of objects. @DataProvider(name=”getData”) public Object[][] getData(){ //Object [][] data = new Object [rowCount][colCount]; Object [][] data = new Object [2][2]; data [0][0] = “FirstUid”; data [0][1] = “FirstPWD”; data[1][0] = “SecondUid”; data[1][1] = “SecondPWD”; return data; }}

Console Output:

UserName is FirstUid

Password is FirstPWD

UserName is SecondUid

Password is SecondPWD

===============================================

softwaretestingmaterial

Total tests run: 2, Failures: 0, Skips: 0

===============================================

Parallel Test Execution In TestNG

There are situations where we want to run multiple tests with same or different browsers at the same time. In such cases, we can use “parallel” attribute in testng.xml to accomplish parallel test execution in TestNG

The parallel attribute of suite tag can accept four values:

tests – All the test cases inside <test> tag of testng.xml file will run parallel

classes – All the test cases inside a java class will run parallel

methods – All the methods with @Test annotation will execute parallel

instances – Test cases in same instance will execute parallel but two methods of two different instances will run in different thread.

let us look at example of Parallel Test execution in TestNG.

In the below program, I took two methods. First methods opens Firefox driver and navigate to http://www.softwaretestingmaterial.com and closes the browser. Second methods opens Chrome driver and navigate to the same URL and closes the browser.

Java

12345678910111213141516171819
123456789101112131415161718192021222324252627282930package softwareTestingMaterial; import org.openqa.selenium.WebDriver;import org.openqa.selenium.chrome.ChromeDriver;import org.openqa.selenium.firefox.FirefoxDriver;import org.testng.annotations.Test; public class ParallelTests { @Test public void getFirefox(){                //System.setProperty(“webdriver.gecko.driver”, “geckodriver.exe path”); System.setProperty(“webdriver.gecko.driver”, “D://Selenium Environment//Drivers//geckodriver.exe”);                System.out.println(“GetFirefox Method is running on Thread : ” + Thread.currentThread().getId()); WebDriver driver = new FirefoxDriver(); driver.get(“http://www.softwaretestingmaterial.com”); driver.close(); } @Test public void getChorme(){                //System.setProperty(“webdriver.chrome.driver”, “chromedriver.exe path”); System.setProperty(“webdriver.chrome.driver”, “D://Selenium Environment//Drivers//chromedriver.exe”);                System.out.println(“GetChrome Method is running on Thread : ” + Thread.currentThread().getId()); WebDriver driver = new ChromeDriver(); driver.get(“http://www.softwaretestingmaterial.com”); driver.close(); } }

testng.xml file without mentioning parallel attribute:

123456789<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”softwaretestingmaterial”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.ParallelTests” /> </classes> </test> </suite>

After running the testng.xml using the above mentioned code, first you could see the firefox browser in action and then you can see the chrome driver in action.

To run both the browsers in parallel, use the below code in your testng.xml file.

123456789<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”softwaretestingmaterial” parallel=”methods” thread-count=”2″> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.ParallelTests” /> </classes> </test> </suite>

Once you run the testng.xml using the above code, you could see both the browsers in action at a time.

Here in the above testng.xml file, I have passed parallel=methods and thread-count=2 at the suite level. I would like to execute selenium scripts in parallel in different threads. Most of the times, these two methods will execute in different threads. Thread Id may vary on every run. Here we are just passing thread count but we are not assigning any thread id, assigning thread id will be taken care by your system processor.

Testng Asserts – How To Use Assertions In TestNG

TestNG Asserts help us to verify the condition of the test in the middle of the test run. Based on the TestNG Assertions, we will consider a successful test only if it is completed the test run without throwing any exception.

Different TestNG Asserts Statements:

Assert.assertEquals(String actual,String expected) : Asserts that two Strings are equal. If they are not, an AssertionError is thrown.

Parameters:

actual – the actual value

expected – the expected value

Assert.assertEquals(String actual,String expected, String message) : Asserts that two Strings are equal. If they are not, an AssertionError, with the given message, is thrown.

Parameters:

actual – the actual value

expected – the expected value

message – the assertion error message

Assert.assertEquals(boolean actual,boolean expected) : Asserts that two booleans are equal. If they are not, an AssertionError is thrown.

Parameters:

actual – the actual value

expected – the expected value

Assert.assertTrue(condition) : Asserts that a condition is true. If it isn’t, an AssertionError is thrown.

Parameters:

condition – the condition to evaluate

Assert.assertTrue(condition, message) :  Asserts that a condition is true. If it isn’t, an AssertionError, with the given message, is thrown.

Parameters:

condition – the condition to evaluate

message – the assertion error message

Assert.assertFalse(condition) :  Asserts that a condition is false. If it isn’t, an AssertionError is thrown.

Parameters:

condition – the condition to evaluate

Assert.assertFalse(condition, message) : Asserts that a condition is false. If it isn’t, an AssertionError, with the given message, is thrown.

Parameters:

condition – the condition to evaluate

message – the assertion error message

How To Use Soft Assert In TestNG

Asserts are used to perform validations in the test scripts.

There are two types of Assert:

  1. Hard Assert
  2. Soft Assert

When an assert fails the test script stops execution unless handled in some form. We call general assert as Hard Assert

Hard Assert – Hard Assert throws an AssertException immediately when an assert statement fails and test suite continues with next @Test

The disadvantage of Hard Assert – It marks method as fail if assert condition gets failed and the remaining statements inside the method will be aborted.

To overcome this we need to use Soft Assert. Let’s see what is Soft Assert.

Soft Assert – Soft Assert collects errors during @Test. Soft Assert does not throw an exception when an assert fails and would continue with the next step after the assert statement.

If there is any exception and you want to throw it then you need to use assertAll() method as a last statement in the @Test and test suite again continue with next @Test as it is.

We need to create an object to use Soft Assert which is not needed in Hard Assert.

Let’s see a practical example:

Here I took two methods namely softAssert() and hardAssert().

In the softAssert() method, I have used SoftAssert class and intentionally passing value false in the assertTrue() method to make it fail

In thehardAssert() method, I simply used Assert and intentionally passing parameter value false in the assertTrue() method to make it fail

1234567891011121314151617181920212223package softwareTestingMaterial; import org.testng.Assert;import org.testng.annotations.Test;import org.testng.asserts.SoftAssert; public class SoftAssertion {  @Test public void softAssert(){ SoftAssert softAssertion= new SoftAssert(); System.out.println(“softAssert Method Was Started”); softAssertion.assertTrue(false); System.out.println(“softAssert Method Was Executed”); } @Test public void hardAssert(){ System.out.println(“hardAssert Method Was Started”); Assert.assertTrue(false); System.out.println(“hardAssert Method Was Executed”); }}

Execute the above script and see the console output. You could see only one failure.

Console Output:

1234567891011[TestNG] Running:  C:\Users\Administrator\AppData\Local\Temp\testng-eclipse–2097831536\testng-customsuite.xml hardAssert Method Was StartedsoftAssert Method Was StartedsoftAssert Method Was Executed ===============================================Default suiteTotal tests run: 2, Failures: 1, Skips: 0===============================================
Soft Assert

Below script is same as the first one but just added assertAll() method in the end of the first method (i.e., softAssert()).

Note:  If you forget to call assertAll() at the end of your test, the test will pass even if any assert objects threw exceptions as shown in the above example. So don’t forget to add assertAll()

123456789101112131415161718192021222324package softwareTestingMaterial; import org.testng.Assert;import org.testng.annotations.Test;import org.testng.asserts.SoftAssert; public class SoftAssertion {  @Test public void softAssert(){ SoftAssert softAssertion= new SoftAssert(); System.out.println(“softAssert Method Was Started”); softAssertion.assertTrue(false); System.out.println(“softAssert Method Was Executed”); softAssertion.assertAll(); } @Test public void hardAssert(){ System.out.println(“hardAssert Method Was Started”); Assert.assertTrue(false); System.out.println(“hardAssert Method Was Executed”); }}

Execute the above script and see the console output. There are two failures here. Second failure is due to assertAll() method

Console Output:

1234567891011[TestNG] Running:  C:\Users\Administrator\AppData\Local\Temp\testng-eclipse–147736931\testng-customsuite.xml hardAssert Method Was StartedsoftAssert Method Was StartedsoftAssert Method Was Executed ===============================================Default suiteTotal tests run: 2, Failures: 2, Skips: 0===============================================
Soft Assert

TestNG Listeners

Listeners “listen” to the event defined in the selenium script and behave accordingly. The main purpose of using listeners is to create logs. There are many types of listeners such as WebDriver Listeners and TestNG Listeners.

Here in this post, we see TestNG Listeners. Using TestNG listeners we could generate logs and customize TestNG Reports.

Let’s see how to implement TestNG Listeners.

Step 1: Create a Class “ListenerTestNG” to implement ITestListener methods

Java

12345
123456package listeners; public class ListenerTestNG {   }
12345package listeners; public class ListenerTestNG implements ITestListener{  }

Mouse hover on ITestListener and import ITestListener Interface and mouse hover on ListenerTestNG and click on “Add unimplemented methods” to add multiple unimplemented methods (without body)

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950package listeners; import org.testng.ITestContext;import org.testng.ITestListener;import org.testng.ITestResult; public class ListenerTestNG implements ITestListener{  @Override public void onTestStart(ITestResult result) { // TODO Auto-generated method stub }  @Override public void onTestSuccess(ITestResult result) { // TODO Auto-generated method stub }  @Override public void onTestFailure(ITestResult result) { // TODO Auto-generated method stub }  @Override public void onTestSkipped(ITestResult result) { // TODO Auto-generated method stub }  @Override public void onTestFailedButWithinSuccessPercentage(ITestResult result) { // TODO Auto-generated method stub }  @Override public void onStart(ITestContext context) { // TODO Auto-generated method stub }  @Override public void onFinish(ITestContext context) { // TODO Auto-generated method stub } }

Here, I am implementing onTestSuccess, onTestSkipped, onTestFailure methods.

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455package listeners; import org.testng.ITestContext; import org.testng.ITestListener; import org.testng.ITestResult;  public class ListenerTestNG implements ITestListener {      @Override     public void onFinish(ITestContext Result)     {                      }      @Override     public void onStart(ITestContext Result)     {                  }      @Override     public void onTestFailedButWithinSuccessPercentage(ITestResult Result)     {          }      // When Test case get failed, this method is called.     @Override     public void onTestFailure(ITestResult Result)     {     System.out.println(“The name of the testcase failed is :”+Result.getName());     }      // When Test case get Skipped, this method is called.     @Override     public void onTestSkipped(ITestResult Result)     {     System.out.println(“The name of the testcase Skipped is :”+Result.getName());     }     @Override     public void onTestStart(ITestResult Result)     {         }      // When Test case get passed, this method is called.     @Override     public void onTestSuccess(ITestResult Result)     {     System.out.println(“The name of the testcase passed is :”+Result.getName());     }  }

Step 2: Create another Class “ListenerTestNGTestCase” and write a script (which ever script you prefer). Else copy paste the below mentioned script.

Step 3: Add the listeners annotation (@Listeners) in the Class “ListenerTestNGTestCase”

1@Listeners(listeners.ListenerTestNG.class)

The complete “ListenerTestNGTestCase” class after adding Listener annotation is mentioned below:

1234567891011121314151617181920212223242526272829303132333435package listeners; import org.openqa.selenium.WebDriver; import org.openqa.selenium.firefox.FirefoxDriver; import org.testng.SkipException;import org.testng.annotations.Listeners;import org.testng.annotations.Test; @Listeners(listeners.ListenerTestNG.class)public class ListenerTestNGTestCase {  WebDriver driver= new FirefoxDriver();   // Test to pass as to verify listeners . @Test(priority=1) public void TestToPass() {     System.out.println(“This method to pass test”);     driver.get(“https://www.softwaretestingmaterial.com/100-software-testing-interview-questions/”);     driver.getTitle();     driver.quit(); } //Used skip exception to skip the test @Test(priority=2) public void TestToSkip () {            System.out.println(“This method to skip test”);     throw new SkipException(“Skipping – This is not ready for testing “); } // In the above method, we have already closed the browser. So we couldnot get the title here. It is to forcefully fail the test @Test(priority=3) public void TestToFail() {     driver.getTitle();     System.out.println(“This method to test fail”); }}

Step 4: Execute the “ListenerTestNGTestCase” class. Methods in class “ListenerTestNG” are called automatically according to the behavior of methods annotated as @Test.

Step 5: Verify the Output in the console. You could find the logs in the console.

If you want to use listeners in multiple classes.

Add the below lines of code in the TestNG.xml file

123<listeners>        <listener class-name=”listeners.listenerTestNG”/></listeners>

Final testng.xml file will be like this:

123456789101112<xml version=”1.0″ encoding=”UFT-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”Suite”>    <listeners>        <listener class-name=”listeners.listenerTestNG”/>    </listeners>    <test name=”Test”>        <classes>            <class name=”listeners.ListenerTestNGTestCase”>        </classes>    </test></suite>

Execute it by right clicking on testng.xml and run as TestNG Suite

Run Failed Test Cases Using TestNG in Selenium WebDriver

At times, test cases may fail while running automated test scripts.  The reason may be anything (say, Network issue, System issue or browser issue) but as an automation tester, you need to execute the test scripts again. Here is a solution to run failed test cases using TestNG in Selenium.

Don’t you want to take a screenshot on failure and attach that in your report?

Yeah, as a tester we need to capture the failed steps. This post guides you on “How To Capture Screenshot and Insert it in Extent Reports”

We could execute the failed test cases in two ways.

Case 1: Execute failed test cases using TestNG in Selenium – By using “testng-failed.xml”

Steps To follow:

  1. After the first run of an automated test run. Right click on Project – Click on Refresh
  2. A folder will be generated named “test-output” folder. Inside “test-output” folder, you could find “testng-failed.xml”
  3. Run “testng-failed.xml” to execute the failed test cases again.
Run Failed Test Cases Using TestNG

Case 2: Execute failed test cases using TestNG in Selenium – By Implementing TestNG IRetryAnalyzer.

Create a class to implement IRetryAnalyzer. Here I am creating a class (say, RetryFailedTestCases) and implementing IRetryAnalyzer.

RetryFailedTestCases implements IRetryAnalyzer:

123456789101112131415161718192021package softwareTestingMaterial; import org.testng.IRetryAnalyzer;import org.testng.ITestResult; public class RetryFailedTestCases implements IRetryAnalyzer {    private int retryCnt = 0;    //You could mentioned maxRetryCnt (Maximiun Retry Count) as per your requirement. Here I took 2, If any failed testcases then it runs two times    private int maxRetryCnt = 2;        //This method will be called everytime a test fails. It will return TRUE if a test fails and need to be retried, else it returns FALSE    public boolean retry(ITestResult result) {        if (retryCnt < maxRetryCnt) {            System.out.println(“Retrying ” + result.getName() + ” again and the count is ” + (retryCnt+1));            retryCnt++;            return true;        }        return false;    }  }

Let’s create another class ‘RetryListenerClass’ by Implementing ‘IAnnotationTransaformer’ interface. transform method is called for every test during test run. A simple implementation of this ‘IAnnotationTransformer’ interface can help us set the ‘setRetryAnalyzer’ for ‘ITestAnnotation’. Add the above class name (RetryFailedTestCases.class) in the below program. This interface does its work in run time by adding annotation to the test methods.

RetryListenerClass implements IAnnotationTransformer:

1234567891011121314151617181920package softwareTestingMaterial;import java.lang.reflect.Constructor;import java.lang.reflect.Method; import org.testng.IAnnotationTransformer;import org.testng.IRetryAnalyzer;import org.testng.annotations.ITestAnnotation; public class RetryListenerClass implements IAnnotationTransformer {  @Override public void transform(ITestAnnotation testannotation, Class testClass, Constructor testConstructor, Method testMethod) { IRetryAnalyzer retry = testannotation.getRetryAnalyzer();  if (retry == null) { testannotation.setRetryAnalyzer(RetryFailedTestCases.class); }  }}

Let us see the example by executing simple tests below. Here I took two test cases say Test1 and Test2.

Testcase 1:

1234567891011121314package softwareTestingMaterial; import org.testng.Assert;import org.testng.annotations.Test; public class Test1 {  @Test public void test1(){ System.out.println(“Test 1”); Assert.assertTrue(true); }}

Testcase 2:

123456789101112package softwareTestingMaterial; import org.testng.Assert;import org.testng.annotations.Test; public class Test2 { @Test public void test2(){ System.out.println(“Test 2”); Assert.assertTrue(false); }}

As per the lines of code in Test2, it will fail. So it (Test2) will be executed 2 times (we took the maxRetryCnt as 2) in Retry Class. First lets include below mentioned Listener to testng.xml file. Below mentioned syntax is to add Listener for RetryListnereClass

123<listeners>        <listener class-name=”softwareTestingMaterial.RetryListenerClass”/></listeners>

Final testng.xml file should looks like below:

12345678910111213141516171819202122<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”My Suite”> <listeners>        <listener class-name=”softwareTestingMaterial.RetryListenerClass”/>  </listeners>  <test name=”Test1″> <classes> <class name=”softwareTestingMaterial.Test1″ /> </classes> </test> <!– Test –> <test name=”Test2″> <classes> <class name=”softwareTestingMaterial.Test2″ /> </classes> </test> <!– Test –> </suite> <!– Suite –>

Execute the testng.xml. Here is the output which I got. You could see in the below mentioned result that the Test 2 is executed three times as we have mentioned ‘maxRetryCnt = 2’. Even though we have just 2 tests, we could find total test runs are 4 in the result.

1234567891011121314[TestNG] Running:  D:\Selenium\workspace\SeleniumProject\testng.xml Test 1Test 2Retrying test2 again and the count is 1Test 2Retrying test2 again and the count is 2Test 2 ===============================================Everjobs SuiteTotal tests run: 4, Failures: 1, Skips: 2===============================================

This way we could run failed test cases using TestNG in Selenium.

How To Generate TestNG Reports

How to run testng.xml from maven

TestNG Reports come in to the picture once we execute the test cases using TestNG. Once we execute test cases using TestNG, it will generate a default HTML report. Let’s see this process of generating TestNG Reports in detail.

I have created a basic script and mentioned below.

It has three methods namely passTest, failTest and skipTest with @Test annotation.

1234567891011121314151617181920212223package softwareTestingMaterial; import org.testng.Assert;import org.testng.SkipException;import org.testng.annotations.Test; public class DefaultReport { //To make it pass @Test public void passTest(){ Assert.assertTrue(true); } //To make it fail @Test public void failTest(){ Assert.assertTrue(false); } //To make it skip @Test public void skipTest(){ throw new SkipException(“Skipping – This is not ready for testing “); }}

TestNG.xml file

123456789<?xml version=”1.0″ encoding=”UTF-8″?><!DOCTYPE suite SYSTEM “http://testng.org/testng-1.0.dtd”><suite name=”softwaretestingmaterial” parallel=”methods”> <test name=”testngTest”> <classes> <class name=”softwareTestingMaterial.DefaultReport” /> </classes> </test> </suite>

Execute the testng.xml file and refresh the project. You could see your project similar to the below mentioned image.

TestNG Default Report

Navigate to ‘test-output’ folder. Now you should find a report ”emailable-report.html‘. This is the default report generated by TestNG.

TestNG Default Report

Open emailable -report.html using any browser of your choice. Report will be like as shown below:

TestNG Default Report

How To Run TestNG Using Command Prompt 

Goal: To Run TestNG using Command Prompt:

Steps to achieve our goal – Executing TestNG using Command Prompt:

  1. Open Eclipse and create a Java class
  2. Write a Java program
  3. Convert the Java Program into TestNG
  4. Open command prompt
  5. Run the TestNG using command prompt

Step i: Open Eclipse and create a Java class

Step ii. Keep all the library files in a folder (here I create a folder name “lib”)

Step iii: Write a Java program

TestNG Using Command Prompt

Step iv: Convert the Java Program into TestNG

TestNG Using Command Prompt

Step v: Open command prompt

Step vi: Run the TestNG using command prompt

1C:\Users\Admin\Desktop\STMSeleniumTutorial\workspace\SoftwareTestingMaterial
1set classpath=C:\Users\Admin\Desktop\STMSeleniumTutorial\workspace\SoftwareTestingMaterial\bin;C:\Users\Admin\Desktop\STMSeleniumTutorial\workspace\SoftwareTestingMaterial\lib\*
1java org.testng.TestNG C:\Users\Admin\Desktop\STMSeleniumTutorial\workspace\SoftwareTestingMaterial\testng.xml

Also you could run TestNG using Batch file (.bat file)

Copy the below code and place it in a notepad and save the file using .bat extension

123456789set projectLocation=C:\Users\Admin\Desktop\STMSeleniumTutorial\workspace\SoftwareTestingMaterial cd %projectLocation% set classpath=%projectLocation%\bin;%projectLocation%\lib\* java org.testng.TestNG %projectLocation%\testng.xml pause
TestNG using command prompt
TestNG using command prompt