Full docs

Welcome

Note

This is not official documentation . Visit https://testng.org/ for official documentation .Its just a stylized version of the official documentation .

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:

  • Annotations.
  • Run your tests in arbitrarily big thread pools with various policies available (all methods in their own thread, one thread per test class, etc...).
  • Test that your code is multithread safe.
  • Flexible test configuration.
  • Support for data-driven testing (with @DataProvider).
  • Support for parameters.
  • Powerful execution model (no more TestSuite).
  • Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...).
  • Embeds BeanShell for further flexibility.
  • Default JDK functions for runtime and logging (no dependencies).
  • Dependent methods for application server testing.
  • TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc...

Requirements

TestNG requires JDK 7 or higher.

Mailing-lists

Locations of the projects

If you are interested in contributing to TestNG or one of the IDE plug-ins, you will find them in the following locations:

Bug reports

If you think you found a bug, here is how to report it:

  • Create a small project that will allow us to reproduce this bug. In most cases, one or two Java source files and a testng.xml file should be sufficient. Then you can either zip it and email it to the testng-dev mailing-list or make it available on an open source hosting site, such as github and email testng-dev so we know about it. Please make sure that this project is self contained so that we can build it right away (remove the dependencies on external or proprietary frameworks, etc...).
  • If the bug you observed is on the Eclipse plug-in, make sure your sample project contains the .project and .classpath files.
  • File a bug.

For more information, you can either download TestNG, read the manual.

License

Apache 2.0

Introduction

TestNG is a testing framework designed to simplify a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages and even several external frameworks, such as application servers).

Writing a test is typically a three-step process:

  1. Write the business logic of your test and insert TestNG annotations in your code.
  2. Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a testng.xml file or in build.xml.
  3. Run TestNG.

You can find a quick example on the Getting Started page.

The concepts used in this documentation are as follows:

  • A suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag.
  • A test is represented by <test> and can contain one or more TestNG classes.
  • A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods.
  • A test method is a Java method annotated by @Test in your source.
  • A TestNG test can be configured by @BeforeXXX and @AfterXXX annotations which allows to perform some Java logic before and after a certain point, these points being either of the items listed above.

The rest of this manual will explain the following:

  • A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details.
  • A description of the testng.xml file, its syntax and what you can specify in it.
  • A detailed list of the various features and how to use them with a combination of annotations and testng.xml.

Getting Started

I started TestNG out of frustration for some JUnit deficiencies which I have documented on my weblog here and here Reading these entries might give you a better idea of the goal I am trying to achieve with TestNG. You can also check out a quick overview of the main features and an article describing a very concrete example where the combined use of several TestNG's features provides for a very intuitive and maintainable testing design.

Here is a very simple test:

package example1;

import org.testng.annotations.*;

public class SimpleTest {

 @BeforeClass
 public void setUp() {
   // code that will be invoked when this test is instantiated
 }

 @Test(groups = { "fast" })
 public void aFastTest() {
   System.out.println("Fast test");
 }

 @Test(groups = { "slow" })
 public void aSlowTest() {
    System.out.println("Slow test");
 }

}

The method setUp() will be invoked after the test class has been built and before any test method is run. In this example, we will be running the group fast, so aFastTest() will be invoked while aSlowTest() will be skipped.

Things to note:

  • No need to extend a class or implement an interface.
  • Even though the example above uses the JUnit conventions, our methods can be called any name you like, it's the annotations that tell TestNG what they are.
  • A test method can belong to one or several groups.

Once you have compiled your test class into the build directory, you can invoke your test with the command line, an ant task (shown below) or an XML file:

<project default="test">

 <path id="cp">
   <pathelement location="lib/testng-testng-5.13.1.jar"/>
   <pathelement location="build"/>
 </path>

 <taskdef name="testng" classpathref="cp"
          classname="org.testng.TestNGAntTask" />

 <target name="test">
   <testng classpathref="cp" groups="fast">
     <classfileset dir="build" includes="example1/*.class"/>
   </testng>
 </target>

</project>

Use ant to invoke it:

c:> ant
Buildfile: build.xml

test:
[testng] Fast test
[testng] ===============================================
[testng] Suite for Command line test
[testng] Total tests run: 1, Failures: 0, Skips: 0
[testng] ===============================================


BUILD SUCCESSFUL
Total time: 4 seconds

Then you can browse the result of your tests:

start test-output\index.html (on Windows)

Migrating from JUnit

Using Eclipse

The easiest way to convert your JUnit tests to TestNG is to use the Eclipse TestNG plug-in refactoring support. You will find a full description of its features in the Eclipse section.

Asserts

Note that the class org.testng.Assert uses a different argument ordering than the ones used by JUnit. If you are porting code that uses JUnit's asserts, you might want to us a static import of that class:

import static org.testng.AssertJUnit.*;

Running JUnit Tests

TestNG can automatically recognize and run JUnit tests, so you can use TestNG as a runner for all your existing tests and write new tests using TestNG.

All you have to do is to put JUnit library on the TestNG classpath, so it can find and use JUnit classes, change your test runner from JUnit to TestNG in Ant and then run TestNG in "mixed" mode. This way you can have all your tests in the same project, even in the same package, and start using TestNG. This approach also allows you to convert your existing JUnit tests to TestNG incrementally.

Example - replacing JUnit Ant task with TestNG one

JUnit version:

<junit dir="${work.dir}" errorproperty="tests.failed" failureproperty="tests.failed" fork="true">
    <batchtest todir="${build.test.results.dir}">
        <fileset dir="${test.src.dir}">
            <include name="**/*Test.*"/>
    </batchtest>
    <classpath>
        <path path="${run.test.classpath}"/>
    </classpath>
    <syspropertyset>
        <propertyref prefix="test-sys-prop."/>
        <mapper from="test-sys-prop.*" to="*" type="glob"/>
    </syspropertyset>
    <formatter type="xml"/>
    <jvmarg value="-ea"/>
    <jvmarg line="${run.jvmargs}"/>
</junit>

TestNG version:

<taskdef name="testng" classname="org.testng.TestNGAntTask" classpath="${run.test.classpath}"/>

<fileset id="mixed.tests" dir="${test.src.dir}">
    <include name="**/*Test.*"/>
</fileset>

<testng mode="mixed" classfilesetref="mixed.tests" workingDir="${work.dir}" failureProperty="tests.failed" outputdir="${build.test.results.dir}">
    <classpath>
        <pathelement path="${build.test.classes.dir}"/>
        <pathelement path="${run.test.classpath}"/>
        <pathelement path="${junit.lib}"/>
    </classpath>
    <propertyset>
        <propertyref prefix="test-sys-prop."/>
        <mapper from="test-sys-prop.*" to="*" type="glob"/>
    </propertyset>
    <jvmarg line="${run.jvmargs}"/>
</testng>

Annotations

Here is a quick overview of the annotations available in TestNG along with their attributes.

Configuration information for a TestNG class:

Anotation Description
@BeforeSuite The annotated method will be run before all tests in this suite have run.
@AfterSuite The annotated method will be run after all tests in this suite have run.
@BeforeTest The annotated method will be run before any test method belonging to the classes inside the tag is run.
@AfterTest: The annotated method will be run after all the test methods belonging to the classes inside the tag have run.
@BeforeGroups The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked.
@AfterGroups The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked.
@BeforeClass The annotated method will be run before the first test method in the current class is invoked.
@AfterClass The annotated method will be run after all the test methods in the current class have been run.
@BeforeMethod The annotated method will be run before each test method.
@AfterMethod The annotated method will be run after each test method.

Behaviour of annotations in superclass of a TestNG class.

The annotations above will also be honored (inherited) when placed on a superclass of a TestNG class. This is useful for example to centralize test setup for multiple test classes in a common superclass.In that case, TestNG guarantees that the "@Before" methods are executed in inheritance order (highest superclass first, then going down the inheritance chain), and the "@After" methods in reverse order (going up the inheritance chain).

\ \
alwaysRun For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
dependsOnGroups The list of groups this method depends on.
dependsOnMethods The list of methods this method depends on.
enabled Whether methods on this class/method are enabled.
groups The list of groups this class/method belongs to.
inheritGroups If true, this method will belong to groups specified in the @Test annotation at the class level.
onlyForGroups Only for @BeforeMethod and @AfterMethod. If specified, then this setup/teardown method will only be invoked if the corresponding test method belongs to one of the listed groups.
\ \
@DataProvider Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.
name The name of this data provider. If it's not supplied, the name of this data provider will automatically be set to the name of the method.
parallel If set to true, tests generated using this data provider are run in parallel. Default value is false.
\ \
@Factory Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].
\ \
@Listeners Defines listeners on a test class.
value An array of classes that extend org.testng.ITestNGListener.
\ \
@Parameters Describes how to pass parameters to a @Test method.
value The list of variables used to fill the parameters of this method.
\ \
@Test Marks a class or a method as part of the test.
alwaysRun If set to true, this test method will always be run even if it depends on a method that failed.
dataProvider The name of the data provider for this test method.
dataProviderClass The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
dependsOnGroups The list of groups this method depends on.
dependsOnMethods The list of methods this method depends on.
description The description for this method.
enabled Whether methods on this class/method are enabled.
expectedExceptions The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
groups The list of groups this class/method belongs to.
invocationCount The number of times this method should be invoked.
invocationTimeOut The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.
priority The priority for this test method. Lower priorities will be scheduled first.
successPercentage The percentage of success expected from this method
singleThreaded If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with parallel="methods". This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called sequential (now deprecated).
timeOut The maximum number of milliseconds this test should take.
threadPoolSize The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount.
Note: this attribute is ignored if invocationCount is not specified

Test methods, classes and groups

Test methods

Test methods are annotated with @Test. Methods annotated with @Test that happen to return a value will be ignored, unless you set allow-return-values to true in your testng.xml:

<suite allow-return-values="true">

or

<test allow-return-values="true">

Test groups

TestNG allows you to perform sophisticated groupings of test methods. Not only can you declare that methods belong to groups, but you can also specify groups that contain other groups. Then TestNG can be invoked and asked to include a certain set of groups (or regular expressions) while excluding another set. This gives you maximum flexibility in how you partition your tests and doesn't require you to recompile anything if you want to run two different sets of tests back to back.

Groups are specified in your testng.xml file and can be found either under the <test> or <suite> tag. Groups specified in the <suite> tag apply to all the <test> tags underneath.

Note that groups are accumulative in these tags: if you specify group "a" in <suite> and "b" in <test>, then both "a" and "b" will be included.

For example, it is quite common to have at least two categories of tests

  • Check-in tests :- These tests should be run before you submit new code. They should typically be fast and just make sure no basic functionality was broken.

  • Functional tests :- These tests should cover all the functionalities of your software and be run at least once a day, although ideally you would want to run them continuously.

Typically, check-in tests are a subset of functional tests. TestNG allows you to specify this in a very intuitive way with test groups. For example, you could structure your test by saying that your entire test class belongs to the "functest" group, and additionally that a couple of methods belong to the group "checkintest":

public class Test1 {
  @Test(groups = { "functest", "checkintest" })
  public void testMethod1() {
  }

  @Test(groups = {"functest", "checkintest"} )
  public void testMethod2() {
  }

  @Test(groups = { "functest" })
  public void testMethod3() {
  }
}

Invoking TestNG with

<test name="Test1">
  <groups>
    <run>
      <include name="functest"/>
    </run>
  </groups>
  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

will run all the test methods in that classes, while invoking it with checkintest will only run testMethod1() and testMethod2().

Here is another example, using regular expressions this time. Assume that some of your test methods should not be run on Linux, your test would look like:

@Test
public class Test1 {
  @Test(groups = { "windows.checkintest" })
  public void testWindowsOnly() {
  }

  @Test(groups = {"linux.checkintest"} )
  public void testLinuxOnly() {
  }

  @Test(groups = { "windows.functest" )
  public void testWindowsToo() {
  }
}

You could use the following testng.xml to launch only the Windows methods:

<test name="Test1">
  <groups>
    <run>
      <include name="windows.*"/>
    </run>
  </groups>

  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

Warning

TestNG uses regular expressions, and not wildmats. Be aware of the difference (for example, "anything" is matched by ".*" -- dot star -- and not "*").

Method groups

You can also exclude or include individual methods:

<test name="Test1">
  <classes>
    <class name="example1.Test1">
      <methods>
        <include name=".*enabledTestMethod.*"/>
        <exclude name=".*brokenTestMethod.*"/>
      </methods>
     </class>
  </classes>
</test>

This can come in handy to deactivate a single method without having to recompile anything, but I don't recommend using this technique too much since it makes your testing framework likely to break if you start refactoring your Java code (the regular expressions used in the tags might not match your methods any more).

Groups of groups

Groups can also include other groups. These groups are called "MetaGroups". For example, you might want to define a group "all" that includes "checkintest" and "functest". "functest" itself will contain the groups "windows" and "linux" while "checkintest will only contain "windows". Here is how you would define this in your property file:

<test name="Regression1">
  <groups>
    <define name="functest">
      <include name="windows"/>
      <include name="linux"/>
    </define>

    <define name="all">
      <include name="functest"/>
      <include name="checkintest"/>
    </define>

    <run>
      <include name="all"/>
    </run>
  </groups>

  <classes>
    <class name="test.sample.Test1"/>
  </classes>
</test>

Exclusion groups

TestNG allows you to include groups as well as exclude them.

For example, it is quite usual to have tests that temporarily break because of a recent change, and you don't have time to fix the breakage yet. 4 However, you do want to have clean runs of your functional tests, so you need to deactivate these tests but keep in mind they will need to be reactivated. A simple way to solve this problem is to create a group called "broken" and make these test methods belong to it. For example, in the above example, I know that testMethod2() is now broken so I want to disable it:

@Test(groups = {"checkintest", "broken"} )
public void testMethod2() {
}

All I need to do now is to exclude this group from the run:

<test name="Simple example">
  <groups>
    <run>
      <include name="checkintest"/>
      <exclude name="broken"/>
    </run>
  </groups>

  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

This way, I will get a clean test run while keeping track of what tests are broken and need to be fixed later.

Info

you can also disable tests on an individual basis by using the "enabled" property available on both @Test and @Before/After annotations.

Partial groups

You can define groups at the class level and then add groups at the method level:

@Test(groups = { "checkin-test" })
public class All {

  @Test(groups = { "func-test" )
  public void method1() { ... }

  public void method2() { ... }
}

In this class, method2() is part of the group "checkin-test", which is defined at the class level, while method1() belongs to both "checkin-test" and "func-test".

Parameters

Test methods don't have to be parameterless. You can use an arbitrary number of parameters on each of your test method, and you instruct TestNG to pass you the correct parameters with the @Parameters annotation.

There are two ways to set these parameters: with testng.xml or programmatically.

Parameters from testng.xml

If you are using simple values for your parameters, you can specify them in your testng.xml:

@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) {
  System.out.println("Invoked testString " + firstName);
  assert "Cedric".equals(firstName);
}

In this code, we specify that the parameter firstName of your Java method should receive the value of the XML parameter called first-name. This XML parameter is defined in testng.xml:

<suite name="My suite">
  <parameter name="first-name"  value="Cedric"/>
  <test name="Simple example">
  <-- ... -->

The same technique can be used for @Before/After and @Factory annotations:

@Parameters({ "datasource", "jdbcDriver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
  m_dataSource = ...;    // look up the value of datasource
  m_jdbcDriver = driver;
}

This time, the two Java parameter ds and driver will receive the value given to the properties datasource and jdbc-driver respectively.

Parameters can be declared optional with the Optional annotation:

@Parameters("db")
@Test
public void testNonExistentParameter(@Optional("mysql") String db) { ... }

If no parameter named "db" is found in your testng.xml file, your test method will receive the default value specified inside the @Optional annotation: "mysql".

The @Parameters annotation can be placed at the following locations:

  • On any method that already has a @Test, @Before/After or @Factory annotation.
  • On at most one constructor of your test class. In this case, TestNG will invoke this particular constructor with the parameters initialized to the values specified in testng.xml whenever it needs to instantiate your test class. This feature can be used to initialize fields inside your classes to values that will then be used by your test methods.

Info

The XML parameters are mapped to the Java parameters in the same order as they are found in the annotation, and TestNG will issue an error if the numbers don't match. Parameters are scoped. In testng.xml, you can declare them either under a <suite> tag or under <test>. If two parameters have the same name, it's the one defined in <test> that has precedence. This is convenient if you need to specify a parameter applicable to all your tests and override its value only for certain tests.

Parameters with DataProviders

Specifying parameters in testng.xml might not be sufficient if you need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc...).

In this case, you can use a Data Provider to supply the values you need to test. A Data Provider is a method on your class that returns an array of array of objects. This method is annotated with @DataProvider:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)},
 };
}

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
 System.out.println(n1 + " " + n2);
}

will print

Cedric 36
Anne 37

A @Test method specifies its Data Provider with the dataProvider attribute. This name must correspond to a method on the same class annotated with @DataProvider(name="...") with a matching name.

By default, the data provider will be looked for in the current test class or one of its base classes. If you want to put your data provider in a different class, it needs to be a static method or a class with a non-arg constructor, and you specify the class where it can be found in the dataProviderClass attribute:

public class StaticProvider {
  @DataProvider(name = "create")
  public static Object[][] createData() {
    return new Object[][] {
      new Object[] { new Integer(42) }
    };
  }
}

public class MyTest {
  @Test(dataProvider = "create", dataProviderClass = StaticProvider.class)
  public void test(Integer n) {
    // ...
  }
}

The data provider supports injection too. TestNG will use the test context for the injection. The Data Provider method can return one of the following types:

  • An array of array of objects (Object[][]) where the first dimension's size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method. This is the case illustrated by the example above.
  • An Iterator<Object[]>. The only difference with Object[][] is that an Iterator lets you create your test data lazily. TestNG will invoke the iterator and then the test method with the parameters returned by this iterator one by one. This is particularly useful if you have a lot of parameter sets to pass to the method and you don't want to create all of them upfront.
  • An array of objects (Object[]). This is similar to Iterator<Object[]> but causes the test method to be invoked once for each element of the source array.
  • An Iterator<Object>. Lazy alternative of Object[]. Causes the test method to be invoked once for each element of the iterator.

It must be said that return type is not limited to Object only thus MyCustomData[][] or Iterator<Supplier> are also possible. The only limitation is that in case of iterator its parameter type can't be explicitly parametrized itself. Here is an example of this feature:

@DataProvider(name = "test1")
public Iterator<Object[]> createData() {
  return new MyIterator(DATA);
}

Using MyCustomData[] as a return type

@DataProvider(name = "test1")
public MyCustomData[] createData() {
  return new MyCustomData[]{ new MyCustomData() };
}

Or its lazy option with Iterator<MyCustomData>

@DataProvider(name = "test1")
public Iterator<MyCustomData> createData() {
  return Arrays.asList(new MyCustomData()).iterator();
}

Parameter type (Stream) of Iterator can't be explicitly parametrized

@DataProvider(name = "test1")
public Iterator<Stream> createData() {
  return Arrays.asList(Stream.of("a", "b", "c")).iterator();
}

If you declare your @DataProvider as taking a java.lang.reflect.Method as first parameter, TestNG will pass the current test method for this first parameter. This is particularly useful when several test methods use the same @DataProvider and you want it to return different values depending on which test method it is supplying data for.

For example, the following code prints the name of the test method inside its @DataProvider:

@DataProvider(name = "dp")
public Object[][] createData(Method m) {
  System.out.println(m.getName());  // print test method name
  return new Object[][] { new Object[] { "Cedric" }};
}

@Test(dataProvider = "dp")
public void test1(String s) {
}

@Test(dataProvider = "dp")
public void test2(String s) {
}

and will therefore display:

test1
test2

Data providers can run in parallel with the attribute parallel:

@DataProvider(parallel = true)
// ...

Parallel data providers running from an XML file share the same pool of threads, which has a size of 10 by default. You can modify this value in the <suite> tag of your XML file:

<suite name="Suite1" data-provider-thread-count="20" >
...

If you want to run a few specific data providers in a different thread pool, you need to run them from a different XML file.

Parameters in reports

Parameters used to invoke your test methods are shown in the HTML reports generated by TestNG. Here is an example:

test.dataprovider.Sample1Test.verifyNames(java.lang.String,java.lang.Integer)
// Parameters Cedric, 36 
test.dataprovider.Sample1Test.verifyNames(java.lang.String,java.lang.Integer)
// Parameters Anne Marie , 37

Dependencies

Sometimes, you need your test methods to be invoked in a certain order. Here are a few examples:

To make sure a certain number of test methods have completed and succeeded before running more test methods.

To initialize your tests while wanting this initialization methods to be test methods as well (methods tagged with @Before/ After will not be part of the final report).

TestNG allows you to specify dependencies either with annotations or in XML.

Dependencies with annotations

You can use the attributes dependsOnMethods or dependsOnGroups, found on the @Test annotation.

There are two kinds of dependencies:

  • Hard dependencies. All the methods you depend on must have run and succeeded for you to run. If at least one failure occurred in your dependencies, you will not be invoked and marked as a SKIP in the report.
  • Soft dependencies. You will always be run after the methods you depend on, even if some of them have failed. This is useful when you just want to make sure that your test methods are run in a certain order but their success doesn't really depend on the success of others. A soft dependency is obtained by adding "alwaysRun=true" in your @Test annotation.

Here is an example of a hard dependency:

@Test
public void serverStartedOk() {}

@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}

In this example, method1() is declared as depending on method serverStartedOk(), which guarantees that serverStartedOk() will always be invoked first.

You can also have methods that depend on entire groups:

@Test(groups = { "init" })
public void serverStartedOk() {}

@Test(groups = { "init" })
public void initEnvironment() {}

@Test(dependsOnGroups = { "init.*" })
public void method1() {}

In this example, method1() is declared as depending on any group matching the regular expression "init.*", which guarantees that the methods serverStartedOk() and initEnvironment() will always be invoked before method1().

Summary

As stated before, the order of invocation for methods that belong in the same group is not guaranteed to be the same across test runs.

If a method depended upon fails and you have a hard dependency on it (alwaysRun=false, which is the default), the methods that depend on it are not marked as FAIL but as SKIP. Skipped methods will be reported as such in the final report (in a color that is neither red nor green in HTML), which is important since skipped methods are not necessarily failures.

Both dependsOnGroups and dependsOnMethods accept regular expressions as parameters. For dependsOnMethods, if you are depending on a method which happens to have several overloaded versions, all the overloaded methods will be invoked. If you only want to invoke one of the overloaded methods, you should use dependsOnGroups.

For a more advanced example of dependent methods, please refer to this article, which uses inheritance to provide an elegant solution to the problem of multiple dependencies.

By default, dependent methods are grouped by class. For example, if method b() depends on method a() and you have several instances of the class that contains these methods (because of a factory of a data provider), then the invocation order will be as follows:

a(1)
a(2)
b(2)
b(2)

TestNG will not run b() until all the instances have invoked their a() method.

This behavior might not be desirable in certain scenarios, such as for example testing a sign in and sign out of a web browser for various countries. In such a case, you would like the following ordering:

signIn("us")
signOut("us")
signIn("uk")
signOut("uk")

For this ordering, you can use the XML attribute group-by-instances. This attribute is valid either on <suite> or <test>:

<suite name="Factory" group-by-instances="true">
or
  <test name="Factory" group-by-instances="true">

Dependencies in XML

Alternatively, you can specify your group dependencies in the testng.xml file. You use the <dependencies> tag to achieve this:

<test name="My suite">
  <groups>
    <dependencies>
      <group name="c" depends-on="a  b" />
      <group name="z" depends-on="c" />
    </dependencies>
  </groups>
</test>

The <depends-on> attribute contains a space-separated list of groups.

Factories

Factories allow you to create tests dynamically. For example, imagine you want to create a test method that will access a page on a Web site several times, and you want to invoke it with different values:

public class TestWebServer {
  @Test(parameters = { "number-of-times" })
  public void accessPage(int numberOfTimes) {
    while (numberOfTimes-- > 0) {
     // access the web page
    }
  }
}

<test name="T1">
  <parameter name="number-of-times" value="10"/>
  <classes>
    <class name= "TestWebServer" />
  </classes>
</test>

<test name="T2">
  <parameter name="number-of-times" value="20"/>
  <classes>
    <class name= "TestWebServer"/>
  </classes>
</test>

<test name="T3">
  <parameter name="number-of-times" value="30"/>
  <classes>
    <class name= "TestWebServer"/>
  </classes>
</test>

This can become quickly impossible to manage, so instead, you should use a factory:

public class WebTestFactory {
  @Factory
  public Object[] createInstances() {
   Object[] result = new Object[10]; 
   for (int i = 0; i < 10; i++) {
      result[i] = new WebTest(i * 10);
    }
    return result;
  }
}

and the new test class is now:

public class WebTest {
  private int m_numberOfTimes;
  public WebTest(int numberOfTimes) {
    m_numberOfTimes = numberOfTimes;
  }

  @Test
  public void testServer() {
   for (int i = 0; i < m_numberOfTimes; i++) {
     // access the web page
    }
  }
}

Your testng.xml only needs to reference the class that contains the factory method, since the test instances themselves will be created at runtime:

<class name="WebTestFactory" />

Or, if building a test suite instance programatically, you can add the factory in the same manner as for tests:

TestNG testNG = new TestNG();
testNG.setTestClasses(WebTestFactory.class);
testNG.run();

The factory method can receive parameters just like @Test and @Before/After and it must return Object[]. The objects returned can be of any class (not necessarily the same class as the factory class) and they don't even need to contain TestNG annotations (in which case they will be ignored by TestNG).

Factories can also be used with data providers, and you can leverage this functionality by putting the @Factory annotation either on a regular method or on a constructor. Here is an example of a constructor factory:

@Factory(dataProvider = "dp")
public FactoryDataProviderSampleTest(int n) {
  super(n);
}

@DataProvider
static public Object[][] dp() {
  return new Object[][] {
    new Object[] { 41 },
    new Object[] { 42 },
  };
}

The example will make TestNG create two test classes, on with the constructor invoked with the value 41 and the other with 42.

Class level annotations

The @Test annotation can be put on a class instead of a test method:

@Test
public class Test1 {
  public void test1() {
  }

  public void test2() {
  }
}

The effect of a class level @Test annotation is to make all the public methods of this class to become test methods even if they are not annotated. You can still repeat the @Test annotation on a method if you want to add certain attributes.

For example:

@Test
public class Test1 {
  public void test1() {
  }

  @Test(groups = "g1")
  public void test2() {
  }
}

will make both test1() and test2() test methods but on top of that, test2() now belongs to the group "g1".

Ignoring tests

TestNG lets you ignore all the @Test methods :

  • In a class (or)
  • In a particular package (or)
  • In a package and all of its child packages

using the new annotation @Ignore.

When used at the method level @Ignore annotation is functionally equivalent to @Test(enabled=false). Here's a sample that shows how to ignore all tests within a class.

import org.testng.annotations.Ignore;
import org.testng.annotations.Test;

@Ignore
public class TestcaseSample {

    @Test
    public void testMethod1() {
    }

    @Test
    public void testMethod2() {
    }
}

The @Ignore annotation has a higher priority than individual @Test method annotations. When @Ignore is placed on a class, all the tests in that class will be disabled.

To ignore all tests in a particular package, you just need to create package-info.java and add the @Ignore annotation to it. Here's a sample :

@Ignore
package com.testng.master;

import org.testng.annotations.Ignore;

This causes all the @Test methods to be ignored in the package com.testng.master and all of its sub-packages.

Parallelism and time-outs

You can instruct TestNG to run your tests in separate threads in various ways.

Parallel suites

This is useful if you are running several suite files (e.g. "java org.testng.TestNG testng1.xml testng2.xml") and you want each of these suites to be run in a separate thread. You can use the following command line flag to specify the size of a thread pool:

java org.testng.TestNG -suitethreadpoolsize 3 testng1.xml testng2.xml testng3.xml

The corresponding ant task name is suitethreadpoolsize.

Parallel tests, classes and methods

The parallel attribute on the tag can take one of following values:

<suite name="My suite" parallel="methods" thread-count="5">
<suite name="My suite" parallel="tests" thread-count="5">
<suite name="My suite" parallel="classes" thread-count="5">
<suite name="My suite" parallel="instances" thread-count="5">
  • parallel="methods": TestNG will run all your test methods in separate threads. Dependent methods will also run in separate threads but they will respect the order that you specified.
  • parallel="tests": TestNG will run all the methods in the same <test> tag in the same thread, but each <test> tag will be in a separate thread. This allows you to group all your classes that are not thread safe in the same <test> and guarantee they will all run in the same thread while taking advantage of TestNG using as many threads as possible to run your tests.
  • parallel="classes": TestNG will run all the methods in the same class in the same thread, but each class will be run in a separate thread.
  • parallel="instances": TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads.

Additionally, the attribute thread-count allows you to specify how many threads should be allocated for this execution.

Note

The @Test attribute timeOut works in both parallel and non-parallel mode.

You can also specify that a @Test method should be invoked from different threads. You can use the attribute threadPoolSize to achieve this result:

@Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
public void testServer() {

In this example, the function testServer will be invoked ten times from three different threads. Additionally, a time-out of ten seconds guarantees that none of the threads will block on this thread forever.

Rerunning failed tests

Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests. Therefore, a typical session would look like this:

java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs testng.xml
java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs test-outputs\testng-failed.xml

Warning

Note that testng-failed.xml will contain all the necessary dependent methods so that you are guaranteed to run the methods that failed without any SKIP failures.

Sometimes, you might want TestNG to automatically retry a test whenever it fails. In those situations, you can use a retry analyzer. When you bind a retry analyzer to a test, TestNG automatically invokes the retry analyzer to determine if TestNG can retry a test case again in an attempt to see if the test that just fails now passes.

Here is how you use a retry analyzer:

  1. Build an implementation of the interface org.testng.IRetryAnalyzer
  2. Bind this implementation to the @Test annotation for e.g., @Test(retryAnalyzer = LocalRetry.class)

Following is a sample implementation of the retry analyzer that retries a test for a maximum of three times.

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class MyRetry implements IRetryAnalyzer {

  private int retryCount = 0;
  private static final int maxRetryCount = 3;

  @Override
  public boolean retry(ITestResult result) {
    if (retryCount < maxRetryCount) {
      retryCount++;
      return true;
    }
    return false;
  }
}
import org.testng.Assert;
import org.testng.annotations.Test;

public class TestclassSample {

  @Test(retryAnalyzer = MyRetry.class)
  public void test2() {
    Assert.fail();
  }
}

JUnit tests

TestNG can run JUnit 3 and JUnit 4 tests. All you need to do is put the JUnit jar file on the classpath, specify your

JUnit test classes in the testng.classNames property and set the testng.junit property to true:

<test name="Test1" junit="true">
  <classes>
    <!-- ... -->

The behavior of TestNG in this case is similar to JUnit depending on the JUnit version found on the class path:

JUnit 3:

All methods starting with test* in your classes will be run * If there is a method setUp() on your test class, it will be invoked before every test method. * If there is a method tearDown() on your test class, it will be invoked before after every test method. * If your test class contains a method suite(), all the tests returned by this method will be invoked.

JUnit 4:

  • TestNG will use the org.junit.runner.JUnitCore runner to run your tests.

Running TestNG programmatically

You can invoke TestNG from your own programs very easily:

TestListenerAdapter tla = new TestListenerAdapter();
TestNG testng = new TestNG();
testng.setTestClasses(new Class[] { Run2.class });
testng.addListener(tla);
testng.run();

This example creates a TestNG object and runs the test class Run2. It also adds a TestListener. You can either use the adapter class org.testng.TestListenerAdapter or implement org.testng.ITestListener yourself. This interface contains various callback methods that let you keep track of when a test starts, succeeds, fails, etc...

Similarly, you can invoke TestNG on a testng.xml file or you can create a virtual testng.xml file yourself. In order to do this, you can use the classes found the package org.testng.xml: XmlClass, XmlTest, etc... Each of these classes correspond to their XML tag counterpart.

For example, suppose you want to create the following virtual file:

<suite name="TmpSuite" >
  <test name="TmpTest" >
    <classes>
      <class name="test.failures.Child"  />
    <classes>
    </test>
</suite>

You would use the following code:

XmlSuite suite = new XmlSuite();
suite.setName("TmpSuite");

XmlTest test = new XmlTest(suite);
test.setName("TmpTest");
List<XmlClass> classes = new ArrayList<XmlClass>();
classes.add(new XmlClass("test.failures.Child"));
test.setXmlClasses(classes) ;

And then you can pass this XmlSuite to TestNG:

List<XmlSuite> suites = new ArrayList<XmlSuite>();
suites.add(suite);
TestNG tng = new TestNG();
tng.setXmlSuites(suites);
tng.run();

Please see the JavaDocs for the entire API.

BeanShell and advanced group selection

If the <include> and <exclude> tags in testng.xml are not enough for your needs, you can use a BeanShell expression to decide whether a certain test method should be included in a test run or not. You specify this expression just under the <test> tag:

<test name="BeanShell test">
   <method-selectors>
     <method-selector>
       <script language="beanshell"><![CDATA[
         groups.containsKey("test1")
       ]]></script>
     </method-selector>
   </method-selectors>
  <!-- ... -->

When a <script> tag is found in testng.xml, TestNG will ignore subsequent <include> and <exclude> of groups and methods in the current <test> tag: your BeanShell expression will be the only way to decide whether a test method is included or not.

Here are additional information on the BeanShell script:

  • It must return a boolean value. Except for this constraint, any valid BeanShell code is allowed (for example, you might want to return true during week days and false during weekends, which would allow you to run tests differently depending on the date).

  • TestNG defines the following variables for your convenience:

  • java.lang.reflect.Method method: the current test method.
  • org.testng.ITestNGMethod testngMethod: the description of the current test method.
  • java.util.Map<String, String> groups: a map of the groups the current test method belongs to.

You might want to surround your expression with a CDATA declaration (as shown above) to avoid tedious quoting of reserved XML characters).

Annotation Transformers

TestNG allows you to modify the content of all the annotations at runtime. This is especially useful if the annotations in the source code are right most of the time, but there are a few situations where you'd like to override their value. In order to achieve this, you need to use an Annotation Transformer.

An Annotation Transformer is a class that implements the following interface:

public interface IAnnotationTransformer {

  /**
   * This method will be invoked by TestNG to give you a chance
   * to modify a TestNG annotation read from your test classes.
   * You can change the values you need by calling any of the
   * setters on the ITest interface.
   *
   * Note that only one of the three parameters testClass,
   * testConstructor and testMethod will be non-null.
   *
   * @param annotation The annotation that was read from your
   * test class.
   * @param testClass If the annotation was found on a class, this
   * parameter represents this class (null otherwise).
   * @param testConstructor If the annotation was found on a constructor,
   * this parameter represents this constructor (null otherwise).
   * @param testMethod If the annotation was found on a method,
   * this parameter represents this method (null otherwise).
   */
  public void transform(ITest annotation, Class testClass,
      Constructor testConstructor, Method testMethod);
}

Like all the other TestNG listeners, you can specify this class either on the command line or with ant:

java org.testng.TestNG -listener MyTransformer testng.xml

or programmatically:

TestNG tng = new TestNG();
tng.setAnnotationTransformer(new MyTransformer());
// ...'

When the method transform() is invoked, you can call any of the setters on the ITest test parameter to alter its value before TestNG proceeds further.

For example, here is how you would override the attribute invocationCount but only on the test method invoke() of one of your test classes:

public class MyTransformer implements IAnnotationTransformer {
  public void transform(ITest annotation, Class testClass,
      Constructor testConstructor, Method testMethod)
  {
    if ("invoke".equals(testMethod.getName())) {
      annotation.setInvocationCount(5);
    }
  }
}

IAnnotationTransformer only lets you modify a @Test annotation. If you need to modify another TestNG annotation (a configuration annotation, @Factory or @DataProvider), use an IAnnotationTransformer2.

Method Interceptors

Once TestNG has calculated in what order the test methods will be invoked, these methods are split in two groups: Methods run sequentially. These are all the test methods that have dependencies or dependents. These methods will be run in a specific order. Methods run in no particular order. These are all the methods that don't belong in the first category. The order in which these test methods are run is random and can vary from one run to the next (although by default, TestNG will try to group test methods by class). In order to give you more control on the methods that belong to the second category, TestNG defines the following interface:

public interface IMethodInterceptor {

  List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context);

}

The list of methods passed in parameters are all the methods that can be run in any order. Your intercept method is expected to return a similar list of IMethodInstance, which can be either of the following:

  • The same list you received in parameter but in a different order.
  • A smaller list of IMethodInstance objects.
  • A bigger list of IMethodInstance objects.

Once you have defined your interceptor, you pass it to TestNG as a listener. For example:

java -classpath "testng-jdk15.jar:test/build" org.testng.TestNG -listener test.methodinterceptors.NullMethodInterceptor
   -testclass test.methodinterceptors.FooTest

For the equivalent ant syntax, see the listeners attribute in the ant documentation.

For example, here is a Method Interceptor that will reorder the methods so that test methods that belong to the group "fast" are always run first:

public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
  List<IMethodInstance> result = new ArrayList<IMethodInstance>();
  for (IMethodInstance m : methods) {
    Test test = m.getMethod().getConstructorOrMethod().getAnnotation(Test.class);
    Set<String> groups = new HashSet<String>();
    for (String group : test.groups()) {
      groups.add(group);
    }
    if (groups.contains("fast")) {
      result.add(0, m);
    }
    else {
      result.add(m);
    }
  }
  return result;
}

TestNG Listeners

There are several interfaces that allow you to modify TestNG's behavior. These interfaces are broadly called "TestNG Listeners". Here are a few listeners:

When you implement one of these interfaces, you can let TestNG know about it with either of the following ways:

  • Using -listener on the command line.
  • Using <listeners> with ant.
  • Using <listeners> in your testng.xml file.
  • Using the @Listeners annotation on any of your test classes.
  • Using ServiceLoader.

Specifying listeners with testng.xml or in Java

Here is how you can define listeners in your testng.xml file:

<suite>

  <listeners>
    <listener class-name="com.example.MyListener" />
    <listener class-name="com.example.MyMethodInterceptor" />
  </listeners>

...

Or if you prefer to define these listeners in Java:

@Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class })
public class MyTest {
  // ...
}

The @Listeners annotation can contain any class that extends org.testng.ITestNGListener except IAnnotationTransformer and IAnnotationTransformer2. The reason is that these listeners need to be known very early in the process so that TestNG can use them to rewrite your annotations, therefore you need to specify these listeners in your testng.xml file.

Warning

The @Listeners annotation will apply to your entire suite file, just as if you had specified it in a testng.xml file. If you want to restrict its scope (for example, only running on the current class), the code in your listener could first check the test method that's about to run and decide what to do then. Here's how it can be done.

First define a new custom annotation that can be used to specify this restriction:

@Retention(RetentionPolicy.RUNTIME)
@Target ({ElementType.TYPE})
public @interface DisableListener {}

Add an edit check as below within your regular listeners:

public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
  ConstructorOrMethod consOrMethod =iInvokedMethod.getTestMethod().getConstructorOrMethod();
  DisableListener disable = consOrMethod.getMethod().getDeclaringClass().getAnnotation(DisableListener.class);
  if (disable != null) {
    return;
  }
  // else resume your normal operations
}

Annotate test classes wherein the listener is not to be invoked:

@DisableListener
@Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class })
public class MyTest {
  // ...
}

Specifying listeners with ServiceLoader

Finally, the JDK offers a very elegant mechanism to specify implementations of interfaces on the class path via the ServiceLoader class.

With ServiceLoader, all you need to do is create a jar file that contains your listener(s) and a few configuration files, put that jar file on the classpath when you run TestNG and TestNG will automatically find them.

Here is a concrete example of how it works.

Let's start by creating a listener (any TestNG listener should work):

package test.tmp;

public class TmpSuiteListener implements ISuiteListener {
  @Override
  public void onFinish(ISuite suite) {
    System.out.println("Finishing");
  }

  @Override
  public void onStart(ISuite suite) {
    System.out.println("Starting");
  }
}

Compile this file, then create a file at the location META-INF/services/org.testng.ITestNGListener, which will name the implementation(s) you want for this interface.

You should end up with the following directory structure, with only two files:

$ tree
|____META-INF
| |____services
| | |____org.testng.ITestNGListener
|____test
| |____tmp
| | |____TmpSuiteListener.class
$ cat META-INF/services/org.testng.ITestNGListener
test.tmp.TmpSuiteListener

Create a jar of this directory:

$ jar cvf ../sl.jar .
added manifest
ignoring entry META-INF/
adding: META-INF/services/(in = 0) (out= 0)(stored 0%)
adding: META-INF/services/org.testng.ITestNGListener(in = 26) (out= 28)(deflated -7%)
adding: test/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/TmpSuiteListener.class(in = 849) (out= 470)(deflated 44%)

Next, put this jar file on your classpath when you invoke TestNG:

$ java -classpath sl.jar:testng.jar org.testng.TestNG testng-single.yaml
Starting
f2 11 2
PASSED: f2("2")
Finishing

This mechanism allows you to apply the same set of listeners to an entire organization just by adding a jar file to the classpath, instead of asking every single developer to remember to specify these listeners in their testng.xml file.

Dependency injection

TestNG supports two different kinds of dependency injection: native (performed by TestNG itself) and external (performed by a dependency injection framework such as Guice).

Native dependency injection

TestNG lets you declare additional parameters in your methods. When this happens, TestNG will automatically fill these parameters with the right value.

Dependency injection can be used in the following places:

  • Any @Before method or @Test method can declare a parameter of type ITestContext.
  • Any @AfterMethod method can declare a parameter of type ITestResult, which will reflect the result of the test method that was just run.
  • Any @Before and @After methods (except @BeforeSuite and @AfterSuite) can declare a parameter of type XmlTest, which contain the current <test> tag.
  • Any @BeforeMethod (and @AfterMethod) can declare a parameter of type java.lang.reflect.Method. This parameter will receive the test method that will be called once this @BeforeMethod finishes (or after the method as run for @AfterMethod).
  • Any @BeforeMethod can declare a parameter of type Object[]. This parameter will receive the list of parameters that are about to be fed to the upcoming test method, which could be either injected by TestNG, such as java.lang.reflect.Method or come from a @DataProvider.
  • Any @DataProvider can declare a parameter of type ITestContext or java.lang.reflect.Method. The latter parameter will receive the test method that is about to be invoked.

You can turn off injection with the @NoInjection annotation:

public class NoInjectionTest {

  @DataProvider(name = "provider")
  public Object[][] provide() throws Exception {
      return new Object[][] { { CC.class.getMethod("f") } };
  }

  @Test(dataProvider = "provider")
  public void withoutInjection(@NoInjection Method m) {
      Assert.assertEquals(m.getName(), "f");
  }

  @Test(dataProvider = "provider")
  public void withInjection(Method m) {
      Assert.assertEquals(m.getName(), "withInjection");
  }
}

The below table summarises the parameter types that can be natively injected for the various TestNG annotations:

Annotation ITestContext XmlTest Method Object[] ITestResult
BeforeSuite Yes No No No No
BeforeTest Yes Yes No No No
BeforeGroups Yes Yes No No No
BeforeClass Yes Yes No No No
BeforeMethod Yes Yes Yes Yes Yes
Test Yes No No No No
DataProvider Yes No Yes No No
AfterMethod Yes Yes Yes Yes Yes
AfterClass Yes Yes No No No
AfterGroups Yes Yes No No No
AfterTest Yes Yes No No No
AfterSuite Yes No No No No

Guice dependency injection

If you use Guice, TestNG gives you an easy way to inject your test objects with a Guice module:

@Guice(modules = GuiceExampleModule.class)
public class GuiceTest extends SimpleBaseTest {

  @Inject
  ISingleton m_singleton;

  @Test
  public void singletonShouldWork() {
    m_singleton.doSomething();
  }

}

In this example, GuiceExampleModule is expected to bind the interface ISingleton to some concrete class:

public class GuiceExampleModule implements Module {

  @Override
  public void configure(Binder binder) {
    binder.bind(ISingleton.class).to(ExampleSingleton.class).in(Singleton.class);
  }

}

If you need more flexibility in specifying which modules should be used to instantiate your test classes, you can specify a module factory:

@Guice(moduleFactory = ModuleFactory.class)
public class GuiceModuleFactoryTest {

  @Inject
  ISingleton m_singleton;

  @Test
  public void singletonShouldWork() {
    m_singleton.doSomething();
  }
}

The module factory needs to implement the interface IModuleFactory:

public interface IModuleFactory {
 /**
   * @param context The current test context
   * @param testClass The test class
   *
   * @return The Guice module that should be used to get an instance of this
   * test class.
   */
  Module createModule(ITestContext context, Class<?> testClass);
}

Your factory will be passed an instance of the test context and the test class that TestNG needs to instantiate. Your createModule method should return a Guice Module that will know how to instantiate this test class. You can use the test context to find out more information about your environment, such as parameters specified in testng.xml, etc... You will get even more flexibility and Guice power with parent-module and guice-stage suite parameters. guice-stage allow you to chose the Stage used to create the parent injector. The default one is DEVELOPMENT. Other allowed values are PRODUCTION and TOOL.

Here is how you can define parent-module in your test.xml file:

<suite parent-module="com.example.SuiteParenModule" guice-stage="PRODUCTION">
</suite>

TestNG will create this module only once for given suite. Will also use this module for obtaining instances of test specific Guice modules and module factories, then will create child injector for each test class. With such approach you can declare all common bindings in parent-module also you can inject binding declared in parent-module in module and module factory.

Here is an example of this functionality:

package com.example;

public class ParentModule extends AbstractModule {
  @Override
  protected void conigure() {
    bind(MyService.class).toProvider(MyServiceProvider.class);
    bind(MyContext.class).to(MyContextImpl.class).in(Singleton.class);
  }
}
package com.example;

public class TestModule extends AbstractModule {
  private final MyContext myContext;

  @Inject
  TestModule(MyContext myContext) {
    this.myContext = myContext
  }

  @Override
  protected void configure() {
    bind(MySession.class).toInstance(myContext.getSession());
  }
}

<suite parent-module="com.example.ParentModule">
</suite>

package com.example;

@Test
@Guice(modules = TestModule.class)
public class TestClass {
  @Inject
  MyService myService;
  @Inject
  MySession mySession;

  public void testServiceWithSession() {
    myService.serve(mySession);
  }
}

As you see ParentModule declares binding for MyService and MyContext classes. Then MyContext is injected using constructor injection into TestModule class, which also declare binding for MySession. Then parent-module in test XML file is set to ParentModule class, this enables injection in TestModule. Later in TestClass you see two injections: * MyService - binding taken from ParentModule * MySession - binding taken from TestModule This configuration ensures you that all tests in this suite will be run with same session instance, the MyContextImpl object is only created once per suite, this give you possibility to configure common environment state for all tests in suite.

Listening to method invocations

The listener IInvokedMethodListener allows you to be notified whenever TestNG is about to invoke a test (annotated with @Test) or configuration (annotated with any of the @Before or @After annotation) method.

You need to implement the following interface:

public interface IInvokedMethodListener extends ITestNGListener {
  void beforeInvocation(IInvokedMethod method, ITestResult testResult);
  void afterInvocation(IInvokedMethod method, ITestResult testResult);
}

and declare it as a listener, as explained in the section about TestNG listeners.

Overriding test methods

TestNG allows you to override and possibly skip the invocation of test methods. One example of where this is useful is if you need to your test methods with a specific security manager. You achieve this by providing a listener that implements IHookable.

Here is an example with JAAS:

public class MyHook implements IHookable {
  public void run(final IHookCallBack icb, ITestResult testResult) {
    // Preferably initialized in a @Configuration method
    mySubject = authenticateWithJAAs();

    Subject.doAs(mySubject, new PrivilegedExceptionAction() {
      public Object run() {
        icb.callback(testResult);
      }
    };
  }
}

Altering suites (or) tests

Sometimes you may need to just want to alter a suite (or) a test tag in a suite xml in runtime without having to change the contents of a suite file.

A classic example for this would be to try and leverage your existing suite file and try using it for simulating a load test on your "Application under test". At the minimum you would end up duplicating the contents of your <test> tag multiple times and create a new suite xml file and work with. But this doesn't seem to scale a lot.

TestNG allows you to alter a suite (or) a test tag in your suite xml file at runtime via listeners. You achieve this by providing a listener that implements IAlterSuiteListener. Please refer to Listeners section to learn about listeners.

Here is an example that shows how the suite name is getting altered in runtime:

public class AlterSuiteNameListener implements IAlterSuiteListener {

    @Override
    public void alter(List<XmlSuite> suites) {
        XmlSuite suite = suites.get(0);
        suite.setName(getClass().getSimpleName());
    }
}

This listener can only be added with either of the following ways:

  • Through the <listeners> tag in the suite xml file.
  • Through a Service Loader

This listener cannot be added to execution using the @Listeners annotation.

Run TestNG

TestNG can be invoked in different ways:

This section only explains how to invoke TestNG from the command line. Please click on one of the links above if you are interested in one of the other ways.

Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows:

java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...]

You need to specify at least one XML file describing the TestNG suite you are trying to run. Additionally, the following command-line switches are available:

Command Line Parameters

Option Argument Documentation
-configfailurepolicy skip/continue Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. Default behavior is skip.
-d A directory The directory where the reports will be generated (defaults to test-output).
-dataproviderthreadcount The default number of threads to use for data providers when running tests in parallel. This sets the default maximum number of threads to use for data providers when running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.
-excludegroups A comma-separated list of groups. The list of groups you want to be excluded from this run.
-groups A comma-separated list of groups. The list of groups you want to run (e.g. "windows,linux,regression").
-listener A comma-separated list of Java classes that can be found on your classpath. Lets you specify your own test listeners. The classes need to implement org.testng.ITestListener
-usedefaultlisteners true/false Whether to use the default listeners
-methods A comma separated list of fully qualified class name and method. For example com.example.Foo.f1,com.example.Bar.f2. Lets you specify individual methods to run.
-methodselectors A comma-separated list of Java classes and method priorities that define method selectors. Lets you specify method selectors on the command line. For example: com.example.Selector1:3,com.example.Selector2:2
-parallel methods/tests/classes If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.
-reporter The extended configuration for a custom report listener. Similar to the -listener option, except that it allows the configuration of JavaBeans-style properties on the reporter instance.
Example: -reporter com.test.MyReporter:methodFilter=insert,enableFiltering=true
You can have as many occurrences of this option, one for each reporter that needs to be added.
-sourcedir A semi-colon separated list of directories. The directories where your javadoc annotated test sources are. This option is only necessary if you are using javadoc type annotations. (e.g. "src/test" or "src/test/org/testng/eclipse-plugin;src/test/org/testng/testng").
-suitename The default name to use for a test suite. This specifies the suite name for a test suite defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different suite name. It is possible to create a suite name with spaces in it if you surround it with double-quotes "like this".
-testclass A comma-separated list of classes that can be found in your classpath. A list of class files separated by commas (e.g. "org.foo.Test1,org.foo.test2").
-testjar A jar file. Specifies a jar file that contains test classes. If a testng.xml file is found at the root of that jar file, it will be used, otherwise, all the test classes found in this jar file will be considered test classes.
-testname The default name to use for a test. This specifies the name for a test defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different test name. It is possible to create a test name with spaces in it if you surround it with double-quotes "like this".
-testnames A comma separated list of test names. Only tests defined in a <test> tag matching one of these names will be run.
-testrunfactory A Java classes that can be found on your classpath. Lets you specify your own test runners. The class needs to implement org.testng.ITestRunnerFactory.
-threadcount The default number of threads to use when running tests in parallel. This sets the default maximum number of threads to use for running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.
-xmlpathinjar The path of the XML file inside the jar file. This attribute should contain the path to a valid XML file inside the test jar (e.g. "resources/testng.xml"). The default is "testng.xml", which means a file called "testng.xml" at the root of the jar file. This option will be ignored unless -testjar is specified.

This documentation can be obtained by invoking TestNG without any arguments.

You can also put the command line switches in a text file, say c:\command.txt, and tell TestNG to use that file to retrieve its parameters:

C:> more c:\command.txt
-d test-output testng.xml
C:> java org.testng.TestNG @c:\command.txt

Additionally, TestNG can be passed properties on the command line of the Java Virtual Machine, for example

java -Dtestng.test.classpath="c:/build;c:/java/classes;" org.testng.TestNG testng.xml

Here are the properties that TestNG understands:

System properties

Property Type Documentation
testng.test.classpath A semi-colon separated series of directories that contain your test classes. If this property is set, TestNG will use it to look for your test classes instead of the class path. This is convenient if you are using the package tag in your XML file and you have a lot of classes in your classpath, most of them not being test classes.

Example:

java org.testng.TestNG -groups windows,linux -testclass org.test.MyTest

The ant task and testng.xml allow you to launch TestNG with more parameters (methods to include, specifying parameters, etc...), so you should consider using the command line only when you are trying to learn about TestNG and you want to get up and running quickly.

Important: The command line flags that specify what tests should be run will be ignored if you also specify a testng.xml file, with the exception of -includedgroups and -excludedgroups, which will override all the group inclusions/exclusions found in testng.xml.

TestNG XML

You can invoke TestNG in several different ways:

  • With a testng.xml file
  • With ant
  • From the command line

This section describes the format of testng.xml (you will find documentation on ant and the command line below).

The current DTD for testng.xml : testng-1.0.dtd

Here is an example testng.xml file:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >

<suite name="Suite1" verbose="1" >
  <test name="Nopackage" >
    <classes>
       <class name="NoPackageTest" />
    </classes>
  </test>

  <test name="Regression1">
    <classes>
      <class name="test.sample.ParameterSample"/>
      <class name="test.sample.ParameterTest"/>
    </classes>
  </test>
</suite>

You can specify package names instead of class names:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >

<suite name="Suite1" verbose="1" >
  <test name="Regression1"   >
    <packages>
      <package name="test.sample" />
   </packages>
 </test>
</suite>

In this example, TestNG will look at all the classes in the package test.sample and will retain only classes that have TestNG annotations.

You can also specify groups and methods to be included and excluded:

<test name="Regression1">
  <groups>
    <run>
      <exclude name="brokenTests"  />
      <include name="checkinTests"  />
    </run>
  </groups>

  <classes>
    <class name="test.IndividualMethodsTest">
      <methods>
        <include name="testMethod" />
      </methods>
    </class>
  </classes>
</test>

You can also define new groups inside testng.xml and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc...

By default, TestNG will run your tests in the order they are found in the XML file. If you want the classes and methods listed in this file to be run in an unpredictable order, set the preserve-order attribute to false

<test name="Regression1" preserve-order="false">
  <classes>

    <class name="test.Test1">
      <methods>
        <include name="m1" />
        <include name="m2" />
      </methods>
    </class>

    <class name="test.Test2" />

  </classes>
</test>

Please see the DTD for a complete list of the features, or read on.

--8<-- "testresults.md"

YAML

TestNG supports YAML as an alternate way of specifying your suite file. For example, the following XML file:

<suite name="SingleSuite" verbose="2" thread-count="4">

  <parameter name="n" value="42" />

  <test name="Regression2">
    <groups>
      <run>
        <exclude name="broken" />
      </run>
    </groups>

    <classes>
      <class name="test.listeners.ResultEndMillisTest" />
    </classes>
  </test>
</suite>

and here is its YAML version:

name: SingleSuite
threadCount: 4
parameters: { n: 42 }

tests:
  - name: Regression2
    parameters: { count: 10 }
    excludedGroups: [ broken ]
    classes:
      - test.listeners.ResultEndMillisTest

Here is TestNG's own suite file, and its YAML counterpart. You might find the YAML file format easier to read and to maintain. YAML files are also recognized by the TestNG Eclipse plug-in. You can find more information about YAML and TestNG in this blog post.

Info

TestNG by default does not bring in the YAML related library into your classpath. So depending upon your build system (Gradle/Maven) you need to add an explicit reference to YAML library in your build file.

For e.g, If you were using Maven, you would need to add a dependency as below into your pom.xml file:

<dependency>
  <groupid>org.yaml</groupid>
  <artifactid>snakeyaml</artifactid>
  <version>1.23</version>
</dependency>

Or if you were using Gradle, you would add a dependency as below into your build.gradle file:

compile group: 'org.yaml', name: 'snakeyaml', version: '1.23'

Dry Run for your tests

When launched in dry run mode, TestNG will display a list of the test methods that would be invoked but without actually calling them.

You can enable dry run mode for TestNG by passing the JVM argument 

-Dtestng.mode.dryrun=true

Back to my home page.

Or check out some of my other projects:

  • EJBGen: an EJB tag generator.
  • TestNG: A testing framework using annotations, test groups and method parameters.
  • Doclipse: a JavaDoc tag Eclipse plug-in.
  • J15: an Eclipse plug-in to help you migrate your code to the new JDK 1.5 constructs.
  • SGen: a replacement for XDoclet with an easy plug-in architecture.
  • Canvas: a template generator based on the Groovy language.

TestNG Ant Task

You define the TestNG ant task as follows:

<taskdef resource="testngtasks" classpath="testng.jar"/>

This task runs TestNG tests and is always run in a forked JVM. It accepts the following attributes:

Attribute Description Required
classfilesetref A reference to a ResourceCollection containing the test classes to be run. Only File based ResourceCollections are supported (ie. FileSet).
classpath A PATH-like structure for the tests to be run.
classpathref A reference to a PATH-like structure for the tests to be run.
configFailurePolicy Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. No. Defaults to skip
dataProviderThreadCount The number of threads to use for data providers for this run. Ignored unless the parallel mode is also specified 1
delegateCommandSystemProperties Pass the command line properties as system properties. No. Defaults to false
dumpCommand Print the TestNG launcher command. No. Defaults to false
failureProperty The name of a property to set in the event of a failure. It is used only if the haltonfailure is not set. No.
haltonfailure Stop the build process if a failure has occurred during the test run. No. Defaults to false
haltonskipped Stop the build process if there is at least on skipped test. No. Default to false
groups The list of groups to run, separated by spaces or commas.
excludedgroups The list of groups to exclude, separated by spaces or commas
jvm The JVM to use, which will be run by Runtime.exec() java
listeners A comma or space-separated list of fully qualified classes that are TestNG listeners (for example org.testng.ITestListener or org.testng.IReporter) No.
methods A comma separated list of fully qualified class name and method. For example com.example.Foo.f1,com.example.Bar.f2. No.
mode Either "testng", "junit" or "mixed". Whether TestNG should run only TestNG tests, JUnit tests or both. No. Defaults to "testng".
outputdir Directory for reports output. No. Defaults to test-output.
skippedProperty The name of a property to set in the event of a skipped test. It is used only if the haltonskipped is not set. No.
suiteRunnerClass A fully qualified name of a TestNG starter. No. Defaults to org.testng.TestNG
suiteThreadPoolSize The size of a thread pool to run suites. No. Defaults to 1.
parallel The parallel mode to use for running the tests - either methods or tests No - if not present, parallel mode will not be selected
suitename Sets the default name of the test suite, if one is not specified in a suite xml file or in the source code No. Defaults to "Ant suite"
testJar Path to a jar containing tests and a suite definition.
testname Sets the default name of the test, if one is not specified in a suite xml file or in the source code No. defaults to "Ant test"
testnames A comma separated list of test names, as defined in the tag. Only these tests will be run. No. defaults to "Ant test"
threadCount The number of threads to use for this run. Ignored unless the parallel mode is also specified 1
timeOut The maximum time out in milliseconds that all the tests should run under.
useDefaultListeners Whether the default listeners and reporters should be used. Defaults to true.
workingDir The directory where the ant task should change to before running TestNG.
xmlfilesetref A reference to a ResourceCollection containing the test classes to be run. Only File based ResourceCollections are supported (ie. FileSet).
xmlPathInJar The path of the XML file inside the jar file, only applicable if testJar was specified testng.xml

One of attributes classpath, classpathref or nested must be used for providing the tests classpath. One of the attributes xmlfilesetref, classfilesetref or nested , respectively must be used for providing the tests.

TestNG modes

The TestNG mode gets applied when tests are passed to TestNG using classfilesetref, methods or nested and tells TestNG what kind of tests it should look for and run:

  • "testng": find and run TestNG tests.
  • "junit": find and run JUnit tests.
  • "mixed": run both TestNG and JUnit tests.

Note

"junit" and "mixed" modes require the JUnit jar file on the classpath.

Nested Elements

classpath

The task supports a nested element that represents a PATH-like structure.

bootclasspath

The location of bootstrap class files can be specified using this PATH-like structure - will be ignored if fork is not set.

xmlfileset

The suite definitions (testng.xml) can be passed to the task with a FileSet structure.

classfileset

TestNG can also run directly on classes, also supplied with a FileSet structure.

jvmarg

Additional parameters may be passed to the new VM via nested <jvmarg> elements. For example:

<testng>
   <jvmarg value="-Djava.compiler=NONE" />
   <!-- ... -->
</testng>
sysproperty

Use nested <sysproperty> elements to specify system properties required by the class. These properties will be made available to the virtual machine during the execution of the test. The attributes for this element are the same as for

environment variables:
<testng>
   <sysproperty key="basedir" value="${basedir}"/>
   <!-- ... -->
</testng>

will run the test and make the basedir property available to the test.

propertyset

You may also use a nested <propertyset> element to specify a set of system properties that are defined outside of the TestNG ant task. This allows for more flexible definitions of system properties, for instance selecting all properties with a specific prefix or matching a regex. See the PropertySet page in the Ant manual for full details. Here's a simple example:

<property name="myprop1" value="value 1"/>
<property name="myprop2" value="value 2"/>

<propertyset id="propset1">
    <propertyref name="myprop1"/>
    <propertyref name="myprop2"/>
</propertyset>

<testng outputdir="${testng.report.dir}" classpathref="run.cp">
    <xmlfileset dir="${test15.dir}" includes="testng-single3.xml"/>
    <propertyset refid="propset1"/>
</testng>

In this case, the system properties named "myprop1" and "myprop2" are passed along to the TestNG process.

reporter

An inner <reporter> element is an alternative way to inject a custom report listener allowing the user to set custom properties in order to fine-tune the behavior of the reporter at run-time. The element has one classname attribute which is mandatory, indicating the class of the custom listener. In order to set the properties of the reporter, the <reporter> element can contain several nested <property> elements which will provide the name and value attributes as seen below:

<testng ...>
   ...
   <reporter classname="com.test.MyReporter">
      <property name="methodFilter" value="*insert*"/>
      <property name="enableFiltering" value="true"/>
   </reporter>
   ...
</testng>

public class MyReporter {

  public String getMethodFilter() {...}
  public void setMethodFilter(String methodFilter) {...}
  public boolean isEnableFiltering() {...}
  public void setEnableFiltering(boolean enableFiltering) {...}
  ...
}

You have to consider though that for the moment only a limited set of property types are supported: String, int, boolean, byte, char, double, float, long, short.

env

It is possible to specify environment variables to pass to the TestNG forked virtual machine via nested elements. For a description of the <env> element's attributes, see the description in the exec task.

Examples

Suite xml
<testng classpathref="run.cp"
        outputDir="${testng.report.dir}"
        sourcedir="${test.src.dir}"
        haltOnfailure="true">

   <xmlfileset dir="${test14.dir}" includes="testng.xml"/>
</testng>
Class FileSet
<testng classpathref="run.cp"
        outputDir="${testng.report.dir}"
        haltOnFailure="true" verbose="2">
    <classfileset dir="${test.build.dir}" includes="**/*.class" />
</testng>

TestNG Maven plug-ins

Maven 2

Maven 2 supports TestNG out of the box without the need to download any additional plugins (other than TestNG itself). It is recommended that you use version 2.4 or above of the Surefire plugin (this is the case in all recent versions of Maven).

You can find the full instructions on the Maven Surefire Plugin web site. There are also TestNG-specific instructions.

Specifying your pom.xml The dependency in your project should look like the following:

<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>6.8</version>
  <scope>test</scope>
</dependency>

Sample Report

A sample surefire report with TestNG can be found here.

Maven TestNG Archetype (Martin Gilday)

Martin Gilday has added a new archetype for Maven2: to create a project using the archetype you simply have to specify my repository and the archetype ID.

mvn archetype:create -DgroupId=org.martingilday -DartifactId=test1 -DarchetypeGroupId=org.martingilday -DarchetypeArtifactId=testng-archetype
  -DarchetypeVersion=1.0-SNAPSHOT -DremoteRepositories=http://www.martingilday.org/repository/

Of course substitute in your own groupId and artifactId.

Don't forget to keep checking back at Martin's blog for more updates.

Maven 1 (by Andrew Glover)

The TestNG Maven plug-in is quite simple and consists of two goals and a series of optional properties.

Currently the 1.1 version of the plug-in is bundled with official releases of TestNG. To utilize the plug-in, copy the maven-testng-plugin-<version>.jar to the $MAVEN_HOME/plugins directory.

For the latest version of the plug-in (1.2 as of 12/12/05), update your maven.repo.remote to include http://www.vanwardtechnologies.com/repository/ and then issue the following 

command: maven plugin:download.

Maven will issue a series of questions, answer them as follows:

\ \
artifactId: maven-testng-plugin
groupId: testng
version: 1.2

Goals

Goal Description
testng Runs TestNG
testng:junit-report Creates a JUnit style report

Properties

Property Optional? Description
maven.testng.suitexml.name Yes XML file name- defaults to testng.xml
maven.testng.suitexml.dir Yes Directory where XML file lives. Defaults to ${basedir}/test/conf
maven.testng.output.dir Yes Default report directory. Defaults to ${maven.build.dir}/testng-output
maven.testng.report.dir Yes Directory for JUnit reports. Defaults to ${maven.build.dir}/testngJunitReport

Selenium and TestNG

This documentation was written by Felipe Knorr Kuhn and is adapted from a series of articles posted on his blog. Content * How to use TestNG configuration methods with parameters * How to configure your test * Creating the XML file for TestNG * Lauching your tests with Eclipse * How to make the test design a little better for the future

Modeling your test case

Before writing a test case, you need to know how and what will be validated. Let's use the WordPress "Create New Post" test case.

  1. Go to http://demo.opensourcecms.com/wordpress/wp-login.php
  2. Enter "admin" in the "Username" field
  3. Enter "demo123" in the "Password" field
  4. Click on the "Log In" button
  5. Verify that the text "Howdy, admin" is present
  6. Click on the "Posts" link
  7. Click on the "Add New" button
  8. Type "Selenium Demo Post" in the title field
  9. Click on the "Publish" button
  10. Verify that the text "Post published" is present

Considering this scenario, the first thing that comes to mind is creating a long test case that goes through all the steps. This might be a good approach if you are writing a manual test case. However, since we are writing an automated test, we want to write our script as modular as possible to be able to reuse parts of it in future scenarios.

This is how I would break down the test:

  1. Launch the WordPress site
  2. Open the Admin Login page
  3. Enter valid login data
  4. Navigate to the Write Post page
  5. Write the post
  6. Publish the post
  7. Verify that it was actually post

Keep in mind that this is just an example. You are free to model your tests in any way you want, as long as they have business value and will validate your business logic.

Let's see how to do that with actual Java code:

@Test(description="Launches the WordPress site")
public void launchSite(){
  selenium.open("");
  selenium.waitForPageToLoad("30000");
  assertEquals(selenium.getTitle(), "Demo | Just another WordPress site");
}

@Test(description="Navigates to the admin page")
  public void openAdminPage() {
  selenium.open("wp-admin");
  selenium.waitForPageToLoad("30000");
  assertEquals(selenium.getTitle(), "Demo › Log In");
}

@Test(description="Enters valid login data")
  public void loginAsAdmin() {
  selenium.type("user_login", "admin");
  selenium.type("user_pass", "demo123");
  selenium.click("wp-submit");
  selenium.waitForPageToLoad("30000");
  assertTrue(selenium.isTextPresent("Howdy, admin"));
}

@Test(description="Navigates to the New Post screen")
public void navigateNewPost() {
  selenium.click("//a[contains(text(),'Posts')]/following::a[contains(text(),'Add New')][1]");
  selenium.waitForPageToLoad("30000");
  assertTrue(selenium.isTextPresent("Add New Post"));
}

@Test(description="Writes the new post")
public void writeBlogPost() {
  selenium.type("title", "New Blog Post");
  selenium.click("edButtonHTML");
  selenium.type("content", "This is a new post");
  //TODO:Assert
}

@Test(description="Publishes the post")
public void publishBlogPost() {
  selenium.click("submitdiv");
  selenium.click("publish");
  selenium.waitForPageToLoad("30000");
  assertTrue(selenium.isTextPresent("Post published."));
}

@Test(description="Verifies the post")
public void verifyBlogPost() {
  selenium.click("//a[contains(text(),'Posts') and contains(@class,'wp-first-item')]");
  selenium.waitForPageToLoad("30000");
  assertTrue(selenium.isElementPresent("//a[text()='New Blog Post']"));
}

@Test(description="Logs out")
public void logout() {
  selenium.click("//a[text()='Log Out']");
  //TODO:Assert
}

These are the test methods (or steps) we are going to use.

Configuration methods

If you are familiar with unit testing frameworks, you probably know about the setup and teardown methods. TestNG goes beyond that idea and allows you to define methods that will be run after or before your test suites, test groups or test methods.

This is very useful for our Selenium tests because you can create a Selenium server and browser instance before you start running your test suite.)

To achieve this, we will use two TestNG annotations: @BeforeSuite and @AfterSuite:

@BeforeSuite(alwaysRun = true)
public void setupBeforeSuite(ITestContext context) {
  String seleniumHost = context.getCurrentXmlTest().getParameter("selenium.host");
  String seleniumPort = context.getCurrentXmlTest().getParameter("selenium.port");
  String seleniumBrowser = context.getCurrentXmlTest().getParameter("selenium.browser");
  String seleniumUrl = context.getCurrentXmlTest().getParameter("selenium.url");

  RemoteControlConfiguration rcc = new RemoteControlConfiguration();
  rcc.setSingleWindow(true);
  rcc.setPort(Integer.parseInt(seleniumPort));

  try {
    server = new SeleniumServer(false, rcc);
    server.boot();
  } catch (Exception e) {
    throw new IllegalStateException("Can't start selenium server", e);
  }

  proc = new HttpCommandProcessor(seleniumHost, Integer.parseInt(seleniumPort),
      seleniumBrowser, seleniumUrl);
  selenium = new DefaultSelenium(proc);
  selenium.start();
}

@AfterSuite(alwaysRun = true)
public void setupAfterSuite() {
  selenium.stop();
  server.stop();
}

PS: Did you notice those weird parameters? They are stored in the XML file (we are going to see in the next section) and accessed by a ITestContext object, which was injected.

By adding these annotations, the TestNG engine will invoke the configuration methods automatically before/after your test suite (make sure the test methods are annotated with @Test), launching the Selenium server and instantiating the Selenium client object only once, reusing the same browser session across the tests.

Creating the XML file

To define the order of the tests, we will have to create an XML file listing the test methods we would like to run. Make sure that the test methods are annotated with @Test, or else the TestNG engine will not invoke them.

Before TestNG 5.13.1, you had to use Method Interceptors if you wanted to run the tests in the order defined in the XML file. I have posted my implementation of a Method Interceptor on my Github account. From TestNG 5.13.1+, you can just add the preserve-order parameter to your test tag and include the methods you would like to run, reducing unnecessary code in your test suite.

Here is the XML file:

<suite name="Knorrium.info - Wordpress Demo" verbose="10">
  <parameter name="selenium.host" value="localhost" />
  <parameter name="selenium.port" value="3737" />
  <parameter name="selenium.browser" value="*firefox" />
  <parameter name="selenium.url" value="http://demo.opensourcecms.com/wordpress/" />

  <test name="Write new post" preserve-order="true">
    <classes>
      <class name="test.Wordpress">
        <methods>
          <include name="launchSite" />
          <include name="openAdminPage" />
          <include name="loginAsAdmin" />
          <include name="navigateNewPost" />
          <include name="writeBlogPost" />
          <include name="publishBlogPost" />
          <include name="verifyBlogPost" />
        </methods>
      </class>
    </classes>
  </test>
</suite>

Launching your tests in Eclipse

We finished writing our tests, now how can we run them?

You can launch TestNG from the command line, using a Eclipse plugin or even programatically. We are going to use the Eclipse plugin. Follow the steps described on the official TestNG documentation over here

If you installed TestNG correctly, you will see this menu when you right click on the XML file:

screenshots

Click on “Run as TestNG Suite” and your test will start running. You will then see this nice results tree:

screenshots

Thinking about the future

If you really want to think about the future of your test suite, I would recommend you to read Adam Goucher’s article published on PragPub. He talks about Selenium 2 and the Page Objects Model (a very nice way to model your tests, especially if you use Selenium 2).

Since there are lots of people still using Selenium 1, I'll stick to that for a while, but Selenium 2 will eventually be covered here.

As the number of tests in your test suite grows, you will find that grouping them in different test classes is a good idea. If you do that, you can take advantage of object oriented programming and create a new class named BaseTest (for example), and leave your configuration logic there. That way, every test class must extend the BaseTest class and use static attributes.

public class WordPressAdmin extends BaseTest {
@Test
public void test1(){
  selenium.open("");
  //...
}

@Test
public void test2(){
  selenium.open("");
  //...
}
}

This is better than leaving your configuration methods in the test class.

TestNG Eclipse plug-in

The TestNG Eclipse plug-in allows you to run your TestNG tests from Eclipse and easily monitor their execution and their output. It has its own project repository called testng-eclipse.

Installation

Follow the instructions to install the plug-in.

Info

Since TestNG Eclipse Plugin 6.9.10, there is a new optional plug-in for M2E (Maven Eclipse Plugin) integration. It's recommended to install it if your Java project(s) are managed by Maven.

Once done, restart Eclipse and select the menu Window / Show View / Other... and you should see the TestNG view listed in the Java category.

screenshot

Info

since TestNG Eclipse Plugin 6.9.8, the minimum required TestNG version is 6.5.1

Creating a TestNG class

To create a new TestNG class, select the menu File / New / TestNG:

screenshot

If you currently have a Java file open in the editor or if you have a Java file selected in the Navigator, the first page of the wizard will show you a list of all the public methods of that class and it will give you the option to select the ones you want to test.

screenshot

Each method you select on this page will be included in the new TestNG class with a default implementation that throws an exception, so you remember to implement it.

The next page lets you specify where that file will be created, whether it should contain default implementation for some configuration methods, if you'd like a data provider and finally, if a testng.xml file should be generated.

The plug-in will make a guess about the best location where this file should be created (for example, if you are using Maven, the default location will be under src/test/java).

Launch configuration

Once you have created classes that contain TestNG annotations and/or one or more testng.xml files, you can create a TestNG Launch Configuration. Select the Run / Run... (or Run / Debug...) menu and create a new TestNG configuration:

screenshot

You should change the name of this configuration and pick a project, which can be selected by clicking on the Browse... button at the top of the window.

Runtime options:

  • Log Level: specify the value (0-10) for different verbose log levels
  • Verbose: enable the runtime TestNG verbose log
  • Debug: enable more runtime TestNG debug info
  • Serialization Protocol: the serialization protocol used for communicating between TestNG Eclipse Plugin and TestNG runtime.
  • Json Serialization: This protocol was introduced in the TestNG Eclipse plug-in 6.9.11 to better communicate with a different JRE running TestNG.
  • Object Serialization: This protocol packs the message data with Java serialization.
  • String Serialization: Deprecated.

Then you choose to launch your TestNG tests in the following ways:

From a class file

Make sure the box near Class is checked and then pick a class from your project. You can click on the Browse... button and pick it directly from a list. This list only contains classes that contain TestNG annotations:

screenshot

From groups

If you only want to launch one or several groups, you can type them in the text field or pick them from a list by clicking on the Browse... button

screenshot

From a definition file

Finally, you can select a suite definition from your project. It doesn't have to be named testng.xml, the plug-in will automatically identify all the applicable TestNG XML files in your project:

screenshot

You can type the regex on the filter text field to narrow down to suite definition files matching your search from a long list of files.

From a method

This launch isn't accomplished from the Launch dialog but directly from your Outline view:

screenshot

You can right-click on any test methods and select Run as... / TestNG test and only the selected method will be run (not shown on the above screenshot because I couldn't find a way to capture a contextual menu). Method launching is also available from the Package Explorer view and from the Java Browser perspective.

Once you have selected one of these launches, you can also choose the logging of level. Then you can launch the tests by pressing the Debug (or Run) button, which will switch you to the Debug perspective and will open the main TestNG view.

Specifying listeners and other settings

As you saw above, the plug-in will let you start tests in many different ways: from an XML file, from a method, a class, etc... When you are running an XML file, you can specify all the settings you want for this run in the XML file, but what

if you want to run a package in parallel mode with specific listeners? How can you configure the settings for all the launches that are not done from an XML file?

In order to give you access to the most flexibility, TestNG lets you specify an XML suite file for all these launches, which you can find in the Preferences menu:

screenshot

If you specify a valid suite file as "XML template file", TestNG will reuse all the settings found in this XML file, such as parallel, name, listeners, thread pool size, etc... Only the <test> tags in this file will be ignored since the plug-in will replace these by a generated <test> tag that represents the launch you chose.

Viewing the test results

screenshot

The above view shows a successful run of the tests: the bar is green and no failed tests are reported. The All tests tab shows you a list of all the classes and methods that were run.

If your test run contains failures, the view will look like this:

screenshot

You can use the Failed tests tab to display only these tests that failed, and when you select such a test, the stack trace will be shown on the right-hand pane. You can double click on the offending line to be taken directly to the failure in your code.

screenshot

When you have hundreds of tests running, finding a specific one is not always easy, so you can type a few letters of the test method or its parameters in the Search box and the content of the tree will automatically narrow down to methods matching your search. Note in the screen shot above that the search also works on parameters provided by @DataProvider.

Summary

screenshot

The Summary tab gives you statistics on your test run, such as the timings, the test names, the number of methods and classes, etc… Since the results are shown in a table, you can also sort on any criterion you like for easier parsing. This is especially handy when you are trying to determine what tests take the longest time.

The search box works in this view as well, and note that in the screen shot below, the Time column is sorted in decreasing order:

screenshot

Converting JUnit tests

You can easily convert JUnit 3 and JUnit 4 tests to TestNG. Your first option is to use the Quick Fix function:

screenshot

Convert from JUnit 3

screenshot

Convert from JUnit 4

You can also convert packages or entire source folders with the conversion refactoring:

screenshot

The refactoring wizard contains several pages:

screenshot

This page lets you generate a testng.xml automatically. You can configure whether to include your test classes individually or by package, the suite and test name and also whether these tests should run in parallel.

screenshot

This page gives you an overview of the changes that are about to be performed. You can also decide to exclude certain files from the refactoring.

When you are done, press the "Finish" button. Like all Eclipse refactorings, you can undo all these changes in one click:

screenshot

Quick fixes

The TestNG Eclipse plug-in offers several quick fixes while you are editing a TestNG class (accessible with Ctrl-1 on Windows/Linux and ⌘-1 on Mac OS):

Convert to JUnit

This was covered in the previous section.

Pushing and pulling @Test annotations

If you have several test methods annotated with @Test and you'd like to replace them all with a single @Test annotation at the class level, choose the "Pull annotation" quick fix. Reciprocally, you can move a class level @Test annotation onto all your public methods or apply a quick fix on an assert method to automatically import it.

Preferences and Properties

Workbench Preferences

TestNG workbench preferences:

screenshot

The preferences here are shared among projects and launch configurations.

  • Output directory: the path where to store the output including temp files, report files, etc... By default, the path is relative to each project except if you check the option Absolute output path below.
  • Absolute output path: whether the path above is absolute or relative to the current project.
  • Disable default listeners: disable the default listeners when launching TestNG.
  • Show view when test complete: activate the TestNG result view when the test completes.
  • Template XML file: the absolute path of the template XML file used to genernate the custom test suite XML file before launching.
  • Excluded stack traces:
  • Predefined Listeners:

Project Properties

Project level properties:

screenshot

Here are properties on each project level, it will override the same properties if defined in TestNG workbench preferences

  • Output directory: for example, in the figure above, I prefer to put the output to maven 'target' directory rather than the default one under project root
  • Watch testng-result.xml:
  • Template XML file: see in TestNG workbench preferences
  • Predefined Listeners: see in TestNG workbench preferences

M2E Integration

The (optional) TestNG M2E Integration plug-in was introduced in 6.9.10. It allows you to run your tests with System Properties or JVM settings, which are defined by maven-surefire-plugin or maven-failsafe-plugin of pom.xml, to be appended to the runtime TestNG process. Once this plugin installed, you can see a dedicated preference page (workspace level settings):

screenshot

or on the project properties page. You can override workspace settings with project specific ones:

screenshot

Let's say there is maven-surefire-plugin confguration in your pom.xml:

<artifactId>maven-surefire-plugin</artifactId>
<configuration>
    <suiteXmlFiles>
    <suiteXmlFile>test-suite/testng.xml</suiteXmlFile>
    </suiteXmlFiles>
    <argLine>-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
                -Xmx1024m -XX:MaxPermSize=512m -Xms256m -Xmx1024m -XX:PermSize=128m
                -Dversion=${project.version}
    </argLine>
    <systemPropertyVariables>
    <foo>${foo.value}</foo>
    </systemPropertyVariables>
    <environmentVariables>
    <bar>${bar.value}</bar>
    </environmentVariables>
</configuration>

You can pass the following Maven configuration parameters to the TestNG process:

  • argLine: the JVM arguments
  • systemPropertyVariables: the system properties
  • environmentVariables: the environment variables

Note

As the snippet of the configuration above shows, properties placeholders (e.g. ${aspectj.version}) in argLine or systemPropertyVariables will be substituted and correctly passed to the TestNG process as long as the properties are visible on the Maven project (e.g. defined in the current pom.xml, or inherited from the parent pom.xml, etc.).

Info

If your maven-surefire-plugin is defined in a Maven profile, you will need to select the Maven profile which contains the maven-surefire-plugin configuration: "Right click on the project -> Maven -> Select Maven Profiles...", then check the profile you need.

TestNG IDEA Plug-in

TestNG is bundled in IDEA

Creating a TestNG Run/Debug configuration

Screenshot

Once you have installed the plug-in and restarted IDEA, and have some TestNG classes you would like to run, simply open the Run/Debug window. You will see a TestNG tab, where you can add a configuration.

There are a number of methods for determining the set of tests that will be run. These are:

  • Package: Specify a package to run. All tests in this package and below will be included.
  • Group: Specify a TestNG group to run.
  • Suite: Specify an external testng.xml file to run.
  • Class: Run all tests in a single class.
  • Method: Run a single test method.

Once you create the run configuration, you can run it. Upon running, the plug-in will launch an external process to run your tests. The test results will be display in a tree view, with passed and failed tests highlighted. You can narrow down on the console output for a specify test by clicking on it, while double clicking a test will navigate to its source code.

Screenshot

Book

screenshot

Our book is now available from Amazon .

Book Description

Enterprise Java developers must achieve broader, deeper test coverage, going beyond unit testing to implement functional and integration testing with systematic acceptance. Next Generation Java Testing introduces breakthrough Java testing techniques and TestNG, a powerful open source Java testing platform.

Cédric Beust, TestNG's creator, and leading Java developer Hani Suleiman, present powerful, flexible testing patterns that will work with virtually any testing tool, framework, or language. They show how to leverage key Java platform improvements designed to facilitate effective testing, such as dependency injection and mock objects. They also thoroughly introduce TestNG, demonstrating how it overcomes the limitations of older frameworks and enables new techniques, making it far easier to test today's complex software systems.

Pragmatic and results-focused, Next Generation Java Testing will help Java developers build more robust! code for today's mission-critical environments.

This book

  • Illuminates the tradeoffs associated with testing, so you can make better decisions about what and how to test
  • Introduces TestNG, explains its goals and features, and shows how to apply them in real-world environments
  • Shows how to integrate TestNG with your existing code, development frameworks, and software libraries
  • Demonstrates how to test crucial code features, such as encapsulation, state sharing, scopes, and thread safety
  • Shows how to test application elements, including JavaEE APIs, databases, Web pages, and XML files
  • Presents advanced techniques: testing partial failures, factories, dependent testing, remote invocation, cluster-based test farms, and more
  • Walks through installing and using TestNG plug-ins for Eclipse, and IDEA
  • Contains extensive code examples

Whether you use TestNG, JUnit, or another testing framework, the ! testing design patterns presented in this book will show you how to im prove your tests by giving you concrete advice on how to make your code and your design more testable.

About the Author

Cédric Beust, a senior software engineer at Google, is an active member of the Java Community Process who has been extensively involved in the development of the latest Java release. He is the creator and main contributor to the TestNG project.

Hani Suleiman is CTO of Formicary, a consulting and portal company specializing in financial applications. He is one of two individual members who has been elected to the Executive Committee of the Java Community Process.