Top Python Testing Frameworks In 2020 For Selenium Automation

Python is the fastest-growing programming language in 2019 as per the Developer Survey by StackoverFlow. It even edged past Java in the overall ranking and came out on 2nd as the most loved language only after Rust. One of the several reasons for the popularity of Python is the extensive support of test automation frameworks. Most of the popular Python frameworks are compatible with the Selenium test automation framework and used for automation browser testing & cross browser testing.

With the increasing number of Python testing frameworks in 2020, it becomes necessary to choose the ideal Selenium test automation framework that suits your requirements. Apart from unittest (or PyUnit) which is the default Python testing framework in 2020, there are many more python frameworks to choose from.

Here are the top Python testing frameworks in 2020 to realize your Selenium test automation needs.

1. PyTest

PyTest is an open-source Python testing framework that is primarily used for unit testing. This particular Python testing framework in 2020 is scalable as it is useful for writing simple automation tests as well as complex functional tests for applications & libraries. It is easy to get started with PyTest. It is compatible with Python 3.5+ and PyPy 3.

Test suites written using PyTest are more compact as a lot of boiler-plate code is not required and there is no requirement to include tests into large test classes. Unlike the default Python testing framework 2020 which mandates tests to start with test_ (or end with _test), there is no fixed test case nomenclature in PyTest. This is because PyTest has built-in features that supports auto-discovery of test modules and functions.

There is no need to remember self.assert* names due to the introduction of the useful feature of assert rewriting that helps in providing detailed information on failing assert statements.

PyTest Official Website – https://www.pytest.org/

PyTest on GitHub – https://github.com/pytest-dev/pytest/

Here are the details about the PyTest project:

What Are The Prerequisites?

The best part about the PyTest, Python testing framework, is that it involves a gentle learning curve. It is easy to get started with PyTest, especially if you have prior working knowledge about any existing Python testing frameworks.

You will just need a Python IDE for development and Python Package Manager for installing PyTest.

Pros of PyTest

There are a number of advantages of the PyTest framework, major ones are below:

• It is compatible with other Python testing frameworks such as unittest (or PyUnit).
• Along with simple tests, PyTest can also be used for creating complex functional test cases.
• It requires less effort to port existing implementations using other Python testing frameworks to PyTest.
• PyTest can be used for projects that practice TDD (Test Driven Development) as well as open-source projects.
• PyTest is instrumental in creating effective test cases (and test suites) as it supports parameterization. Using parameterization, test cases can be executed with different input configurations thereby resulting in minimal code repetition.
• Tests using PyTest are easy to understand as there is very little boiler-plate code.
• It supports fixtures and classes due to which common test objects are available through the lifecycle of a module/class/function/session.
• PyTest is extensible and the Python testing framework already has rich plugin architecture. There are currently 315+ external plugins with PyTest.
• It supports Parallel test execution through the pytest-xdist plugin.
• Asserts in PyTest provide detailed information about the failure scenarios.

Cons of PyTest as a Python testing framework

PyTest is not compatible with other Selenium Python testing frameworks as it uses its own special routines for development. Rewriting the complete code is the only way to port an existing implementation using PyTest to some other Python testing framework.

Is PyTest The best Python Testing Framework For You?

You can choose PyTest over the default Selenium Python framework i.e. unittest in case you are developing complex functional tests. Fixtures, Classes, as well as Parameterization can be extremely handy for automation test development.

2. Behave

Behave is one of the widely used Selenium Python testing frameworks for BDD (Business Driven Development) testing. Behave uses the Gherkin language for development of scenarios and feature files. As Gherkin makes use of simple readable language for test case development, tests can also be created for non-technical personnel in the team thus enabling a Business Driven Development.

Usage of a BDD framework enables better communication between the team members as there is increased collaboration between engineers, managers, quality analysts, business development managers, and more. It is relatively easy to get started with a behavior-driven Python testing framework like Behave if you have working knowledge about other BDD frameworks such as SpecFlow, Cucumber, Cucumber-JVM , etc.

As Behave is a BDD framework, it is fundamentally different from other Selenium Python testing frameworks such as unittest (or PyUnit), PyTest, etc. The latest version of Behave is 1.2.7.dev1

Behave Official Website – https://behave.readthedocs.io/en/latest/

Behave on GitHub – https://github.com/behave/behave

Here are the details about the Behave framework:

What Are The Prerequisites?

Installation of Behave requires Python above 2.7.14. Prior experience with any of the BDD tools can be handy to get started with the Behave framework.

Pros Of Behave

Here are the advantages of using the Behave framework for Selenium test automation:

• It is easy to ramp-up if the team has prior experience with any existing BDD framework such as SpecFlow, Cucumber, etc.
• Availability of environmental functions, configuration settings, fixtures, etc. enables easy setup & cleanup.
• Behave is supported with the PyCharm (Professional Edition).
• It supports integration with other web frameworks such as Django and Flask.
• It has excellent support documentation and tutorials that can be helpful in getting started with behave framework.
• It is simpler to add newer test cases as tests consist of different scenario steps, scenario outcomes, feature files, and more.
• Apart from technical people, non-technical members can also play a vital role in test case (or test scenario) development.

Cons Of Behave

Below are some of the major cons of using the behave framework in Selenium test automation:

• It is not supported in the PyCharm (Community Edition).
• It is very difficult to share scenario steps (*_steps.py) between different feature files (*.feature) in Behave.
• There is no built-in support for parallel test execution which is one of the primary requirements for automated browser testing. Workarounds for parallel test execution require installation of additional software as mentioned in popular discussion threads here and here.
• Behave can only be used for black-box testing.
• The plugin behave-parallel that facilitated parallel test execution with behave is now deprecated.

Is Behave The Best Python Testing Framework For You?

Behave framework is suited for web testing as test scenarios can be developed using simple English language. On the other hand, behave is not ideal for integration testing as the verbosity can lead to unnecessary complications of the test scenarios.

Many developers prefer pytest-bdd, the BDD library for pytest.runner as it uses the good features available in pytest hence suited for BDD testing.

3. Robot

Robot is another widely-used Python testing framework that is used for Selenium test automation, RPA (Robotic Process Automation), and ATDD (Acceptance Test Driven Development). It is an open-source and extensible Python testing framework. It can be easily integrated with any other tool due to which it is ideal for creating flexible automation testing solutions.

The Python testing framework is operating system and application independent i.e. the same implementation is compatible across different platforms namely – Windows, Linux, and MacOS. The syntax of Robot framework is easy to understand as it uses human-readable keywords. The capabilities of Robot framework can be extended by libraries that are implemented in Selenium with Python . It has a rich ecosystem consisting of libraries and tools developed as separate projects.

Robot Official Website – https://robotframework.org/

Robot on GitHub – https://github.com/robotframework/robotframework

Here are the details about the Robot framework:

What Are The Prerequisites?

Robot Framework can be used if you have Python version 2.7.14 or higher installed. Python Package Manager (pip) command is used to install the Python testing framework.

Pros of Robot framework

Here are the advantages of using the Robot framework for Selenium test automation:

• You do not require programming knowledge to write test cases using the Robot framework.
• As the Python testing framework is platform and application independent, there is no porting effort required when executing code from one platform to another.
• It can be used for BDD, ATDD, and key-word driven Selenium test automation.
• It is easy to get started with the Robot framework as the test data syntax can be used with ease.
• Robot framework has a rich library ecosystem e.g. Appium for mobile automation, database library for DB testing, etc. for testing different applications.
• It has a good built-in reporting mechanism as a HTML log is generated after every build.

Cons of Robot

Below are some of the major cons of using the Robot framework:

• There is no in-built support for parallel test execution.
• Creating customized HTML reports is not an easy task.

Is Robot The Best Python Testing Framework For You?

Due to its rich set of libraries and easy syntax, Robot framework is easier to use when compared to PyUnit (or unittest) and PyTest. You should choose Robot framework if we are looking for an ideal Python automation framework for ATDD.

4. PyUnit (or unittest)

PyUnit (also called unittest) is the default Python testing framework that is a part of the Python standard library. The Python Testing Framework, is inspired by the JUnit framework. It is a part of the unittest module that has shipped since the Python version 2.1 was released.

As the unittest Python testing framework is available out of the box, many developers who are getting started with Selenium test automation prefer the unittest framework. Test cases follow a particular nomenclature where tests either start with test_ or end with _test.

TestCase is the base class that contains the assertion methods, SetUp, and TearDown routines. Number of testcases can be grouped together to form a TestSuite class and build customized test runners.

What Are The Prerequisites?

As PyUnit framework is available out of the box with the Python testing framework, it does not require any additional package installation.

Pros of PyUnit (or unittest)

Here are some of the advantages of the PyUnit framework:

• As PyUnit is a derivative of the xUnit framework and has distinct similarities with the xUnit framework, newbie developers in Python also find it easy to get started with unittest framework.
• No additional package installation is required.
• Test scenarios can be executed independently or they can be grouped together into a TestSuite. The execution process is simple as tests can be executed only by mentioning their names on the terminal.
• The default execution output is concise and easy to understand. This Python testing framework also has a number of command-line options that help in providing more detailed output.
• Test report generation is very fast and takes only a few milliseconds.

Cons of PyUnit (or unittest)

Here are some of the major shortcomings of the PyUnit framework:

• Naming of testcases still based on the camelCase naming convention using in Java.
• A large amount of boilerplate code is used for test implementation.
• There is no provision for color output.

Is PyUnit The Best Python Automation Framework For You?

As PyUnit (or unittest) is the default Selenium Python testing framework that is shipped with Python, most developers start their automating testing with Python journey with PyUnit. Too much boilerplate code and support for too much abstraction acts as a major hindrance in using the Python Testing Framework for large projects.

5. Lettuce

Lettuce is a Python testing framework that facilitates BDD testing. It draws its inspiration from Cucumber (that was designed for Ruby). The release names of Lettuce are inspired by green stuff. The latest version of Lettuce is 0.2.23 and is termed Kryptonite.

This particular Python testing framework is open-source and hosted on GitHub. As mentioned on their official website, Lettuce was developed to make testing easy and funny.

Lettuce Official Website – http://lettuce.it/

Lettuce on GitHub – https://github.com/gabrielfalcao/lettuce

Here are the details about the Lettuce framework:

What Are The Prerequisites?

Lettuce can be installed using the Python Package Manager (pip). The prerequisite is Python 2.7.14 (or higher).

Pros of Lettuce

Here are some of the advantages of the Lettuce framework:

• Like other Python testing frameworks such as Behave that facilitate BDD, Lettuce also uses the Gherkin language for creation of scenarios and feature files. Getting started with Lettuce is easy if you have prior experience with any BDD testing tool.
• Apart from developers and other technical personnel, other project stakeholders namely marketing folks, project managers, etc. can also play a major role in the test process.
• It is easy to add new test cases as primary updates are required in feature files and scenario files.

Cons of Lettuce

Here are some of the major shortcomings of the Lettuce framework:

• It is primarily used for black-box testing.
• Perfect execution using a BDD testing tool like Lettuce requires timely communication between all the respective stakeholders of the project.

Is Lettuce The Best Python Automation Framework For You?

Lettuce makes the entire process of Selenium test automation scalable, readable, and easy. The best part of using a BDD tool like Lettuce is that non-technical people can also participate in the test creation process as tests are derived from business requirements. Python developers and testers have greater inclination towards pytest-bdd as it has majority of the features supported by other BDD tools.

6. Nose2

Nose2 is a Python unit testing framework and a successor of the Nose framework. It is based on the unittest hence; it is also referred to as an extension to unittest. Nose2 adds support to the PyUnit framework by providing plugins for test execution, enabling automatic discovery of test cases, and collecting documentation.

All these added functionalities make testing with PyUnit simple and easy. The plugin ecosystem of Nose2 adds support for decorators, fixtures, parameterized testing, modules, classes, and test discovery. Such in Nose2 is a DSL (Domain-Specific Language) for writing tests with expensive, nested features, also called as functional tests. More details of Such in Nose2 is available on the official website of Such.

Nose2 Official Website – https://docs.nose2.io/en/latest/

Nose2 on GitHub – https://github.com/nose-devs/nose2

Here are the details about the Nose2, python unit testing framework:

What Are The Prerequisites?

Nose2 only has a dependency on the unittest module which is shipped along with the Python library. Nose2 can be installed using Python Package Manager (pip).

Pros of Nose2

Here are some of the advantages of the Nose2 python testing framework:

• It is easy to install and get started with Nose2 as it extends the existing unittest framework.
• With features like test discovery, fixtures, parameterized testing, etc., nose2 makes unittest equally useful as PyTest.
• Scripts written in Nose2 support a number of command-line options and extensive configuration via config files.
• Using the mp plugin in Nose2, tests can be executed in parallel with multiple processes. This speeds up the test execution in case your tests are heavily IO or CPU Bound.
• Nose2 has a large number of plugins that can aid in accelerating the Selenium test automation process.

Cons of Nose2

Here are some of the minor shortcomings of the Nose2 python testing framework:

• The documentation of Nose2 is not as extensive as the PyTest framework.
• Deprecation of nose framework that was supported on Python 2.4 (and above) could be a potential dampener for developers who want to use Nose2 in their projects.

Is Nose2 The Best Python Testing Framework For You?

If your team members are using the unittest framework and do not intend to switch to any other framework (e.g. Lettuce, Behave, PyTest, etc.), it is worth giving Nose2 a try as it is an extension of unittest.

As Nose2 also has support for parallelized testing, it can be used for automated browser testing kind of scenarios where test scenarios are executed on different browser & platform combinations.

7. Testify

Testify is another popular Python testing framework in 2020 that is considered as a replacement of unittest and nose framework. As the framework is modeled after unittest, the tests written for unittest will require minimal adjustments to work with Testify.

Testify can be used for performing automated unit, integration, and system testing. The latest version of Testify is 0.11.0. It has an extensible plugin system that provides useful functionality around reporting. Like the Nose2 framework, Testify also provides enhanced test discovery and class-level SetUp & TearDown fixture methods that are executed once for the entire set of test methods.

Testify Official Website – https://pypi.org/project/testify/

Testify on GitHub – https://github.com/Yelp/Testify

Here are the details about the Testify framework:

What Are The Prerequisites?

The major advantage of the Testify framework is that it involves a gentle learning curve as it is based on the existing unittest framework. Installation is fairly simple as it only requires the Python Package Manager.

Pros of Testify

Here are some of the advantages of the Testify framework:

• It is easy to get started with Testify as it is based on the unittest framework.
• The framework has an extensible plugin ecosystem.
• It has additional testing utilities that aid in integration, profiling, and code coverage.
• Like the Nose2 framework, Testify also comes with enhanced test discovery which makes finding tests very easy.
• It has a decorator-based approach for fixture methods.

Cons of Testify

Here are some of the minor shortcomings of Testify framework:

• Testify also has the same issue as Nose2, the documentation is not so extensive which could be a roadblock if you are just getting started with the Testify framework.
• Parallel testing with Testify could be a hurdle as pointed in a couple of online discussion forums.

Is Testify The Best Python Testing Framework For You?

Like Nose2, if your team is looking for a Python testing framework built on top of unittest, then it is worth trying out Testify. As achieving parallelized testing with Testify is a bit difficult, it might not be suited for scenarios such as automated browser testing.

Wrapping it Up

In this article, we had a look at some of the popular Python testing frameworks that can be used with Selenium. Each of these frameworks have their own share of advantages and disadvantages. If you intend to include the non-technical members in the test creation process, you should try BDD tools such as Behave or Lettuce. To summarize, choosing the right Python testing framework purely depends on the project requirements and the competency available within the team.

Which particular Python testing framework do you use? Do share your experience in the comments section. Also, please feel free to share this article with your friends by retweeting us on twitter. Happy Testing ☺