Build status Maven Central Javadoc FOSSA Status

🇸🇬 lah (Singlish) - Placed at the end of a phrase or sentence either for emphasis or reassurance.

JustTestLah! is a JAVA test framework. It follows a BDD approach and allows testing on different platforms (Android, iOS and Web) using the same test scenarios. JustTestLah’s main aim is to make the configuration as easy and the test code as simple and readable as possible.

Getting started

Pull the repo and run the example.

git clone https://github.com/martinschneider/justtestlah.git
cd justtestlah-demos
mvn test -Dtest=TestRunner

The default platform is web. To test one of the mobile apps you need to setup Appium and start an Appium server. You also need at least one physical or emulated device connected. Then simply execute the tests by setting platform=android or platform=ios in justtestlah.properties. Please note that the Stackoverflow demo is only available for web and android (upvote this question to help change this). For the Carousell demo, you need to have a Carousell account (it’s free). Configure username and password in justtestlah-demos/src/test/resources/qa/justtestlah/examples/carousell/testdata/user/valid.yml.

mvn test -Dtest=TestRunner -Djusttestlah.properties=/absolute/path/to/your/justtestlah.properties

The second parameter (justtestlah.properties) is optional; the default configuration can be found under justtestlah-demos/src/test/resources.

Use in your own projects

Add the following Maven dependency to your pom.xml.

<properties>
  <justtestlah.version>1.7-RC5</justtestlah.version>
</properties>

<dependency>
  <groupId>qa.justtestlah</groupId>
  <artifactId>justtestlah-core</artifactId>
  <version>${justtestlah.version}</version>
</dependency>

Page objects, steps and feature files

There are three main ingredients for tests in JustTestLah!:

  • Page objects are a representation of a UI element (a page, a dialog, a screen etc.).
  • Step definitions use page objects to define the actions of a test. They form the building blocks to write
  • feature files which represent the test cases.

Steps and page objects are designed to be highly re-usable.

Demo of a feature file:

Feature: Search and tags

@web
Scenario: Filter by tags
  Given I am on the homepage
  When I go to the tags page
  And I filter for "selenium"
  And I select the tag "selenium"
  And I select the first question
  Then the question is tagged with "selenium"

@web @android
Scenario: Use the search function
  Given I am on the homepage
  When I search for "selenium"
  And I select the first question
  Then the question is tagged with "selenium"

Demo of a step definition class:

public class HomeSteps extends BaseSteps {
  private HomePage home;

  @Given("I am on the homepage")
  public void homepage() {
    home.load();
  }

  @When("I go to the tags page")
  public void goToTags() {
    home.navigateToTagsPage();
  }

  @When("I search for {string}")
  public void search(String query) {
    home.search(query);
  }
}

Demo of a page object:

@Component
@Profile(Platform.WEB)
public class HomePage extends BasePage<HomePage> {

  private QuestionsPage questions;

  private TagsPage tags;

  public HomePage load() {
    open(configuration.getBaseUrl());
    return this;
  }

  public TagsPage navigateToTagsPage() {
    $("MENU_TAGS").click();
    return tags;
  }

  public QuestionsPage search(String query) {
    $("SEARCH_FIELD").sendKeys(query);
    $("SEARCH_BUTTON").should(appear).click();
    return questions;
  }
}

You can inject page objects in steps by declaring a private field:

private HomePage home;

As long as the page object class extends qa.justtestlah.base.BasePage JustTestLah! (and Spring) will take care of the rest. In the same way you can also use page objects inside other page objects.

Configuration

All configuration goes in a file called justtestlah.properties.

# GENERAL settings
platform=web
pages.package=qa.justtestlah.examples.stackoverflow.pages
steps.package=qa.justtestlah.examples.stackoverflow.steps
features.directory=src/test/resources/features/stackoverflow
cucumber.report.directory=target/report/cucumber

# optional
galen.report.directory=target/report/galen
galen.enabled=false
eyes.enabled=false
opencv.enabled=false
eyes.apiKey=
cloudprovider=local


# WEB settings
web.baseUrl=https://www.stackoverflow.com
web.browser=chrome
web.headless=true


# MOBILE settings
mobile.appiumUrl=http://127.0.0.1:4723/wd/hub


# ANDROID settings
android.appPackage=com.stackexchange.stackoverflow
android.appActivity=com.stackexchange.stackoverflow.MainActivity
android.appPath=/Users/martinschneider/stackoverflow.apk
android.deviceName=Google Nexus 6


# IOS settings
ios.appPath=
ios.deviceName=iPhone 6


# BROWSERSTACK settings (requires cloudprovider=browserstack)
browserstack.debug=true
browserstack.accessKey=
browserstack.username=

You can specify the location of justtestlah.properties on start-up by providing it as a system property: -DjusttestlahProperties=/path/to/justtestlah.properties. If no path is specified it will be loaded from the classpath.

Test runner

JustTestLah! uses JUnit to run the tests. All you need to do is add an empty class which extends qa.justtestlah.JustTestLahTest:

public class TestRunner extends JustTestLahTest {}

Alternatively, you can also use the JUnit test runner directly:

@RunWith(JustTestLahRunner.class)
public class SomeTestClass {}

The feature files and steps are automatically picked up from the locations provided in justtestlah.properties.

Locators

Elements can be identified by a unique id, a css or an xpath expression. AccesibilityId (for iOS) and UIAutomator (for Android) are supported as well. Each element has a unique key (e.g. SEARCH_FIELD) which is mapped to its corresponding locator expression in a .yaml file. For example, let’s say the page object for the home page is demoproject.pages.HomePage (under /src/main/java). Then the corresponding locators are expected in /demoproject/pages/HomePage.yaml (under /src/main/resources).

Example of a locator YAML file:

LOGIN_BUTTON:
  web:
    type: xpath
    value: "//BUTTON//SPAN[text()='Log in']"
  ios:
    type: accesibilityId
    value: login_page_login_button
  android:
    type: id
    value: com.thecarousell.Carousell:id/login_page_login_button

The correct locator will be automatically resolved for the current platform. Taking the above example, the search field can be accessed in the HomePage page object by calling $("SEARCH_BUTTON"). This will return an instance of com.codeborne.selenide.SelenideElement. See the Selenide quick start to learn about all the cool ways you can interact with it. Two caveats to take note of:

  1. It is not possible to directly use elements in step definitions (only in page objects). This is by design as UI elements are meant to be encapsulated in the page objects.
  2. While we wrap Selenide’s $ method for the locator handling the methods you can call on the returned SelenideElement instances remains the same.

If omitted the default type of locators is css.

Placeholders

Locators can include both dynamic and static placeholders which will be replaced by variables passed to the $ method.

Static placeholders

You can think of static placeholders as variables. They can be defined in a file called placeholder.properties in the root of the pages package (specified as pages.package in justtestlah.properties). This is the same folder the locator YAML files are placed in.

This file continues key/value pairs in the following format:

SOME_KEY=someValue
PACKAGE_NAME=com.stackexchange.stackoverflow

One use-case, as shown above, can be to define the Android package name as a variable and use it in all id-based locators like this:

POST_TAG:
  android:
    type: id
    value: ${PACKAGE_NAME}:id/question_view_item_tags  

If you want to override static placeholders during runtime, you can pass an extra placeholder file by setting locator.placeholders.file to its absolute path in justtestlah.properties. For any placeholders which exist in both (the one under pages.package and locator.placeholders.file), the latter one will override the former.

Dynamic placeholders

Sometimes, you might require a locator which depends on some dynamic values defined only at runtime. You can achieve this by putting %s as a placeholder in the locator and use the $(String locatorKey, Object... params) and $$(String locatorKey, Object... params) methods in BasePage to pass the String which should be inserted at its place.

Let’s see an example:

POST_TAG:
  web:
    type: xpath
    value: "//A[contains(@class,'post-tag') and contains(text(),'%s')]"

Calling $("POST_TAG", "selenium") will return an element matching the following Xpath expression: //A[contains(@class,'post-tag') and contains(text(),'selenium').

Test data handling

JustTestLah! supports loading test data from YAML files. Each test data entity is represented by a Java class (the model) and one or many YAML files which contain the actual test data. For example:

@TestData("user")
public class User {
  private String username;
  private String password;

  public User() {}

  public User(String username, String password) {
    this.username = username;
    this.password = password;
  }

  public String getUsername() {
    return username;
  }

  public void setUsername(String username) {
    this.username = username;
  }

  public String getPassword() {
    return password;
  }

  public void setPassword(String password) {
    this.password = password;
  }
}
user:
  username: myUsername
  password: myPassword

Note, that the top level key in the YAML file must match the value of the @TestData annotation.

You can then load test data in your tests as easy as this:

User user = testdata(User.class);
User user = testdata(User.class, "validUser");
User user = testdata(User.class, "userWithInvalidPassword");

The second parameter points to the name of the test entity which is the filename of the YAML file. If ommited it defaults to default. In the above example, you would have three YAML files: default.yaml, validUser.yaml and userWithInvalidPassword.

There are three configuration values for this feature:

model.package=
testdata.filter=

Setting testdata.enabled=true enables the YAML test data resolution. The default is false!

model.package is mandatory and specifies the root package to scan for Java objects representing test entities (those need to be marked with @TestData).

testdata.filter allows restricting the path to scan for test data YAML files. If left empty everything matching **/testdata/**/*.y*ml (under src/test/resources) will be considered.

Cloud service integrations

JustTestLah! supports integration with various cloud service provides. Some of them are in PoC state. Please feel free to contribute.

Browserstack

You can run tests against BrowserStack by adding the following configuration in justtestlah.properties:

cloudprovider=browserstack

# Browserstack username
browserstack.username=
# Browserstack access key
browserstack.accessKey=

# Optional settings, see https://www.browserstack.com/automate/capabilities
browserstack.debug=true
browserstack.appiumLogs=true
browserstack.video=true
browserstack.geoLocation=SG
browserstack.networkProfile=
browserstack.customNetwork=
browserstack.timezone=SG
browserstack.appium_version=1.8.0

Make sure justtestlah-browserstack is on your classpath:

<dependency>
  <groupId>qa.justtestlah</groupId>
  <artifactId>justtestlah-browserstack</artifactId>
  <version>${project.version}</version>
</dependency>

Please note that BrowserStack is a paid service.

AWS Devicefarm

You can run tests against AWS Devicefarm by adding the following configuration in justtestlah.properties:

cloudprovider=aws

# The arn of your AWS Devicefarm project (mandatory)
aws.projectArn=

# App package to use. If this value is empty it will be created and uploaded to AWS Devicefarm before the test execution
aws.appPackageArn=

# Test package to use. If this value is empty it will be created and uploaded to AWS Devicefarm before the test execution
#aws.testPackageArn=

# Optional extra data
aws.extraDataArn=

# Fully-qualified path to the justtestlah-demos project (required to build the test package)
aws.demo.path=

# Name for the test package (must match <finalName> in the justtestlah-demos pom.xml)
aws.testpackage.name=justtestlah-awsdevicefarm

# Device filters (optional)
aws.minOsVersion=9.0
aws.maxOsVersion=
aws.osVersion=
aws.model=
aws.manufacturer=
aws.formFactor=PHONE
aws.waitForDevice=true

# Optional device configuration
aws.deviceLatitude=
aws.deviceLongitude
aws.bluetooth=
aws.gps=
aws.nfc=
aws.wifi=
# set this to true if you use device slots
aws.runUnmetered=false

# Additional AWS Devicefarm configuration
aws.accountsCleanup=
aws.appPackagesCleanup=
aws.jobTimeOut=
aws.skipAppResign=

Make sure justtestlah-awsdevicefarm is on your classpath:

<dependency>
  <groupId>qa.justtestlah</groupId>
  <artifactId>justtestlah-awsdevicefarm</artifactId>
  <version>${project.version}</version>
</dependency>

You can refer to this article for a more detailled description of how to tweak AWS Devicefarm.

Please note that AWS Devicefarm is a paid service.

Template matching

JustTestLah! allows locating elements using a template image:

boolean isImagePresent = homePage.hasImage("questionIcon.png");

Match image = homePage.findImage("questionIcon.png");

The images are expected under /src/test/resources/images.

The Match object contains the x and y coordinate of the matched image (more precisely, the center of the rectangle representing the match). These can be used to interact with an element located this way. For example, we can tap on an element like this:

new TouchAction((PerformsTouchActions) WebDriverRunner.getWebDriver())
.tap(PointOption.point(questionIcon.getX(), questionIcon.getY()))
.perform();

Future versions of JustTestLah! will include wrappers to perform these actions more conveniently.

The TemplateMatcher is scale-invariant (to some extent). The algorithm used to achieve this scales the target image (a screenshot of the device) up and down until either a match is found or a minimum (320) or maximum (3200) image width is reached.

Note, that the closer the size of the template matches the size of the image on the screen the faster and more accurate the matching will be.

Matching threshold

Both the hasImage and findImage method take an optional threshold parameter which can be used to define the accuracy of a match. The possible values range from 0 (no match) to 1 (pixel-perfect match). The default is 0.9.

Client and server-mode matching

There are two modes to use template matching which can be configured in justtestlah.properties:

opencv.mode=client performs the image matching on the client (i.e. the machine running the test code). It requires OpenCV which is imported as a Maven dependency (https://github.com/openpnp/opencv).

opencv.mode=server utilises the image matching feature of Appium. This requires OpenCV to be installed on the machine which runs the Appium server.

Note, that not all cloud providers support this.

Applitools

There is a proof-of-concept integration of Applitools. It can be enabled by setting eyes.enabled=true in justtestlah.properties. In addition a valid API key must be specified: eyes.apiKey=....

Checks can then be triggered by calling checkWindow() on any page object class (the initial run will create baseline images).

Please note that Applitools is a paid service.

Galen

JustTestLah! includes a proof-of-concept integration of the Galen framework. It can be enabled by setting galen.enabled=true in justtestlah.properties.

Similar to properties-file holding the locator information, there is an (optional) spec file for each page object (in the same package as the Java class under src/main/resources).

Checks can be triggered by calling checkLayout() on any page object class. An HTML report is generated in the directory defined in galen.report.directory in justtestlah.properties (the default is target/galen-reports/).

@objects
  username_field  id  com.thecarousell.Carousell:id/login_page_username_text_field
  password_field  id  com.thecarousell.Carousell:id/login_page_password_text_field
  login_button    id  com.thecarousell.Carousell:id/login_page_login_button

= Login =

  username_field:
      above password_field
      aligned vertically all password_field
      width 100 % of password_field/width
      width 100 % of login_button/width

  password_field:
      below username_field
      aligned vertically all username_field
      width 100 % of username_field/width
      width 100 % of login_button/width

  login_button:
      below password_field
      below username_field
      width 100 % of username_field/width
      width 100 % of password_field/width
      text is "Log In"

See the Galen documentation for more examples.

Used libraries

JustTestLah! makes use of a variety of frameworks to make writing and executing tests as transparent and simple as possible.

  • Selenium, the main test framework used by JustTestLah!
  • Appium, an extension of Selenium for native mobile app testing
  • Cucumber, the BDD framework
  • JUnit, the unit testing framework (mostly used as the runner for the tests)
  • Selenide, a convenience wrapper around Selenium
  • AssertJ, fluent assertions for unit tests
  • OpenCV, used for image comparison
  • Galen, used for layout based testing
  • Applitools, used for visual regression testing
  • BrowserStack, cloud provider for automated tests
  • Spring, IoC container for some added “magic” behind the scenes

Known issues & limitations

  • JustTestLah! requires Java 10 or higher (and has been tested on Java 10, 11, 12 and 13). Java 9 support has been dropped because of JDK-8193802 which isn’t fixed on Java below 10.

  • The OpenCV integration (used for client-side template matching) doesn’t work with Java 12 yet.

  • The Galen PoC has only been tested against Appium 1.7. Please feel free to contribute an update for this feature.

Contact and support

Please let me know about any feedback, questions or ideas for improvement.

Martin Schneider - mart.schneider@gmail.com

Buy me a coffee