Binary download & install

Debian / Ubuntu

The TestFarm packages can be installed from our APT repository.
All packages are built and tested under Ubuntu 14.04 LTS.

Import APT repository settings

$ wget -q -O- https://testfarm.org/dnld/testfarm.asc | sudo apt-key add -
$ sudo wget -N https://testfarm.org/dnld/testfarm.list -O /etc/apt/sources.list.d/testfarm.list
$ sudo apt-get update

Install TestFarm system

This will also install the TestFarm tools and applications and their dependencies.

$ sudo apt-get install testfarm-core testfarm-vu

Logout and relogin, in order for the TestFarm environment settings to be taken into account by the shell.

Install TestFarm demo

As a good start point for deploying your own testing system, you can install the EWD demo. This demo is described in this section.

$ sudo apt-get install testfarm-demo-ewd

Then you can deploy the demo code to your home directory:

$ /opt/testfarm/demo/install.sh

Sources

TestFarm source files are available on GitHub: https://github.com/testfarm

TestFarm EWD Demo

This is a reference demo for TestFarm, a test automation tool targeted at embedded software. Please visit the main TestFarm.org web site for product description and philosophy.

Overview

EWD stands for “Engine/Warning Display”, an aircraft cockpit visualisation application.

This demo application works with a unique interface : a Xvnc virtual display. It does not show all the capabilities of TestFarm for using several i/o interfaces simultaneously, but it does not require special hardware equipments, and can easily be run on a Linux computer.

The following diagram shows the architecture of the demo-EWD testing system:

Download & Install

Ready-To-Use Virtual Machine

For the impatient, a Ready-To-Use VirtualBox appliance can be downloaded here.

This is an Ubuntu 14.04 LTS virtual machine, with the TestFarm software and demo pre-installed.

Distribution packages

If your prefer installing the demo natively on your Linux computer, follow the instructions in the Binary download & install section.

Run the demo

Start the Launch Pad

Start the TestFarm Launch Pad from the Desktop Main Menu :

Alternatively, you can start the Launch Pad from a shell:

$ testfarm-launch &

The Launch Pad appears and shows the EWD demo in the list of available test suites :

Build the Test Suite

Select the demo-EWD Test Suite and start the TestFarm Test Suite Builder by clicking the Edit button:

In the Test Suite Builder, click on the Make button :

Wait for the build to complete. The Make button keeps unsensitive while the build is in progress. You can also scroll the output message window to see the build operations. Once completed, you can quit the Test Suite Builder.

Execute the Test Suite

In the Launch Pad, select the demo-EWD Test Suite and start the TestFarm Runner by clicking the Execute button:

In the Launch Pad, start the TestFarm Virtual User Display Tool.

The Display Tool displays what the testing system “sees” from the application under test. The test suite can run without it : it is an inspection tool for the test operator. When you start the Display Tool at this step, it shows a blank screen. This is a normal situation, because no test suite is being executed for the moment.

Now it is time to start the Test Suite : in the Runner, click the Go button:

Browse the Test Report

Click the Report button:

This will automatically generate the Test Report in HTML format, and show it in a web browser.

You can download or browse a sample Test Report.

You can also view the Test Output file in raw XML format. TestFarm Core uses XSLT transformation scripts to generate the HTML report from this file .

Guided Tour

This section will give you a short description of the demo-EWD test suite, along with the various features it uses.

System Configuration

The file EWDsystem.xml contains the test system configuration.

Info fields

The INFO elements provide some values that are used as config items by the test scripts :

Field Id Default Value Description
Host Addr 127.0.0.1 The IP adresse of System Under Test (embedding a VNC server).
On the demo, the SUT runs on the same machine as the test system, so we use the localhost IP address
Color Fuzz 8 Color Fuzz value. This specifies the level of precision required for Color Matching.
8 is equivalent to #080808 : it means we tolerate +/- 8 levels of color on each RGB component.
Pixel Loss 10 Specify the percentage of bad pixels we tolerate when matching a bitmap image.
This could be adjusted, depending on the SUT hardware. On modern graphical user interfaces, a non-nul value is required to avoid false rejects due to anti-aliasing, drop shadows, …

Interfaces

The EWD demo contains one interface named VISU, defined in a INTERFACE element : the VNC display capture interface, implemented by the TestFarm Virtual User software. This interface provides the feature Display.

Test Suite

The EWD Test Suite contains 7 scenarios. A scenario is a set of Test Cases, executed sequentially to test a product feature. Scenarios can be executed in any order. Within a scenario, Test Cases may not function if they are not executed in order.

Scenario Description Basic TestFarm features demonstrated
F1::S1 Check labels Color matching to check that an area is blank.
Bitmap matching from PNG reference images.
F1::S2 Check outlines of thrust indicators Bitmap matching from PNG reference images, exploiting PNG transparency to ignore some areas.
F2::S1 Check labels at various thrust limits Bitmap matching from PNG reference images.
Screenshot in the test report
F3::S1 Check left needle position against requested values Needle position tracking using Color Matching
Animated GIF screenshot in the test report
F3::S2 Check right needle position against requested values
F4::S1 Check left digits against requested values Characater recognition using Tesseract OCR agent
Blinking period check by post-processing the test log file
F4::S2 Check right digits against requested values