Binary download & install
Debian / Ubuntu
The TestFarm packages can be installed from our APT repository.
All packages are built and tested under Ubuntu 14.04 LTS.
Import APT repository settings
$ wget -q -O- https://testfarm.org/dnld/testfarm.asc | sudo apt-key add - $ sudo wget -N https://testfarm.org/dnld/testfarm.list -O /etc/apt/sources.list.d/testfarm.list $ sudo apt-get update
Install TestFarm system
This will also install the TestFarm tools and applications and their dependencies.
$ sudo apt-get install testfarm-core testfarm-vu
Logout and relogin, in order for the TestFarm environment settings to be taken into account by the shell.
Install TestFarm demo
As a good start point for deploying your own testing system, you can install the EWD demo. This demo is described in this section.
$ sudo apt-get install testfarm-demo-ewd
Then you can deploy the demo code to your home directory:
$ /opt/testfarm/demo/install.sh
Sources
TestFarm source files are available on GitHub: https://github.com/testfarm
TestFarm EWD Demo
This is a reference demo for TestFarm, a test automation tool targeted at embedded software. Please visit the main TestFarm.org web site for product description and philosophy.
Overview
EWD stands for “Engine/Warning Display”, an aircraft cockpit visualisation application.
This demo application works with a unique interface : a Xvnc virtual display. It does not show all the capabilities of TestFarm for using several i/o interfaces simultaneously, but it does not require special hardware equipments, and can easily be run on a Linux computer.
The following diagram shows the architecture of the demo-EWD testing system:
Download & Install
Ready-To-Use Virtual Machine
For the impatient, a Ready-To-Use VirtualBox appliance can be downloaded here.
This is an Ubuntu 14.04 LTS virtual machine, with the TestFarm software and demo pre-installed.
Distribution packages
If your prefer installing the demo natively on your Linux computer, follow the instructions in the Binary download & install section.
Run the demo
Start the Launch Pad
Start the TestFarm Launch Pad from the Desktop Main Menu :
Alternatively, you can start the Launch Pad from a shell:
$ testfarm-launch &
The Launch Pad appears and shows the EWD demo in the list of available test suites :
Build the Test Suite
Select the demo-EWD Test Suite and start the TestFarm Test Suite Builder by clicking the Edit
button:
In the Test Suite Builder, click on the Make
button :
Wait for the build to complete. The Make
button keeps unsensitive while the build is in progress. You can also scroll the output message window to see the build operations. Once completed, you can quit the Test Suite Builder.
Execute the Test Suite
In the Launch Pad, select the demo-EWD Test Suite and start the TestFarm Runner by clicking the Execute
button:
In the Launch Pad, start the TestFarm Virtual User Display Tool.
The Display Tool displays what the testing system “sees” from the application under test. The test suite can run without it : it is an inspection tool for the test operator. When you start the Display Tool at this step, it shows a blank screen. This is a normal situation, because no test suite is being executed for the moment.
Now it is time to start the Test Suite : in the Runner, click the Go
button:
Browse the Test Report
This will automatically generate the Test Report in HTML format, and show it in a web browser.
You can download or browse a sample Test Report.
You can also view the Test Output file in raw XML format. TestFarm Core uses XSLT transformation scripts to generate the HTML report from this file .
Guided Tour
This section will give you a short description of the demo-EWD test suite, along with the various features it uses.
System Configuration
The file EWDsystem.xml contains the test system configuration.
Info fields
The INFO
elements provide some values that are used as config items by the test scripts :
Field Id | Default Value | Description |
---|---|---|
Host Addr | 127.0.0.1 | The IP adresse of System Under Test (embedding a VNC server). On the demo, the SUT runs on the same machine as the test system, so we use the localhost IP address |
Color Fuzz | 8 | Color Fuzz value. This specifies the level of precision required for Color Matching. 8 is equivalent to #080808 : it means we tolerate +/- 8 levels of color on each RGB component. |
Pixel Loss | 10 | Specify the percentage of bad pixels we tolerate when matching a bitmap image. This could be adjusted, depending on the SUT hardware. On modern graphical user interfaces, a non-nul value is required to avoid false rejects due to anti-aliasing, drop shadows, … |
Interfaces
The EWD demo contains one interface named VISU, defined in a INTERFACE
element : the VNC display capture interface, implemented by the TestFarm Virtual User software. This interface provides the feature Display.
Test Suite
The EWD Test Suite contains 7 scenarios. A scenario is a set of Test Cases, executed sequentially to test a product feature. Scenarios can be executed in any order. Within a scenario, Test Cases may not function if they are not executed in order.
Scenario | Description | Basic TestFarm features demonstrated |
---|---|---|
F1::S1 | Check labels | Color matching to check that an area is blank. Bitmap matching from PNG reference images. |
F1::S2 | Check outlines of thrust indicators | Bitmap matching from PNG reference images, exploiting PNG transparency to ignore some areas. |
F2::S1 | Check labels at various thrust limits | Bitmap matching from PNG reference images. Screenshot in the test report |
F3::S1 | Check left needle position against requested values | Needle position tracking using Color Matching Animated GIF screenshot in the test report |
F3::S2 | Check right needle position against requested values | |
F4::S1 | Check left digits against requested values | Characater recognition using Tesseract OCR agent Blinking period check by post-processing the test log file |
F4::S2 | Check right digits against requested values |