====== Binary download & install ====== ==== Debian / Ubuntu ==== > The TestFarm packages can be installed from our APT repository. > All packages are built and tested under Ubuntu 14.04 LTS. === Import APT repository settings === <code bash> $ wget -q -O- https://testfarm.org/dnld/testfarm.asc | sudo apt-key add - $ sudo wget -N https://testfarm.org/dnld/testfarm.list -O /etc/apt/sources.list.d/testfarm.list $ sudo apt-get update </code> === Install TestFarm system === This will also install the TestFarm tools and applications and their dependencies. <code bash> $ sudo apt-get install testfarm-core testfarm-vu </code> Logout and relogin, in order for the TestFarm environment settings to be taken into account by the shell. === Install TestFarm demo === As a good start point for deploying your own testing system, you can install the EWD demo. This demo is described in [[#testfarm_ewd_demo|this section]]. <code bash> $ sudo apt-get install testfarm-demo-ewd </code> Then you can deploy the demo code to your home directory: <code bash> $ /opt/testfarm/demo/install.sh </code> ====== Sources ====== TestFarm source files are available on GitHub: https://github.com/testfarm ====== TestFarm EWD Demo ====== This is a reference demo for **TestFarm**, a test automation tool targeted at embedded software. Please visit the main [[http://www.testfarm.org|TestFarm.org web site]] for product description and philosophy. ===== Overview ===== EWD stands for "Engine/Warning Display", an aircraft cockpit visualisation application. {{demo-EWD:screenshot0001.png?500}} This demo application works with a unique interface : a Xvnc virtual display. It does not show all the capabilities of TestFarm for using several i/o interfaces simultaneously, but it does not require special hardware equipments, and can easily be run on a Linux computer. The following diagram shows the architecture of the demo-EWD testing system: {{demo-EWD:archi.png?500}} ===== Download & Install ===== ==== Ready-To-Use Virtual Machine ==== For the impatient, a Ready-To-Use [[https://www.virtualbox.org|VirtualBox]] appliance can be downloaded [[http://www.testfarm.org/documents/demo-EWD/TestFarm-demo-EWD_150903.ova|here]]. This is an Ubuntu 14.04 LTS virtual machine, with the TestFarm software and demo pre-installed. ==== Distribution packages ==== If your prefer installing the demo natively on your Linux computer, follow the instructions in the [[#binary_download_install|Binary download & install]] section. ===== Run the demo ===== ==== Start the Launch Pad ==== Start the **TestFarm Launch Pad** from the Desktop Main Menu : {{testfarm-launch-start.png?400}} Alternatively, you can start the Launch Pad from a shell: <code bash> $ testfarm-launch & </code> The Launch Pad appears and shows the EWD demo in the list of available test suites : {{testfarm-launch.png?500}} ==== Build the Test Suite ==== Select the demo-EWD Test Suite and start the **TestFarm Test Suite Builder** by clicking the ''Edit'' button: {{testfarm-launch-edit.png?300}} In the Test Suite Builder, click on the ''Make'' button : {{testfarm-build.png?500}} Wait for the build to complete. The ''Make'' button keeps unsensitive while the build is in progress. You can also scroll the output message window to see the build operations. Once completed, you can quit the Test Suite Builder. ==== Execute the Test Suite ==== In the Launch Pad, select the demo-EWD Test Suite and start the **TestFarm Runner** by clicking the ''Execute'' button: {{testfarm-launch-execute.png?300}} -> {{testfarm-run.png?400}} In the Launch Pad, start the **TestFarm Virtual User Display** Tool. {{testfarm-launch-display.png?300}} -> {{testfarm-vu-display.png?400}} The Display Tool displays what the testing system "sees" from the application under test. The test suite can run without it : it is an inspection tool for the test operator. When you start the Display Tool at this step, it shows a blank screen. This is a normal situation, because no test suite is being executed for the moment. Now it is time to start the Test Suite : in the Runner, click the ''Go'' button: {{testfarm-run-go.png?200}} {{testfarm-run-running.png?800}} ==== Browse the Test Report ==== Click the ''Report'' button: {{testfarm-run-report.png?200}} This will automatically generate the Test Report in HTML format, and show it in a web browser. You can [[http://www.testfarm.org/documents/demo-EWD/demo-EWD-report.tgz|download]] or [[http://www.testfarm.org/documents/demo-EWD/report/EWD|browse]] a sample Test Report. You can also view the [[http://www.testfarm.org/documents/demo-EWD/report/EWD/output.xml|Test Output file]] in raw XML format. //TestFarm Core// uses XSLT transformation scripts to generate the HTML report from this file . ===== Guided Tour ===== This section will give you a short description of the demo-EWD test suite, along with the various features it uses. ==== System Configuration ==== The file [[http://www.testfarm.org/documents/demo-EWD/EWDsystem.xml|EWDsystem.xml]] contains the test system configuration. === Info fields === The ''INFO'' elements provide some values that are used as config items by the test scripts : ^ Field(nbsp)Id ^ Default(nbsp)Value ^ Description ^ | ''Host(nbsp)Addr'' | 127.0.0.1 | The IP adresse of //System Under Test// (embedding a VNC server). \\ On the demo, the SUT runs on the same machine as the test system, so we use the //localhost// IP address | | ''Color(nbsp)Fuzz'' | 8 | Color Fuzz value. This specifies the level of precision required for Color Matching. \\ 8 is equivalent to #080808 : it means we tolerate +/- 8 levels of color on each RGB component. | | ''Pixel(nbsp)Loss'' | 10 | Specify the percentage of bad pixels we tolerate when matching a bitmap image. \\ This could be adjusted, depending on the SUT hardware. On modern graphical user interfaces, a non-nul value is required to avoid false rejects due to anti-aliasing, drop shadows, ... | === Interfaces === The EWD demo contains one interface named //VISU//, defined in a ''INTERFACE'' element : the VNC display capture interface, implemented by the //TestFarm Virtual User// software. This interface provides the feature //Display//. ==== Test Suite ==== The EWD Test Suite contains 7 scenarios. A scenario is a set of Test Cases, executed sequentially to test a product feature. Scenarios can be executed in any order. Within a scenario, Test Cases may not function if they are not executed in order. ^ Scenario ^ Description ^ Basic TestFarm features demonstrated ^ | F1::S1 | Check labels | Color matching to check that an area is blank. \\ Bitmap matching from PNG reference images. | | F1::S2 | Check outlines of thrust indicators | Bitmap matching from PNG reference images, exploiting PNG transparency to ignore some areas. | | F2::S1 | Check labels at various thrust limits | Bitmap matching from PNG reference images. \\ Screenshot in the test report | | F3::S1 | Check left needle position against requested values | Needle position tracking using Color Matching \\ Animated GIF screenshot in the test report | | F3::S2 | Check right needle position against requested values | ::: | | F4::S1 | Check left digits against requested values | Characater recognition using Tesseract OCR agent \\ Blinking period check by post-processing the test log file | | F4::S2 | Check right digits against requested values | ::: |