Testing Guide

Overview

The current ParaQ testing framework makes it possible to "record" and "play back" user interaction with the ParaQ UI, and record render-window images for later comparison against reference images. By combining these two capabilities it is possible to create regression tests.

Recording Tests

• Start the ParaQ client.
• From the menu, choose Tools > Record Test.
• In the file dialog that opens, choose a path where the test output will be written.
• The "Recording User Input" dialog will appear. Move it somewhere out-of-the-way.
• Use the ParaQ client UI normally, exercising the functionality you wish to test.
• Once the client is in the desired end-state, go to the "Recording User Input" dialog and hit the "Stop Recording" button.

Running Tests

• Start the ParaQ client.
• From the menu, choose Tools > Play Test.
• In the file dialog that appears, choose a previously-recorded test.
• The test will play-back.
• Once the test completes, the client should be in the same end-state as when the test was recorded.

Regression Test Setup

• Start the ParaQ client and record a test.
• Once test recording is complete, choose File > Save Screenshot.
• Use the file dialog to save the contents of the render window as a reference image.
• Exit the ParaQ client.
• To run the test, use the appropriate command-line options to play the test, and compare its output with a reference image:

$ pqClient --run-test /path/to/test.xml --compare-view /path/to/reference/image.png

• The --run-test option plays-back a test scenario as if you had used the Tools > Play Test menu item.
• The --compare-view option compares the contents of the render window with a reference image, exiting the client with a non-zero exit code if the images don't match.
• To make the test a permanent part of the ParaQ regression test suite, add it to the Qt/Client/CMakeLists.txt file (there are existing examples there).

Running Regression Tests

• To run the complete set of regression tests, use the ctest utility included with CMake:

$ cd /path/to/ParaQBuild
$ ctest

Future Work

The current test framework has a number of limitations that will likely be addressed as development progresses:

• The image-capture and comparison functionality should be expanded to include chart windows in-addition-to normal render windows.
• Image-capture and comparison must be expanded to handle multi-view setups involving more-than-one render window.
• Image comparison must be robust in the face of small differences in image size - on different platforms (and using different styles on a single platform), the sizes of windows will vary even with a fixed main window size, due to differences in e.g: the width of a splitter bar by a few pixels.