Testing Guide

Overview

The current ParaQ testing framework makes it possible to "record" and "play back" user interaction with the ParaQ UI, and record render-window images for later comparison against reference images. By combining these two capabilities it is possible to create regression tests.

Recording Tests

• Start the ParaQ client.
• From the menu, choose Tools > Record Test.
• In the file dialog that opens, choose a path where the test output will be written.
• The "Recording User Input" dialog will appear. Move it somewhere out-of-the-way.
• Use the ParaQ client UI normally, exercising the functionality you wish to test.
• Once the client is in the desired end-state, go to the "Recording User Input" dialog and hit the "Stop Recording" button.

Running Tests

• Start the ParaQ client.
• From the menu, choose Tools > Play Test.
• In the file dialog that appears, choose a previously-recorded test.
• The test will play-back.
• Once the test completes, the client should be in the same end-state as when the test was recorded.

Regression Test Setup

• Start the ParaQ client and record a test.
• Once test recording is complete, choose File > Save Screenshot.
• Use the file dialog to save the contents of the render window as a reference image.
• Exit the ParaQ client.
• To run the test, use the appropriate command-line options to play the test, and compare its output with a reference image:

$ pqClient --run-test /path/to/test.xml --compare-view /path/to/reference/image.png

• The --run-test option plays-back a test scenario as if you had used the Tools > Play Test menu item.
• The --compare-view option compares the contents of the render window with a reference image, exiting the client with a non-zero exit code if the images don't match.
• To make the test a permanent part of the ParaQ regression test suite, add it to the Qt/Client/CMakeLists.txt file (there are existing examples there).

Running Regression Tests

• To run the complete set of regression tests, use the ctest utility included with CMake:

$ cd /path/to/ParaQBuild
$ ctest

Python based testing

If building the QtTesting code with python, python scripts can be used to do testing. All features of XML based recording and playback are the same. With python, there are extra features available.

• The normal python language features.
• Query a property of any QObject.

import QtTesting
...
value1 = '1'
if QtTesting.getProperty(object5, 'text') != value1:
  raise ValueError('object5\'s value is not equal to ' + value1)

• Image capture and comparison that works on /any/ QWidget in the GUI.
• Multiple image capture and comparisons are possible in one test.

To use the python based image comparison, one imports the QtTestingImage module, and uses the compareImage function. The following example connects to the builtin server, and loads a pvsm state file. Then it does an image comparison:

import QtTesting
import QtTestingImage
object1 = 'MainWindow/menubar/menuFile'
QtTesting.playCommand(object1, 'activate', 'actionFileLoadServerState')
object2 = 'MainWindow/ServerStartupBrowser/connect'
QtTesting.playCommand(object2, 'activate', )
object3 = 'MainWindow/FileLoadServerStateDialog'
QtTesting.playCommand(object3, 'filesSelected', '$PARAVIEW_DATA_ROOT/Data/LoadStateMultiView.pvsm')
snapshotWidget = 'MainWindow/1pqRenderWindowManager0/SplitterFrame/MultiViewSplitter/0/Viewport'
QtTestingImage.compareImage(snapshotWidget, 'LoadStateMultiView.png', 200, 200);

The width & height of the image are specified. The implementation renders the widget to an offscreen pixmap of that size. Then it uses vtkTesting to compare the image to one with the specified name in the PARAVIEW_DATA_ROOT/Baselines directory. These extra features are used by manually editing the python scripts.

Future Work

The current test framework has a number of limitations that will likely be addressed as development progresses:

• The image-capture and comparison functionality should be expanded to include chart windows in-addition-to normal render windows. -- DONE
• Image-capture and comparison must be expanded to handle multi-view setups involving more-than-one render window. -- DONE
• Image comparison must be robust in the face of small differences in image size - on different platforms (and using different styles on a single platform), the sizes of windows will vary even with a fixed main window size, due to differences in e.g: the width of a splitter bar by a few pixels.