KWWidgets/GUI Testing/Squish/SquishForKWWidgets
Goals
- Create a CMake module to find Squish on the system (paths to Squish server and Squish runner).
- Create CMake macros that will run a Squish test by invoking the squish server and client runner, and parsing its output,
- Invoke the Squish module when configuring/building KWWidgets,
Files
The following files were added to the KWWidgets/CMake directory. Most of them were adapted from Brad Davis' scripts.
- FindSquish.cmake: locates Squish and sets SQUISH_FOUND.
- KWWidgetsSquishTestScript.cmake: implements a Squish testing sequence (start Squish server, run test, stop Squish server, etc.).
- SquishRunTestCase.sh: The shell script launched by KWWidgetsSquishTestScript.cmake to run the testing sequence.
The KWWidgets_ADD_Squish_TEST macro is defined in KWWidgetsTestingMacro.cmake:
# KWWidgets_ADD_Squish_TEST macro(KWWidgets_ADD_Squish_TEST test_name squish_AUT_full_path squish_test_case_path aut_env aut_path_script aut_args ) set(kwwSquishShellScript "${KWWidgets_CMAKE_DIR}/SquishRunTestCase.sh") # Can only handle "Tk" set(squish_script_wrapper "Tk") ADD_TEST(${test_name} ${CMAKE_COMMAND} -V -VV --test "-Dsquish_aut:STRING=${squish_AUT_full_path}" "-Dsquish_server_executable:STRING=${SQUISH_SERVER_EXECUTABLE}" "-Dsquish_client_executable:STRING=${SQUISH_CLIENT_EXECUTABLE}" "-Dsquish_libqtdir:STRING=${QT_LIBRARY_DIR}" "-Dsquish_test_case:STRING=${squish_test_case_path}" "-Dsquish_env_vars:STRING=${aut_env}" "-Dsquish_wrapper:STRING=${squish_script_wrapper}" "-Dsquish_aut_script:STRING=${aut_path_script}" "-Dsquish_shell_script:STRING=${kwwSquishShellScript}" "-Dsquish_aut_args:STRING=${aut_args}" -P "${KWWidgets_CMAKE_DIR}/KWWidgetsSquishTestScript.cmake" ${ARGN}) endmacro(KWWidgets_ADD_Squish_TEST)
Usage
The following CMakeLists.txt snippet demonstrates how to add a Squish test to one of the KWWidgets C++ example: Examples/Cxx/Callbacks. The KWWidgets_USE_Squish option should be turned to ON for KWWidgets to add the corresponding tests; make sure Squish is found by pointing KWWidgets to the appropriate SQUISH_INSTALL_DIR directory. The test itself was created from Squish, and saved to Examples/Cxx/Callbacks/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale.
find_package(Squish) [...] if(BUILD_TESTING AND KWWidgets_BUILD_TESTING) include("${KWWidgets_CMAKE_DIR}/KWWidgetsTestingMacros.cmake") [...] if(KWWidgets_USE_Squish AND SQUISH_FOUND) set(Squish_TST_NAME "${KWWidgets_SOURCE_DIR}/Examples/Cxx/${PROJECT_NAME}/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale") kwwidgets_add_squish_test_from_c_example( KWWidgets-${PROJECT_NAME}Squish ${EXE_NAME} "${Squish_TST_NAME}") endif(KWWidgets_USE_Squish AND SQUISH_FOUND) endif(BUILD_TESTING AND KWWidgets_BUILD_TESTING)
The following command will list all the Squish-enabled tests available in KWWidgets:
barre@ubuntu804desktop:~/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug$ ctest -N -R Squish Start processing tests Test project /home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug 7/ 24 Testing KWWidgets-CallbacksSquish 9/ 24 Testing KWWidgets-HelloWorldSquish 11/ 24 Testing KWWidgets-WidgetsTourSquish 13/ 24 Testing KWWidgets-WindowWithPanelsSqui 15/ 24 Testing KWWidgets-WizardDialogSquish 17/ 24 Testing KWWidgets-MedicalImageViewerSq 19/ 24 Testing KWWidgets-PolygonalObjectViewe
And here is how one would launch the Squish test for the Callbacks example, in verbose (-V) mode:
barre@ubuntu804desktop:~/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug$ ctest -R CallbacksSquish -V [...] /home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/Examples/Cxx/Callbacks 7/ 24 Testing KWWidgets-CallbacksSquish Test command: /usr/local/bin/cmake -V -VV --test ... /home/barre/src/KWWidgets-HEAD/CMake/KWWidgetsSquishTestScript.cmake Test timeout computed to be: 1500 -- squish_aut='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWCallbacksExample' -- squish_aut_args='' -- squish_server_executable='/home/barre/build/squish-20081218-tk-src/bin/squishserver' -- squish_client_executable='/home/barre/build/squish-20081218-tk-src/bin/squishrunner' -- squish_libqtdir ='' -- squish_test_case='/home/barre/src/KWWidgets-...Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale' -- squish_wrapper='Tk' -- squish_env_vars='' -- squish_aut_script='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWWidgetsSetupPaths.sh' -- The_sh_torun='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/CMake/SquishRunTestCase.sh' Starting the squish server.../home/barre/build/squish-20081218-tk-src/bin/squishserver --daemon Running the test case.../home/barre/build/squish-20081218-tk-src/bin/squishrunner --testcase /home/barre/src/KWWidgets-HEAD/Examples/Cxx/Callbacks/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale --wrapper Tk --aut /home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWCallbacksExample * This is an EVALUATION version of Squish. * RUNNERID: 151590003 AUTID: 151590004 AUTHOST: 127.0.0.1 AUTPORT: 4322 RUNNERID: 151590003 application registered 151590004 Initializing AUT context ...2009-01-02T16:15:37 START Start 'tst_ScrollScale' Test 'tst_ScrollScale' started ... done. 2009-01-02T16:15:39 PASS /mnt/hgfs/src/kitware/KWWidgets/KWWidgets-HEAD/... VP1: Object property comparison of ':KWCallbacksExample.vtkKWWindow0.vtkKWFrame3...vtkKWMyWidget0.vtkKWScale0.getvalue' passed'32.0' and '32.0' are equal (VP1) 2009-01-02T16:15:41 END End 'tst_ScrollScale' End of test 'tst_ScrollScale' ******************************************************* Summary: Number of Test Cases: 1 Number of Tests: 1 Number of Errors: 0 Number of Fatals: 0 Number of Fails: 0 Number of Passes: 1 Number of Expected Fails: 0 Number of Unexpected Passes: 0 Number of Warnings: 0 ******************************************************* Stopping the squish server.../home/barre/build/squish-20081218-tk-src/bin/squishserver --stop Attempting to shut down server at port 4322 Done -- Process completed Passed [...] The following tests passed: KWWidgets-CallbacksSquish 100% tests passed, 0 tests failed out of 1
Known Issues
Squish
- Squish does not handle the vtkKWMenuButton widget properly. The replay sequence is sending too many mouse motion events at the application and it takes a long time to playback the corresponding "picking a menu entry" action.
- Similar to the vtkKWMenuButton, too many mouse move events are recorded and played back when performing a mouse "drag" over a render window.
- Squish sends events to the AUT (Application Under Testing) asynchronously during playback; it does not know if the events are actually processed or not, i.e. how long it will actually take the application to react to each event on a computer different than the one that was used to create the test itself.
- Events from secondary toplevel dialogs are not recorded yet. For example, in WidgetsTourExample, select MultiColumnList from the left panel and double click on the color button: this will launch a color-picker dialog, which is KWWidgets's own color dialog. Squish does not seem to be able to record any of the mouse event coming from this toplevel dialog.
- Squish does not record/track the file name from a file dialog, since it just records the mouse events without knowledge of the semantics of the widget. The same is true for the color-picker-dialog, as reported previously.
KWWidgets
- In order to validate tests that operate on image data, a KWWidgets application needs to be able to grab a screenshot of its VTK render windows, so that baseline images can be created and compared at the end of a Squish test, for validation purposes. This shouldn't be too hard to add since most of VTK is tested that way.