ITK/Testing

Testing plays a vital role in the ITK development process. We use the open source, web-based CDash Dashboard Server to display test results.

Is ITK tested regularly?

ITK is tested throughout the day on machines distributed around the world. You can view the results at:

http://open.cdash.org/index.php?project=Insight

ITK is tested on a variety of platforms and configurations. Nightlies and gerrit submissions are ensured to be tested on Windows, OS X and Linux OS. A wide range of compiler versions are tested. Despite intensive testing, note that not all possible compilers and configurations (i.e. external modules, third party-dependent modules, other CMake flags, etc.) are tested.

The ITK components that get tested on a regular basis include:

• The ITK code.
• The Python wrap.
• The Software Guide.
• The Examples.
• The Remote modules (non-exhaustive).

ITK uses CDash to perform builds, run tests and generate dashboards. You can find more information about CDash at http://www.cdash.org.

What is the testing schedule?

CDash opens a daily dashboard at 9:00PM Eastern Daylight Time (EDT) and keeps that dashbooard open for 24 hours. Build/Test results submitted as "nightlies" (Expected Nightly) will appear on the currently opened dashboard. Each test site can select a convenient time to run tests.

Fresh dashboards are generated every hour, on the hour.

In addition to nightly testing, several sites run Continuous build/tests. These sites monitor git repository activity and begin a build/test sequence if there are updates to the repository. If a compile error is detected, the Dashboard server sends an email to the "offending" party (author).

For patches submitted for review to gerrit, the Kitware Build Robot gerrit user automatically builds and tests every patch set on Windows, OS X and Linux OS platforms. These are listed under the Gerrit section.

Valgrind is used to identify dynamic analysis defects (e.g. uninitialized variables, memory leaks, etc.) and is run nightly. Its results are displayed under the Dynamic Analysis section. If a defect is found, the Dashboard server sends an email to the "offending" party (author).

Finally, Experimental builds can also be submitted any time by developers around the world.

What kind of testing is performed?

ITK has three types of tests:

1. Unit Tests. Also called white box tests. These tests reside inside each module's test directory. There are meant to exercise a class's methods and verify that the code compiles, links and produces expected results. These tests do not need to produce results that are useful to a user. If a test returns a 0 status, it passes. A test fails if it returns a non-zero status.
2. Blackbox Tests. These tests are automatically (or semi-automatically) generated. For example, each Code directory has a test called itkXXXHeaderTest. This test is automatically created and has a "#include" for each file in the directory.
3. Regression Tests. These tests generate results and compare them with a known result. The purpose of regression testing is to detect changes in the output of the system.

How Do I Add a Test?

The ITK Software Guide Section 9.4 provides detailed explanations on how to create a test.

What is a continuous build and how can I contribute one?

Continuous builds run throughout the day and report their output in the Continuous Builds portion of the dashboard. The most recent build is at the top of the list. The builds run periodically and check the status of the repository. If a file is added or changed since the last build, the working repository of the continuous build is updated, the code compiled and tests executed. Optional e-mail notifications can be sent if there are errors, warnings or test failures.

You can set up a continuous build. This entails writing some small shell scripts and setting up a scheduler (Scheduler for Windows, cron for Linux/Unix). You will also need to install tcl, checkout Dart and configure your build for testing.

For Windows 2000, here are three batch files that can be modified for your own environment.

AllowInsightContinuous.bat
BuildInsightContinuous.bat
PreventInsightContinuous.bat

The batch files use a lock to prevent/allow the continuous

rem ************* AllowInsightContinuous.bat
rem Allow the Continuous to run
rem
c:
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  del continuous.lock
)
cd ..
rem ************* BuildInsightContinuous.bat
rem DO a C++ Build Continuously
rem
c:
rem Setup proxy it behind a firewall
set HTTP_PROXY=webcache
set HTTP_PROXY_PORT=8080
rem
rem Setup the path
rem
set PATH=c:/lorensen/InsightContinuous/Insight-VC6/bin/Release;%PATH%
rem
rem Tell cvs where .cvspass resides
set HOME=c:/lorensen
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  exit /B
) else (
  date /T > continuous.lock
  time /T >> continuous.lock
  cd c:\lorensen\InsightContinuous\Insight-VC6
 tclsh83 c:/lorensen/Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl Continuous Start Update Configure Build Test Submit
 cd c:\lorensen\InsightContinuous\Insight-VC6
 del continuous.lock
 cd ..
)
rem ***************** PreventInsightContinuous.bat
rem Prevent the Continuous
rem
c:
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  exit /B
) else (
  echo "Prevent continuous for nightly" > continuous.lock
)
cd ..

Now you need to establish a schedule for the three batch jobs. Use the "Scheduled Tasks" on the "Control Panel" to access the Schedule Task Wizard. The following schedule runs the continuous every 10 minutes from 6:05AM until 12:00AM. The Allow and Prevent scripts turn the build ON/OFF at the appropriate time. Schedule each batch file as follows:

AllowInsightContinuous.bat - Daily, Start 6:00AM
BuildInsightContinuous.bat - Daily, Start 6:05AM, Advanced: Repeat Task every 10 minutes, duration 17 hours and 55 minutes
PreventContinuous.bat - Daily, Start 12:00AM

Supporting ITK development process can be as easy as contributing testing results from your platform.

• Submit testing results using CTest

---

ITK: [Welcome | Site Map]