ITK/Release 4/Modularization/Modular Dashboard

Modular Dashboard

• ITK Modular dashboard is no longer in use. Maybe we will use a modular dashboard for remote modules only.

The following is the historical discussion:

Bugs ?

• Dynamic analysis doesn't show up in the summary page of the toolkit.
  • http://www.cdash.org/CDash/index.php?project=ITKModular&display=project
• The main dashboard shows 75 errors. Now I realize that this is a machine count. However, to investigate the errors, I click on ITK-CurvatureFlow. I see one machine with 156 compile errors. Now, I go to look at the next module:ITK-TestKernel which shows the same 156 compile errors. IOBase is the same. In fact most (I did not bother visiting every module page) pages report the same errors.
• Navigate from the top level to here. Although the page looks similar to the old top level page,
  • none of the counts are hyper-linked
  • there are many '-1's'
  • in the Continuous section, some numbers are linked and some are not
• Navigation error
  • Go here.
  • Then go here.
  • Clicking on the cdash Back button, ends up here rather than the page we came from.
• Missing Back button.
  • Starting from the main dashboard.
  • Navigate here.
  • There is no Back button. It is easy not to notice this and hit the "Previous" button instead, which goes to the same page, but the previous day.

Desired Features

• There is no way to move or hide the Sub Projects section. See this example.
• The counts on the main dashboard are inconsistent. In the SubProjects
  • the Configure numbers refer to machine counts
  • the Build numbers refer to machine counts
  • the Test numbers refer to number of tests
• It would be great if one could select the old format using an alternate http: link or a button on the top page. This would certainly help the transition and also allow developers a choice of presentations.

Speed

Please separate speed and performance issues from actual features.

• Kitware is getting a virtual cluster machine to replace the current Dashboard server, so it is anticipated that this hardware replacement will take care of speed and performance issues.
• The performance is much slower than the original dashboard. For example this new page took 47 seconds to load while this old page took 1.1 seconds. New is about 40 times slower than old. Even with a server upgrade the performance will be slower than the old dashboard.

Individual Observations

• Lorensen: My initial impression of the itk modular dashboard is that is not in a form that is usable for ITK at this time. The navigation is cumbersome and it is difficult to use on a day-to-day basis. I hope we can have more discussion about this.
  • Proposal
    • Try the new format for two weeks.
    • Gather feedback on this page.
    • Poll the developers with a go/nogo.
    • If go, convert current submissions to the new format.
    • If nogo, revert all submissions to the non-modular dashboard and wait for a new cdash release.

• Luis: Please phrase this into actionable items or actual well-defined desired features, so we can have a productive discussion.

• Brad L: I have concerns about the performance beyond just the machine. As I do not know if there is IO, or CPU issues my observation is rather general. The response is slow and it appears to do a significant amount of work to load a page, even reloading it a moment later. I wonder if there a a better way to do some type of caching to reduce the computation needed.

How We are Addressing These Concerns

New Hardware

• We will be replacing the Dashboard server with new hardware, a virtual cluster, that should be online several weeks from now.
• This new hardware will improve the performance, but, as some of you already pointed out, it may or many not be enough to fully address the responsiveness of the web pages.

Workarounds

• Follow this link to view all non-green subproject submissions.

Step Back to Monolithic Dashboard

• For the purpose of maintaining the rhythm of the development team, we suggest to step back to the Monolithic Dashboard, while in parallel we continue working on the Modular dashboard to make it a useful tool for the ITK developers community.
• Essential builds should be reconfigured back to submit to the Monolithic Dashboard. In particular
  • Code coverage builds
  • Valgrind builds
  • Unusual platforms
    • MinGW
    • SunCC
• A number of builds should still be submitting to the Modular dashboard, so we have a realistic load to experiment with new organization of the queries and change modifications to CDash. Probably 20 to 30 submissions would be enough for this purpose.

Use Cases

• We need to gather use cases of Modular Dashboard usage, in order to ensure that we provide efficient access to them.
• Some examples are provided below.
• Please help us gather more.

Case 1

• What is broken today ?
  • This is the common question of a developer who merged some changes yesterday, and quickly want to see if any of them broke anything in the Dashboard

Case 2

• What is broken in this platform ?

Case 3

• In what platforms is Test X failing ?

Case 4

• When did Test X start failing ?

Case 5

• What is the code coverage of Module X ?

Case 6

• Are there any valgrind issues today ?