MantisBT - ParaView
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0009808ParaViewFeaturepublic2009-10-30 14:032015-01-11 08:16
ReporterAlan Scott 
Assigned ToUtkarsh Ayachit 
PriorityhighSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target Version4.2Fixed in Version4.2 
ProjectSandia
Topic Nameremove_duplicate_tiff_readers
Typeincorrect functionality
Summary0009808: Ability to automatically select min, max, or average cell or point at a time step.
DescriptionWe need to add the functionality to have ParaView automatically select the cell or point that represents a variable's min or max or average value. This should work for all time steps - thus the selected cell or point would move around with time.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
related to 0010958closed Utkarsh Ayachit Edit/Find data command - find and calculate min and max variable for each time step of an animation 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2009-10-30 14:03Alan ScottNew Issue
2009-10-30 14:10Alan ScottNote Added: 0018304
2010-08-30 22:22Alan ScottRelationship addedrelated to 0010958
2010-09-01 11:32Utkarsh AyachitTarget Version => 3.10.shortlist
2010-10-26 17:10Alan ScottNote Added: 0022711
2010-10-26 17:10Alan ScottNote Added: 0022712
2010-12-01 16:53Utkarsh AyachitAssigned To => Robert Maynard
2010-12-01 16:53Utkarsh AyachitStatusbacklog => tabled
2010-12-01 16:53Utkarsh AyachitPrioritynormal => high
2011-06-16 13:10Zack GalbreathCategoryFeature Request => Feature
2011-06-24 14:24Alan ScottStatusexpired => backlog
2011-06-24 14:24Alan ScottResolutionopen => reopened
2011-06-24 14:24Alan ScottStatusbacklog => todo
2011-06-27 08:32Utkarsh AyachitProject => Sandia
2012-05-31 10:23Utkarsh AyachitStatustodo => backlog
2012-11-07 21:33Alan ScottNote Added: 0029664
2012-11-07 21:33Alan ScottStatusbacklog => todo
2012-12-11 09:47Sebastien JourdainAssigned ToRobert Maynard => David Thompson
2014-01-16 17:00Utkarsh AyachitType => incorrect functionality
2014-01-16 17:00Utkarsh AyachitAssigned ToDavid Thompson => David C. Lonie
2014-01-16 17:00Utkarsh AyachitTarget Version => 4.2
2014-03-18 08:40David C. LonieNote Added: 0032481
2014-03-18 08:40David C. LonieStatustodo => closed
2014-03-18 08:40David C. LonieResolutionreopened => fixed
2014-03-18 08:40David C. LonieFixed in Version => git-master
2014-03-18 13:10Utkarsh AyachitStatusclosed => customer review
2014-03-18 13:11Utkarsh AyachitNote Added: 0032483
2014-03-19 20:03Alan ScottNote Added: 0032486
2014-03-19 20:04Alan ScottNote Added: 0032487
2014-03-19 20:04Alan ScottStatuscustomer review => todo
2014-04-08 10:55David C. LonieTopic Name => 9808-distributed-query-selection
2014-04-08 10:55David C. LonieNote Added: 0032551
2014-04-08 10:55David C. LonieStatustodo => gatekeeper review
2014-04-08 10:55David C. LonieResolutionfixed => open
2014-04-09 09:55Utkarsh AyachitStatusgatekeeper review => customer review
2014-04-09 09:55Utkarsh AyachitNote Added: 0032579
2014-04-09 09:55Utkarsh AyachitAssigned ToDavid C. Lonie => Utkarsh Ayachit
2014-04-09 09:55Utkarsh AyachitNote Added: 0032580
2014-04-09 09:55Utkarsh AyachitStatuscustomer review => gatekeeper review
2014-04-09 09:55Utkarsh AyachitResolutionopen => fixed
2014-04-14 09:33David C. LonieStatusgatekeeper review => active development
2014-04-14 09:34David C. LonieTopic Name9808-distributed-query-selection => 9808-distributed-query-selection, 9808-bugfix
2014-04-14 09:34David C. LonieNote Added: 0032589
2014-04-14 09:34David C. LonieStatusactive development => gatekeeper review
2014-05-17 22:23Utkarsh AyachitNote Added: 0032645
2014-05-17 22:23Utkarsh AyachitStatusgatekeeper review => todo
2014-05-17 22:29Utkarsh AyachitNote Added: 0032646
2014-07-18 10:05Utkarsh AyachitTopic Name9808-distributed-query-selection, 9808-bugfix => fix_find_data
2014-07-18 10:05Utkarsh AyachitStatustodo => gatekeeper review
2014-07-18 10:07Utkarsh AyachitNote Added: 0033034
2014-07-18 10:07Utkarsh AyachitStatusgatekeeper review => customer review
2014-07-18 20:46Alan ScottNote Added: 0033039
2014-07-18 21:11Alan ScottNote Added: 0033042
2014-07-21 14:23Utkarsh AyachitNote Added: 0033045
2014-07-21 21:23Alan ScottNote Added: 0033049
2014-07-24 16:12Utkarsh AyachitNote Added: 0033077
2014-07-24 16:13Utkarsh AyachitTopic Namefix_find_data => remove_duplicate_tiff_readers
2014-07-24 16:13Utkarsh AyachitStatuscustomer review => gatekeeper review
2014-07-25 09:35Utkarsh AyachitStatusgatekeeper review => backlog
2014-07-25 09:35Utkarsh AyachitNote Added: 0033092
2014-07-25 09:38Utkarsh AyachitStatusbacklog => customer review
2014-07-28 18:17Alan ScottNote Added: 0033112
2014-07-28 18:17Alan ScottStatuscustomer review => closed
2014-08-29 10:59Utkarsh AyachitFixed in Versiongit-master => 4.2
2015-01-11 08:15Utkarsh AyachitSource_changeset_attached => ParaView master 879dbb3d
2015-01-11 08:16Utkarsh AyachitSource_changeset_attached => ParaView master f6dc4055

Notes
(0018304)
Alan Scott   
2009-10-30 14:10   
A stretch goal would be to use the output of this filter as input to the plot selection over time filter.
(0022711)
Alan Scott   
2010-10-26 17:10   
A user of mine just asked for this "stretch goal". We really do need to include my note above as part of this feature.
(0022712)
Alan Scott   
2010-10-26 17:10   
A user of mine just asked for this "stretch goal". We really do need to include my note above as part of this feature.
(0029664)
Alan Scott   
2012-11-07 21:33   
Not sure if this belongs assigned to Robert, but issuing a Todo. The Find Data dialog is important.
(0032481)
David C. Lonie   
2014-03-18 08:40   
This bug is quite old, and seems to already be implemented:

1) Open can.ex2, load all arrays
2) Open find data dialog
3) Find Cells of can.ex2, set query to 'EQPS == max(EQPS)'
4) Walk through timesteps. The selection will updated.

Feel free to reopen if there is more work needed.
(0032483)
Utkarsh Ayachit   
2014-03-18 13:11   
(developers should never close bugs directly without going through gatekeeper review/customer review). I've also change workflow permissions to reflect the same.
(0032486)
Alan Scott   
2014-03-19 20:03   
From an e-mail thread between Dave, Utkarsh and me:

From Alan:
> What I want is to have some very basic options such as min and max
> available as a button, or pull down menus, etc. Think of a very
> bright person that has never programmed in their life (such as a PhD
> in Geology or an MD doing medical research). If I were king, and i
> realize this is harder, I would like to be able to select the max and
> min cell per time step, and the max and min over all time.
>
> At the minimum - total minimum, we should have it trivial to find good
> quality documentation - with examples - of how this works.
>
> My/our rule of thumb is that if it doesn't work correctly in parallel,
> it is broken. And, misrepresenting data is significantly worse than
> not giving any data at all. And thanks for testing all the
> combinations (which is what I will do when I close it).

Regarding the query on the Find Data:

From Dave:
>
> BTW, the Query option is in a drop-down menu on the find data dialog.
> It's the first combo box in the second section of the dialog.

From Alan
>
> Ah, I found it. Big picture - I don't like
> it. We shouldn't have a list of variable names intermixed with a
> command. It is confusing. Could we move query to be its own line, or
> its own button, or something? At the least, could we create a space
> above Query and bold any commands (such as Query)?


From Dave:
>
> I'm still tracking down the issues with doing this on a distributed dataset.
> I got the calculations to run without crashing, but got different
> results depending on the data distribution (e.g. multiblock vs. single
> data set, with/without D3, etc). I'll hold off on the GUI elements
> until this is behaving properly.
>
>
(0032487)
Alan Scott   
2014-03-19 20:04   
Nope, doesn't work. See notes from the e-mail thread below.
(0032551)
David C. Lonie   
2014-04-08 10:55   
Fixed problems with the "arrayname == global_max(arrayname)" query on datasets with unevenly distributed data arrays by syncing array names and ensuring that all processors have the same names available in their scope.

This does not address the usability concerns raised by Alan. Further GUI work will be needed to expose the functionality in a user-friendly way, but now the underpinnings should be ready.
(0032579)
Utkarsh Ayachit   
2014-04-09 09:55   
SUMMARY
---------------------------------------------
Topics merged into master:
        14220-hide-decoration-button
(VTK) 14646-scalar-bar-borders
        14663_timesteps_indicator
        9808-distributed-query-selection
        catalyst-rgbz-plugin
(VTK) view-settings


---------------------------------------------
Topics reverted from next:
  expand_fragment_integration
(0032580)
Utkarsh Ayachit   
2014-04-09 09:55   
leaving this one open so I can test it before passing on to Alan.
(0032589)
David C. Lonie   
2014-04-14 09:34   
9808-bugfix should fix the dashboard issues on blight.
(0032645)
Utkarsh Ayachit   
2014-05-17 22:23   
Argh, still doesn't work fully. The following query fails:

+ builtin mode
+ open can.ex2, move to timestep > 0 (e.g. 15)
+ run query for Points, VEL[:,1] == global_min(VEL[:,1]) -- we get two points selected -- min for each block is selected.
+ if you run Python calculator with expression "global_min(VEL[:,1])", it works fine.

+ It's curious that cell query, EQPS == global_min(EQPS) works just fine
(0032646)
Utkarsh Ayachit   
2014-05-17 22:29   
Some more insight into the problem:

Attached is a simple dataset with 2 spheres as 2 blocks with "Result" array set to 1 and 2 respectively.

The following query fails:
  Result[:,0] == global_max(Result[:,0])

The following succeeds:
  Result == global_max(Result)

The problem stems from the fact that Result[:,0] is a new array and hence the code doesn't treat it as a composite-array. I think we may have to give up on Python implementation for this and just and custom extract selection support for max/min/mean.
(0033034)
Utkarsh Ayachit   
2014-07-18 10:07   
Alright, resolving this again, fingers crossed.

The FindData dialog now has "is min", "is max", "is mean", "is mean within threshold" options that will help you find elements that min, max etc.

And it should work in parallel too.

Note, it uses NumPy.

Also another caveat: FindData by just IDs is currently not working and it will raise warnings. I'm working on it. Feel free to leave this bug open till that's resolved. I'm marking this "customer review" so that you can test the min/max functionality.
(0033039)
Alan Scott   
2014-07-18 20:46   
Yes!! OK, I will test.

OK, tested! It died. I did the following:

master, Linux, remote server (8 pvservers).
Numpy support Client side: On (and using my numpy), version 1.7.1
Numpy support Server side: On (and using my numpy), version 1.6.1 (Could this be the problem?)
Python version matches (and is mine), 2.7.5.

  
Opened g1s1 (ask me, Do Not Release).
Find Data/ EQPS/ find max
Died. Here is what the client put out:

paraview: /.../ParaView4/ParaViewCore/ClientServerCore/Rendering/vtkSpreadSheetView.cxx:83: bool<unnamed>::OrderByNames::operator()(vtkAbstractArray*, vtkAbstractArray*): Assertion `a1Index == static_cast< int >(~0u >> 1) && a2Index == static_cast< int >(~0u >> 1)' failed.

Server side just grumbled that the socket had died.
(0033042)
Alan Scott   
2014-07-18 21:11   
Tested remote server, can.exo, and disk_out_ref.exo. Worked great!
(0033045)
Utkarsh Ayachit   
2014-07-21 14:23   
Grrr...this seems to work for me. Can you try again please? Also make sure you start with a clean build on client and server. The python code build/install logic is a bug wacky.
(0033049)
Alan Scott   
2014-07-21 21:23   
Still asserts. I replicated it as follows:

Deleted all .config/ParaView/* files. Master as of today, clean build, Linux, 8 pvservers, client debug, client running under totalview. Assert is from the client.

Open g1s1. There will be 16 files. All variables on. Apply.
Click on the Find Data icon (I didn't use the Edit/ Find Data menu.)
Change Global Id to EQPS. Change "is" to "Is Max". Click on Run Selection Query. Assert.

We hit an assert in vtkSpreadSheetView.cxx, line 83. Basically, a1->GetName() and a2->GetName() don't equal anything in the order[] array. I believe that a1 and a2 GetName() is just null. Thus, a result is never found, and the assert asserts.

Furthermore, in vtkSpreadSheetView.cxx, vtkSpreadSheetView::GetNumberOfColumns(), I believe that this->Internals->ActiveRepresentation and MostRecentlyAccessedBlock may be bad. Sending Utkarsh a screenshot at this point. Note that I am showing this, then what is in *(this), then *(this->Internals).

Next, showing the call stack, and what is in FetchBlock. Notice that block index == 0. Not sure where else to go from here... :-)
(0033077)
Utkarsh Ayachit   
2014-07-24 16:12   
commit f6dc4055de7af65df0e1bde427ca09416611bf42
Author: Utkarsh Ayachit <utkarsh.ayachit@kitware.com>
Date: Thu Jul 24 16:11:40 2014 -0400

    Fix segfault seen in BUG 0009808.
    
    On certain platforms, the comparison operator is called for sorting with
    same arguments. We skipped that case in our "assert". Fixed that.
    
    Change-Id: I8fdc496d2b078878dca21eca64b8b8ce3547285
(0033092)
Utkarsh Ayachit   
2014-07-25 09:35   
SUMMARY
---------------------------------------------
Topics merged into master:
        13575_PV_PLUGIN_DEBUG
        14771_fix_global_annotation_filter
        14885_python_view_crash
        fix_apply_pqProxyWidgetDialog
        fix_get_proxies
        fix-int64_t
(VTK) remove_duplicate_tiff_readers
        remove_matplotlib_utilities
(0033112)
Alan Scott   
2014-07-28 18:17   
Spectacular!!!!!

Tested g1s1, max EQPS, remote server, Linux, master.