View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006065ParaView(No Category)public2007-11-19 18:252008-03-07 07:56
ReporterDavid Karelitz 
Assigned ToUtkarsh Ayachit 
PrioritynormalSeveritycrashReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0006065: (MARS-immediate) Client often hangs when running with very large AMRCTH data in parallel
DescriptionWhen running in parallel with AMR CTH data on 80-120 processes, the client often hangs (does not redraw) when there hasn't been any interaction for a long time (~10 minutes).

The 0th node on the server is waiting in a select() call.
TagsNo tags attached.
Project
Topic Name
Type
Attached Files

 Relationships

  Notes
(0009709)
David Karelitz (reporter)
2007-11-20 11:46

Here's a few stack traces while running on 70 nodes with 140 processes

Client stack trace:
#0 0x000000303460b5b4 in recv () from /lib64/tls/libpthread.so.0
#1 0x0000002a9a6d112b in vtkSocket::Receive (this=0x1047420, data=0x7fbfffd64c, length=4, readFully=1)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocket.cxx:376
#2 0x0000002a9a6d6849 in vtkSocketCommunicator::ReceiveTagged (this=0xdf9020, data=0x11b0280, wordSize=4, numWords=1, tag=834340,
    logName=0x13492a8 "Int32") at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocketCommunicator.cxx:655
#3 0x0000002a9a6d3dd6 in vtkSocketCommunicator::ReceiveVoidArray (this=0xdf9020, data=0x11b0280, length=1, type=6, remoteProcessId=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocketCommunicator.cxx:273
#4 0x0000002a98705785 in vtkCommunicator::Receive (this=0xdf9020, data=0x11b0280, length=1, remoteHandle=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#5 0x0000002a98705720 in vtkMultiProcessController::Receive (this=0x2aa32cb010, data=0x11b0280, length=1, remoteProcessId=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkMultiProcessController.h:1055
#6 0x0000002a9876c8f5 in vtkPVDesktopDeliveryClient::PreRenderProcessing (this=0x11b0150)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Filters/vtkPVDesktopDeliveryClient.cxx:210
#7 0x0000002a9a6bf69c in vtkParallelRenderManager::StartRender (this=0x11b0150)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkParallelRenderManager.cxx:705
#8 0x0000002a9a6c3622 in StartRender (clientData=0x11b0150)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkParallelRenderManager.cxx:1821
0000009 0x0000002a9f7fd7b2 in vtkCallbackCommand::Execute (this=0x11b2340, caller=0x11d5980, event=3, callData=0x0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkCallbackCommand.cxx:42
#10 0x0000002a9f8b6e70 in vtkSubjectHelper::InvokeEvent (this=0x11cb170, event=3, callData=0x0, self=0x11d5980)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkObject.cxx:547
0000011 0x0000002a9f8b72b8 in vtkObject::InvokeEvent (this=0x11d5980, event=3, callData=0x0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkObject.cxx:713
#12 0x0000002a9bc41958 in vtkRenderWindow::Render (this=0x11d5980) at /home/projects/ParaView3-Trunk/ParaView3/VTK/Rendering/vtkRenderWindow.cxx:257
#13 0x0000002a9bd0221c in vtkXOpenGLRenderWindow::Render (this=0x11d5980)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Rendering/vtkXOpenGLRenderWindow.cxx:1782
#14 0x0000002a980ff49f in vtkSMRenderViewProxy::PerformRender (this=0x11585c0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMRenderViewProxy.cxx:556
#15 0x0000002a98147df8 in vtkSMViewProxy::StillRender (this=0x11585c0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMViewProxy.cxx:374
#16 0x0000002a980fc9de in vtkSMRenderViewHelper::EventuallyRender (this=0x1158c20)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMRenderViewHelper.cxx:39
#17 0x0000002a98784810 in vtkPVGenericRenderWindowInteractor::Render (this=0x11b20f0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Filters/vtkPVGenericRenderWindowInteractor.cxx:209
#18 0x0000002a97da9ad1 in QVTKWidget::paintEvent (this=0x11a6860) at /home/projects/ParaView3-Trunk/ParaView3/VTK/GUISupport/Qt/QVTKWidget.cxx:494
#19 0x0000002a9672972a in QWidget::event (this=0x11a6860, event=0x7fbfffe430) at kernel/qwidget.cpp:5847
#20 0x0000002a97da983f in QVTKWidget::event (this=0x11a6860, e=0x7fbfffe430)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/GUISupport/Qt/QVTKWidget.cxx:419
#21 0x0000002a966e28e8 in QApplicationPrivate::notify_helper (this=0x5944f0, receiver=0x11a6860, e=0x7fbfffe430) at kernel/qapplication.cpp:3437
#22 0x0000002a966e2a51 in QApplication::notify (this=0x7fbffff3a0, receiver=0x11a6860, e=0x7fbfffe430) at kernel/qapplication.cpp:3378
#23 0x0000002a9672fd1c in qt_sendSpontaneousEvent (receiver=Variable "receiver" is not available.
) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:184
#24 0x0000002a96810084 in QWidgetPrivate::drawWidget (this=Variable "this" is not available.
) at painting/qbackingstore.cpp:768
#25 0x0000002a9681116e in QWidget::repaint (this=0x11a6860, rgn=@0x7fbfffe640) at ../../include/QtCore/../../src/corelib/tools/qpoint.h:89
#26 0x0000002a9681144c in qt_syncBackingStore (rgn=Cannot access memory at address 0xc
) at painting/qbackingstore.cpp:224
#27 0x0000002a9673badd in QETWidget::translatePaintEvent (this=0x11a6860, event=Variable "event" is not available.
) at kernel/qapplication_x11.cpp:4225
#28 0x0000002a9673c5fc in QApplication::x11ProcessEvent (this=0x7fbffff3a0, event=0x7fbfffecd0) at kernel/qapplication_x11.cpp:2787
#29 0x0000002a9675dfa2 in x11EventSourceDispatch (s=0x5a01c0, callback=0, user_data=0x0)
    at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:75
#30 0x0000003035e266bd in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#31 0x0000003035e28397 in g_main_context_acquire () from /usr/lib64/libglib-2.0.so.0
#32 0x0000003035e288eb in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#33 0x0000002a970776bd in QEventDispatcherGlib::processEvents (this=0x59db30, flags={i = 20}) at kernel/qeventdispatcher_glib.cpp:364
#34 0x0000002a9675da69 in QGuiEventDispatcherGlib::processEvents (this=Variable "this" is not available.
) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1633
#35 0x0000002a9705542e in QEventLoop::processEvents (this=Variable "this" is not available.
) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1633
#36 0x0000002a970555e0 in QEventLoop::exec (this=0x7fbfffefa0, flags={i = 0}) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1650
#37 0x0000002a97059c7a in QCoreApplication::exec () at ../../include/QtCore/../../src/corelib/global/qglobal.h:1634
#38 0x0000002a95be6f20 in pqProcessModuleGUIHelper::RunGUIStart (this=0x5f8940, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Qt/Core/pqProcessModuleGUIHelper.cxx:194
#39 0x0000002a98b5cc3e in vtkProcessModule::StartClient (this=0x65dbb0, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModule.cxx:381
#40 0x0000002a98b5c8cc in vtkProcessModule::Start (this=0x65dbb0, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModule.cxx:327
#41 0x0000002a98b6663a in vtkProcessModuleGUIHelper::Run (this=0x5f8940, options=0x65b0f0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModuleGUIHelper.cxx:38
#42 0x0000002a95bb6233 in pqMain::Run (app=@0x7fbffff3a0, helper=0x5f8940) at /home/projects/ParaView3-Trunk/ParaView3/Qt/Core/pqMain.cxx:129
#43 0x000000000042c2c7 in main (argc=2, argv=0x7fbffff4a8) at /home/projects/ParaView3-Trunk/ParaView3/Applications/Client/main.cxx:62


node0 proc0:
#0 0x0000002aa06bfb02 in btl_openib_component_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_btl_openib.so
#1 0x0000002aa05b2272 in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#2 0x0000002a9d8d9a8a in opal_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libopen-pal.so.0
#3 0x0000002aa04a4c9c in mca_pml_ob1_recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_pml_ob1.so
#4 0x0000002a9d63a009 in PMPI_Recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libmpi.so.0
#5 0x0000002a99ea1e62 in vtkMPICommunicatorReceiveData<char> (data=0x7fbfffe0a0 "�\020\006", length=3, sizeoftype=4, remoteProcessId=-1, tag=1,
    datatype=0x2a9d767ce0, Handle=0xd53540, useCopy=0, senderId=@0xdf1ec0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:248
#6 0x0000002a99e9eb59 in vtkMPICommunicator::ReceiveVoidArray (this=0xdf1e80, data=0x7fbfffe0a0, length=3, type=6, remoteProcessId=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:732
#7 0x0000002a955d6191 in vtkCommunicator::Receive (this=0xdf1e80, data=0x7fbfffe0a0, length=3, remoteHandle=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#8 0x0000002a99dff98a in vtkMultiProcessController::ProcessRMIs (this=0xd783b0, reportErrors=1, dont_loop=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:397
0000009 0x0000002a99dff941 in vtkMultiProcessController::ProcessRMIs (this=0xd783b0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:385
#10 0x0000002a955d8bd2 in vtkMPISelfConnection::InitializeSatellite (this=0xd43f00)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:142
0000011 0x0000002a955d8ae3 in vtkMPISelfConnection::Initialize (this=0xd43f00, argc=1, argv=0x9055f0, partitionId=0x7fbfffe1dc)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:103
#12 0x0000002a955dc0ed in vtkProcessModuleConnectionManager::Initialize (this=0xdd7290, argc=1, argv=0x9055f0, clientMode=0,
    partitionId=0x7fbfffe1dc) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:135
#13 0x0000002a955e4b7e in vtkProcessModule::Start (this=0x934090, argc=1, argv=0x9055f0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:301
#14 0x0000002a95952842 in vtkPVMain::Run (this=0x905010, options=0x934ea0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#15 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68

node0 proc1:
#0 0x00000035160bf8f5 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x0000002a99e53a1c in vtkSocket::SelectSockets (sockets_to_select=0x2f3f9540, size=1, msec=0, selected_index=0x7fbfffe02c)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkSocket.cxx:214
#2 0x0000002a99e547ec in vtkSocketCollection::SelectSockets (this=0xdc1690, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkSocketCollection.cxx:81
#3 0x0000002a955dd04f in vtkProcessModuleConnectionManager::MonitorConnections (this=0xdd7270, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:318
#4 0x0000002a955e5078 in vtkProcessModule::StartServer (this=0x934480, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:413
#5 0x0000002a955e4c64 in vtkProcessModule::Start (this=0x934480, argc=1, argv=0x9053b0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:331
#6 0x0000002a95952842 in vtkPVMain::Run (this=0x904dd0, options=0x935290)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#7 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68

node1 proc0:
#0 0x0000002aa05b228e in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#1 0x0000000000000000 in ?? ()

node1 proc1:
#0 0x0000002aa06bfa63 in btl_openib_component_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_btl_openib.so
#1 0x0000002aa05b2272 in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#2 0x0000002a9d8d9a8a in opal_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libopen-pal.so.0
#3 0x0000002aa04a4c9c in mca_pml_ob1_recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_pml_ob1.so
#4 0x0000002a9d63a009 in PMPI_Recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libmpi.so.0
#5 0x0000002a99ea1e62 in vtkMPICommunicatorReceiveData<char> (data=0x7fbfffe0a0 "�\020\006", length=3, sizeoftype=4, remoteProcessId=-1, tag=1,
    datatype=0x2a9d767ce0, Handle=0xd53540, useCopy=0, senderId=@0xdf1ec0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:248
#6 0x0000002a99e9eb59 in vtkMPICommunicator::ReceiveVoidArray (this=0xdf1e80, data=0x7fbfffe0a0, length=3, type=6, remoteProcessId=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:732
#7 0x0000002a955d6191 in vtkCommunicator::Receive (this=0xdf1e80, data=0x7fbfffe0a0, length=3, remoteHandle=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#8 0x0000002a99dff98a in vtkMultiProcessController::ProcessRMIs (this=0xd783b0, reportErrors=1, dont_loop=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:397
0000009 0x0000002a99dff941 in vtkMultiProcessController::ProcessRMIs (this=0xd783b0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:385
#10 0x0000002a955d8bd2 in vtkMPISelfConnection::InitializeSatellite (this=0xd43f00)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:142
0000011 0x0000002a955d8ae3 in vtkMPISelfConnection::Initialize (this=0xd43f00, argc=1, argv=0x9055f0, partitionId=0x7fbfffe1dc)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:103
#12 0x0000002a955dc0ed in vtkProcessModuleConnectionManager::Initialize (this=0xdd7290, argc=1, argv=0x9055f0, clientMode=0,
    partitionId=0x7fbfffe1dc) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:135
#13 0x0000002a955e4b7e in vtkProcessModule::Start (this=0x934090, argc=1, argv=0x9055f0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:301
#14 0x0000002a95952842 in vtkPVMain::Run (this=0x905010, options=0x934ea0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#15 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68
(0009710)
David Karelitz (reporter)
2007-11-20 15:51

Here's a stack trace from a 2 node, 4 proc job

client:
#0 0x000000303460b5b4 in recv () from /lib64/tls/libpthread.so.0
#1 0x0000002a9a6d112b in vtkSocket::Receive (this=0x106d3b0, data=0x7fbfffd64c, length=4, readFully=1)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocket.cxx:376
#2 0x0000002a9a6d6849 in vtkSocketCommunicator::ReceiveTagged (this=0x110c2e0, data=0x11b0400, wordSize=4, numWords=1, tag=834340,
    logName=0x1b6e028 "Int32") at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocketCommunicator.cxx:655
#3 0x0000002a9a6d3dd6 in vtkSocketCommunicator::ReceiveVoidArray (this=0x110c2e0, data=0x11b0400, length=1, type=6, remoteProcessId=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkSocketCommunicator.cxx:273
#4 0x0000002a98705785 in vtkCommunicator::Receive (this=0x110c2e0, data=0x11b0400, length=1, remoteHandle=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#5 0x0000002a98705720 in vtkMultiProcessController::Receive (this=0x2aa32cc010, data=0x11b0400, length=1, remoteProcessId=1, tag=834340)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkMultiProcessController.h:1055
#6 0x0000002a9876c8f5 in vtkPVDesktopDeliveryClient::PreRenderProcessing (this=0x11b02d0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Filters/vtkPVDesktopDeliveryClient.cxx:210
#7 0x0000002a9a6bf69c in vtkParallelRenderManager::StartRender (this=0x11b02d0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkParallelRenderManager.cxx:705
#8 0x0000002a9a6c3622 in StartRender (clientData=0x11b02d0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Parallel/vtkParallelRenderManager.cxx:1821
0000009 0x0000002a9f7fe7b2 in vtkCallbackCommand::Execute (this=0x11b24c0, caller=0x11d5b00, event=3, callData=0x0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkCallbackCommand.cxx:42
#10 0x0000002a9f8b7e70 in vtkSubjectHelper::InvokeEvent (this=0x11cb2f0, event=3, callData=0x0, self=0x11d5b00)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkObject.cxx:547
0000011 0x0000002a9f8b82b8 in vtkObject::InvokeEvent (this=0x11d5b00, event=3, callData=0x0)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Common/vtkObject.cxx:713
#12 0x0000002a9bc41958 in vtkRenderWindow::Render (this=0x11d5b00) at /home/projects/ParaView3-Trunk/ParaView3/VTK/Rendering/vtkRenderWindow.cxx:257
#13 0x0000002a9bd0221c in vtkXOpenGLRenderWindow::Render (this=0x11d5b00)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/Rendering/vtkXOpenGLRenderWindow.cxx:1782
#14 0x0000002a980ff49f in vtkSMRenderViewProxy::PerformRender (this=0x1158540)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMRenderViewProxy.cxx:556
#15 0x0000002a98147df8 in vtkSMViewProxy::StillRender (this=0x1158540)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMViewProxy.cxx:374
#16 0x0000002a980fc9de in vtkSMRenderViewHelper::EventuallyRender (this=0x11589f0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/ServerManager/vtkSMRenderViewHelper.cxx:39
#17 0x0000002a98784810 in vtkPVGenericRenderWindowInteractor::Render (this=0x11b2270)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Filters/vtkPVGenericRenderWindowInteractor.cxx:209
#18 0x0000002a97da9ad1 in QVTKWidget::paintEvent (this=0x11a69e0) at /home/projects/ParaView3-Trunk/ParaView3/VTK/GUISupport/Qt/QVTKWidget.cxx:494
#19 0x0000002a9672972a in QWidget::event (this=0x11a69e0, event=0x7fbfffe430) at kernel/qwidget.cpp:5847
#20 0x0000002a97da983f in QVTKWidget::event (this=0x11a69e0, e=0x7fbfffe430)
    at /home/projects/ParaView3-Trunk/ParaView3/VTK/GUISupport/Qt/QVTKWidget.cxx:419
#21 0x0000002a966e28e8 in QApplicationPrivate::notify_helper (this=0x5944f0, receiver=0x11a69e0, e=0x7fbfffe430) at kernel/qapplication.cpp:3437
#22 0x0000002a966e2a51 in QApplication::notify (this=0x7fbffff3a0, receiver=0x11a69e0, e=0x7fbfffe430) at kernel/qapplication.cpp:3378
#23 0x0000002a9672fd1c in qt_sendSpontaneousEvent (receiver=Variable "receiver" is not available.
) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:184
#24 0x0000002a96810084 in QWidgetPrivate::drawWidget (this=Variable "this" is not available.
) at painting/qbackingstore.cpp:768
#25 0x0000002a9681116e in QWidget::repaint (this=0x11a69e0, rgn=@0x7fbfffe640) at ../../include/QtCore/../../src/corelib/tools/qpoint.h:89
#26 0x0000002a9681144c in qt_syncBackingStore (rgn=Cannot access memory at address 0xc
) at painting/qbackingstore.cpp:224
#27 0x0000002a9673badd in QETWidget::translatePaintEvent (this=0x11a69e0, event=Variable "event" is not available.
) at kernel/qapplication_x11.cpp:4225
#28 0x0000002a9673c5fc in QApplication::x11ProcessEvent (this=0x7fbffff3a0, event=0x7fbfffecd0) at kernel/qapplication_x11.cpp:2787
#29 0x0000002a9675dfa2 in x11EventSourceDispatch (s=0x5a01c0, callback=0, user_data=0x0)
    at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:75
#30 0x0000003035e266bd in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#31 0x0000003035e28397 in g_main_context_acquire () from /usr/lib64/libglib-2.0.so.0
#32 0x0000003035e288eb in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#33 0x0000002a970776bd in QEventDispatcherGlib::processEvents (this=0x59db30, flags={i = 20}) at kernel/qeventdispatcher_glib.cpp:364
#34 0x0000002a9675da69 in QGuiEventDispatcherGlib::processEvents (this=Variable "this" is not available.
) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1633
#35 0x0000002a9705542e in QEventLoop::processEvents (this=Variable "this" is not available.
) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1633
#36 0x0000002a970555e0 in QEventLoop::exec (this=0x7fbfffefa0, flags={i = 0}) at ../../include/QtCore/../../src/corelib/global/qglobal.h:1650
#37 0x0000002a97059c7a in QCoreApplication::exec () at ../../include/QtCore/../../src/corelib/global/qglobal.h:1634
#38 0x0000002a95be6f20 in pqProcessModuleGUIHelper::RunGUIStart (this=0x603310, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Qt/Core/pqProcessModuleGUIHelper.cxx:194
#39 0x0000002a98b5cc3e in vtkProcessModule::StartClient (this=0x65dbb0, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModule.cxx:381
#40 0x0000002a98b5c8cc in vtkProcessModule::Start (this=0x65dbb0, argc=1, argv=0x65b500)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModule.cxx:327
#41 0x0000002a98b6663a in vtkProcessModuleGUIHelper::Run (this=0x603310, options=0x65b0f0)
    at /home/projects/ParaView3-Trunk/ParaView3/Servers/Common/vtkProcessModuleGUIHelper.cxx:38
#42 0x0000002a95bb6233 in pqMain::Run (app=@0x7fbffff3a0, helper=0x603310) at /home/projects/ParaView3-Trunk/ParaView3/Qt/Core/pqMain.cxx:129
#43 0x000000000042c2c7 in main (argc=2, argv=0x7fbffff4a8) at /home/projects/ParaView3-Trunk/ParaView3/Applications/Client/main.cxx:62


node0 proc0:
#0 0x0000003ae1dbf8f5 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x0000002a99e53a1c in vtkSocket::SelectSockets (sockets_to_select=0x24a9940, size=1, msec=0, selected_index=0x7fbfffe02c)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkSocket.cxx:214
#2 0x0000002a99e547ec in vtkSocketCollection::SelectSockets (this=0xc74ba0, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkSocketCollection.cxx:81
#3 0x0000002a955dd04f in vtkProcessModuleConnectionManager::MonitorConnections (this=0xc8a780, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:318
#4 0x0000002a955e5078 in vtkProcessModule::StartServer (this=0x90c680, msec=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:413
#5 0x0000002a955e4c64 in vtkProcessModule::Start (this=0x90c680, argc=1, argv=0x8ef020)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:331
#6 0x0000002a95952842 in vtkPVMain::Run (this=0x8f21a0, options=0x8ffaa0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#7 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68

node0 proc1:
#0 0x0000002aa08e98f2 in mca_btl_sm_component_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_btl_sm.so
#1 0x0000002aa05b3272 in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#2 0x0000002a9d8daa8a in opal_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libopen-pal.so.0
#3 0x0000002aa04a5c9c in mca_pml_ob1_recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_pml_ob1.so
#4 0x0000002a9d63b009 in PMPI_Recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libmpi.so.0
#5 0x0000002a99ea1e62 in vtkMPICommunicatorReceiveData<char> (data=0x7fbfffe0a0 "�\020\006", length=3, sizeoftype=4, remoteProcessId=-1, tag=1,
    datatype=0x2a9d768ce0, Handle=0xc06b00, useCopy=0, senderId=@0xca54f0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:248
#6 0x0000002a99e9eb59 in vtkMPICommunicator::ReceiveVoidArray (this=0xca54b0, data=0x7fbfffe0a0, length=3, type=6, remoteProcessId=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:732
#7 0x0000002a955d6191 in vtkCommunicator::Receive (this=0xca54b0, data=0x7fbfffe0a0, length=3, remoteHandle=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#8 0x0000002a99dff98a in vtkMultiProcessController::ProcessRMIs (this=0xc2b970, reportErrors=1, dont_loop=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:397
0000009 0x0000002a99dff941 in vtkMultiProcessController::ProcessRMIs (this=0xc2b970)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:385
#10 0x0000002a955d8bd2 in vtkMPISelfConnection::InitializeSatellite (this=0xbf74c0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:142
0000011 0x0000002a955d8ae3 in vtkMPISelfConnection::Initialize (this=0xbf74c0, argc=1, argv=0x8fffe0, partitionId=0x7fbfffe1dc)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:103
#12 0x0000002a955dc0ed in vtkProcessModuleConnectionManager::Initialize (this=0xc8a8d0, argc=1, argv=0x8fffe0, clientMode=0,
    partitionId=0x7fbfffe1dc) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:135
#13 0x0000002a955e4b7e in vtkProcessModule::Start (this=0x90c910, argc=1, argv=0x8fffe0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:301
#14 0x0000002a95952842 in vtkPVMain::Run (this=0x8e77e0, options=0x8ffaa0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#15 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68


node1 proc0
#0 0x0000002aa06c09b2 in btl_openib_component_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_btl_openib.so
#1 0x0000002aa05b3272 in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#2 0x0000002a9d8daa8a in opal_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libopen-pal.so.0
#3 0x0000002aa04a5c9c in mca_pml_ob1_recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_pml_ob1.so
#4 0x0000002a9d63b009 in PMPI_Recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libmpi.so.0
#5 0x0000002a99ea1e62 in vtkMPICommunicatorReceiveData<char> (data=0x7fbfffe0a0 "�\020\006", length=3, sizeoftype=4, remoteProcessId=-1, tag=1,
    datatype=0x2a9d768ce0, Handle=0xc069e0, useCopy=0, senderId=@0xca53a0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:248
#6 0x0000002a99e9eb59 in vtkMPICommunicator::ReceiveVoidArray (this=0xca5360, data=0x7fbfffe0a0, length=3, type=6, remoteProcessId=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:732
#7 0x0000002a955d6191 in vtkCommunicator::Receive (this=0xca5360, data=0x7fbfffe0a0, length=3, remoteHandle=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#8 0x0000002a99dff98a in vtkMultiProcessController::ProcessRMIs (this=0xc2b850, reportErrors=1, dont_loop=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:397
0000009 0x0000002a99dff941 in vtkMultiProcessController::ProcessRMIs (this=0xc2b850)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:385
#10 0x0000002a955d8bd2 in vtkMPISelfConnection::InitializeSatellite (this=0xbf73a0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:142
0000011 0x0000002a955d8ae3 in vtkMPISelfConnection::Initialize (this=0xbf73a0, argc=1, argv=0x8ef020, partitionId=0x7fbfffe1dc)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:103
#12 0x0000002a955dc0ed in vtkProcessModuleConnectionManager::Initialize (this=0xc8a780, argc=1, argv=0x8ef020, clientMode=0,
    partitionId=0x7fbfffe1dc) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:135
#13 0x0000002a955e4b7e in vtkProcessModule::Start (this=0x90c680, argc=1, argv=0x8ef020)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:301
#14 0x0000002a95952842 in vtkPVMain::Run (this=0x8f21a0, options=0x8ffaa0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#15 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68


node1 proc1:
#0 0x0000003f09309175 in pthread_spin_lock () from /lib64/tls/libpthread.so.0
#1 0x0000002aa1129bf1 in mthca_poll_cq (ibcq=0x8f19c0, ne=1, wc=0x7fbfffdd90) at src/cq.c:467
#2 0x0000002aa06c0b97 in btl_openib_component_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_btl_openib.so
#3 0x0000002aa05b3272 in mca_bml_r2_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_bml_r2.so
#4 0x0000002a9d8daa8a in opal_progress () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libopen-pal.so.0
#5 0x0000002aa04a5c9c in mca_pml_ob1_recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/openmpi/mca_pml_ob1.so
#6 0x0000002a9d63b009 in PMPI_Recv () from /apps/x86_64/mpi/openmpi/gcc-3.4.6/openmpi-1.2.4_ofed/lib/libmpi.so.0
#7 0x0000002a99ea1e62 in vtkMPICommunicatorReceiveData<char> (data=0x7fbfffe0a0 "�\020\006", length=3, sizeoftype=4, remoteProcessId=-1, tag=1,
    datatype=0x2a9d768ce0, Handle=0xc06b00, useCopy=0, senderId=@0xca54f0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:248
#8 0x0000002a99e9eb59 in vtkMPICommunicator::ReceiveVoidArray (this=0xca54b0, data=0x7fbfffe0a0, length=3, type=6, remoteProcessId=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMPICommunicator.cxx:732
0000009 0x0000002a955d6191 in vtkCommunicator::Receive (this=0xca54b0, data=0x7fbfffe0a0, length=3, remoteHandle=-1, tag=1)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkCommunicator.h:190
#10 0x0000002a99dff98a in vtkMultiProcessController::ProcessRMIs (this=0xc2b970, reportErrors=1, dont_loop=0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:397
0000011 0x0000002a99dff941 in vtkMultiProcessController::ProcessRMIs (this=0xc2b970)
    at /gscratch1/dbkarel/ParaView3/ParaView3/VTK/Parallel/vtkMultiProcessController.cxx:385
#12 0x0000002a955d8bd2 in vtkMPISelfConnection::InitializeSatellite (this=0xbf74c0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:142
#13 0x0000002a955d8ae3 in vtkMPISelfConnection::Initialize (this=0xbf74c0, argc=1, argv=0x8fffe0, partitionId=0x7fbfffe1dc)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkMPISelfConnection.cxx:103
#14 0x0000002a955dc0ed in vtkProcessModuleConnectionManager::Initialize (this=0xc8a8d0, argc=1, argv=0x8fffe0, clientMode=0,
    partitionId=0x7fbfffe1dc) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModuleConnectionManager.cxx:135
#15 0x0000002a955e4b7e in vtkProcessModule::Start (this=0x90c910, argc=1, argv=0x8fffe0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Common/vtkProcessModule.cxx:301
#16 0x0000002a95952842 in vtkPVMain::Run (this=0x8e77e0, options=0x8ffaa0)
    at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Filters/vtkPVMain.cxx:235
#17 0x000000000040281a in main (argc=3, argv=0x7fbfffe3d8) at /gscratch1/dbkarel/ParaView3/ParaView3/Servers/Executables/pvserver.cxx:68
(0009711)
David Karelitz (reporter)
2007-11-20 18:35

After some debugging time, it appears that the client stops sending messages to the server after a long wait. The server never receives any RMI's from the client even though the client is waiting for an image from the server
(0009721)
David Karelitz (reporter)
2007-11-21 12:21

This appears to be a problem where the network connection is stealthily dropped. The client sends the correct messages over the socket, but it never receives a TCp acknowledgement that the message was sent. For now we're trying to find out if it's a local network issue. One way to fix this from a paraview perspective is to implement a heartbeat every n seconds when paraview is idle.
(0009762)
Ken Moreland (manager)
2007-11-27 11:37

Dave talked with the network guys who verified that a heartbeat is necessary. Note that the heartbeat needs to be given both at times of activity and inactivity. Certainly the heartbeat should be given periodically when ParaView sits idle waiting for user commands, but it should also be given while the server is doing something. That is, the client should still send heartbeats when the server is running filters or rendering images.
(0009763)
Ken Moreland (manager)
2007-11-27 11:43

More talks with Dave: The non-idle heartbeat may not be as important as the idle heartbeat since the server should be giving progress updates when it is busy. Start with the idle heartbeats. If the timeout occurs during filtering, we can try adjusting the frequency of progress before implementing something more complicated.
(0009764)
Utkarsh Ayachit (administrator)
2007-11-27 13:37

Attempt to fix BUG 0006065. A heartbeat messeage is sent to all servers after
a user defined timeout. The timeout can be set from the "View|Settings" dialog.
Default is 10 mins. This only adds the on-idle-heartbeat.

/cvsroot/ParaView3/ParaView3/Qt/Core/pqServer.h,v <-- pqServer.h
new revision: 1.15; previous revision: 1.14
/cvsroot/ParaView3/ParaView3/Qt/Core/pqServer.cxx,v <-- pqServer.cxx
new revision: 1.23; previous revision: 1.22
/cvsroot/ParaView3/ParaView3/Qt/Components/Resources/UI/pqApplicationOptions.ui,v <-- pqApplicationOptions.ui
new revision: 1.2; previous revision: 1.1
/cvsroot/ParaView3/ParaView3/Qt/Components/pqApplicationOptions.cxx,v <-- pqApplicationOptions.cxx
new revision: 1.2; previous revision: 1.1
(0009841)
Utkarsh Ayachit (administrator)
2007-12-06 08:51

Fixed by adding a heartbeat message sent to server periodically.
(0009843)
Alan Scott (manager)
2007-12-06 15:40

As per Dave K, this is fixed.

 Issue History
Date Modified Username Field Change
2007-11-19 18:25 David Karelitz New Issue
2007-11-20 11:46 David Karelitz Note Added: 0009709
2007-11-20 15:51 David Karelitz Note Added: 0009710
2007-11-20 18:35 David Karelitz Note Added: 0009711
2007-11-21 12:21 David Karelitz Note Added: 0009721
2007-11-27 11:37 Ken Moreland Note Added: 0009762
2007-11-27 11:43 Ken Moreland Note Added: 0009763
2007-11-27 13:37 Utkarsh Ayachit Note Added: 0009764
2007-11-27 13:37 Utkarsh Ayachit Status backlog => tabled
2007-11-27 13:37 Utkarsh Ayachit Assigned To => Utkarsh Ayachit
2007-12-06 08:51 Utkarsh Ayachit Status tabled => @80@
2007-12-06 08:51 Utkarsh Ayachit Resolution open => fixed
2007-12-06 08:51 Utkarsh Ayachit Note Added: 0009841
2007-12-06 15:40 Alan Scott Status @80@ => closed
2007-12-06 15:40 Alan Scott Note Added: 0009843
2008-03-07 07:56 Berk Geveci Target Version => MARS
2011-06-16 13:10 Zack Galbreath Category => (No Category)


Copyright © 2000 - 2018 MantisBT Team