MantisBT - ParaView
View Issue Details
0014729ParaView(No Category)public2014-05-14 10:422015-01-11 08:18
Vadim SANDLER 
Joachim Pouderoux 
normalminoralways
closedfixed 
4.1 
4.2 
ParaViS
Fix_utf8_issue_for_StringVectorPropertyWidget
incorrect functionality
0014729: It is impossible use Unicode symbols in the "Text" source
It is impossible use Unicode symbols in the "Text" source from the python or directly from the GUI:

Example 1:

Text1 = Text()
Text1.Text = 'Diff\xe9rence'
RenderView1 = GetRenderView()
DataRepresentation1 = Show()
Render()

Example 2:

Menu Source -> Text,
Insert the "Différence" string in the 'Text' area
Click on Apply

In the result appears "Output Message" window with the following log:

Generic Warning: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Common/Core/vtkUnicodeString.cxx, line 183
vtkUnicodeString::from_utf8(): not a valid UTF-8 string.


Generic Warning: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Common/Core/vtkUnicodeString.cxx, line 183
vtkUnicodeString::from_utf8(): not a valid UTF-8 string.


Generic Warning: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Common/Core/vtkUnicodeString.cxx, line 183
vtkUnicodeString::from_utf8(): not a valid UTF-8 string.


Generic Warning: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Common/Core/vtkUnicodeString.cxx, line 183
vtkUnicodeString::from_utf8(): not a valid UTF-8 string.


ERROR: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Rendering/FreeType/vtkTextActor.cxx, line 803
vtkTextActor (0x36e3d80): Cannot compute bounding box.


Generic Warning: In /misc/dn20/salome/rnv/PRODUCTS_BUILD/ParaView-4.1.0_SRC/VTK/Common/Core/vtkUnicodeString.cxx, line 183
vtkUnicodeString::from_utf8(): not a valid UTF-8 string.
No tags attached.
png paraview.png (75,428) 2014-06-18 04:42
https://www.vtk.org/Bug/file/9665/paraview.png
png

png salome.png (125,209) 2014-06-18 04:43
https://www.vtk.org/Bug/file/9666/salome.png
png
Issue History
2014-05-14 10:42Vadim SANDLERNew Issue
2014-06-05 09:38Joachim PouderouxAssigned To => Joachim Pouderoux
2014-06-05 10:26Joachim PouderouxNote Added: 0032748
2014-06-05 10:27Joachim PouderouxResolutionopen => fixed
2014-06-06 04:32Joachim PouderouxNote Added: 0032757
2014-06-18 04:42Vadim SANDLERFile Added: paraview.png
2014-06-18 04:43Vadim SANDLERFile Added: salome.png
2014-06-18 04:44Vadim SANDLERNote Added: 0032891
2014-06-20 06:06Joachim PouderouxNote Added: 0032907
2014-06-20 06:06Joachim PouderouxTopic Name => Fix_utf8_issue_for_StringVectorPropertyWidget
2014-06-20 06:06Joachim PouderouxReproducibilityhave not tried => always
2014-06-20 06:06Joachim PouderouxStatusbacklog => active development
2014-06-20 06:24Joachim PouderouxNote Edited: 0032907bug_revision_view_page.php?bugnote_id=32907#r773
2014-06-20 07:52Joachim PouderouxNote Edited: 0032907bug_revision_view_page.php?bugnote_id=32907#r774
2014-06-25 11:22Utkarsh AyachitNote Added: 0032916
2014-06-25 11:22Utkarsh AyachitStatusactive development => gatekeeper review
2014-06-25 11:22Utkarsh AyachitFixed in Version => git-next
2014-06-25 11:22Utkarsh AyachitNote Added: 0032917
2014-06-27 08:41Utkarsh AyachitFixed in Versiongit-next => git-master
2014-06-27 08:41Utkarsh AyachitNote Added: 0032938
2014-06-27 09:50Utkarsh AyachitStatusgatekeeper review => customer review
2014-07-08 15:33Alan ScottNote Added: 0032989
2014-07-08 15:33Alan ScottStatuscustomer review => closed
2014-08-29 10:59Utkarsh AyachitFixed in Versiongit-master => 4.2
2015-01-11 08:17Utkarsh AyachitSource_changeset_attached => ParaView master 3ccbba88
2015-01-11 08:18Utkarsh AyachitSource_changeset_attached => ParaView master 4a1d88c5

Notes
(0032748)
Joachim Pouderoux   
2014-06-05 10:26   
Hello,

You can use UTF8 strings using the Python wrapping but you have to use UTF8 hex codes for the accentued chars instead. Example:

  Text1 = Text()
  Text1.Text = 'Diff\xC3\xA9rence'
  RenderView1 = GetRenderView()
  DataRepresentation1 = Show()
  Render()

See for instance this page http://www.utf8-chartable.de [^] The third column shows the hex code you should use as escape sequence.

However, because of the Qt layer, this does not work in the GUI (but you can copy/paste the string with special chars that will be generated from the Python source).
(0032757)
Joachim Pouderoux   
2014-06-06 04:32   
In the UI, you can direcly use the UTF8 code using something like ALT+[code in decimal].
Exemple: Diff[ALT 0195][ALT 0169]rence
(0032891)
Vadim SANDLER   
2014-06-18 04:44   
Dear Joachim,

Thank you for explanations.

Indeed, it seems that passing 'Diff\xe9rence' to Python might be not correct, if we want Unicode strings to be shown.
It seems to be more correct to pass "Différence" or "Diff\xC3\xA9rence", and naturally "Différence" is more preferable way.

However, there are still problems with processing Unicode strings in ParaView. I think, when you say "because of the Qt layer, this does not work in the GUI" - it is not quite correct.

Indeed, as all application components like Qt, Python and VTK declare support of Unicode strings, there potentially should not be problems with interchanging data between these components, provided that all is implemented in a coherent way.

For example, please take a look at the snapshot from SALOME application (salome.png). In SALOME we pass data between different application components, including:
- From Qt widget (GUI side) to server process via CORBA bus : here we use QString::toUtf8() to pass data as CORBA::String (i.e. char*) [case 1 on the image]
- From server process back to Qt widget (GUI side) via CORBA bus : here we use QString::fromUtf8() to ensure data is properly converted from char* and shown in GUI [cases 2a and 2b on the image]
- From Qt widget to embedded Python interpreter : via QString::toUtf8() [case 3 on the image]
- From embedded Python interpreter back to GUI (Python's feedback) : via QString::fromUtf8() [case 4 on the image]
- From Qt widget to VTK presentation : QString::toUtf8() [case 5 on the image]

As demonstrated by this example, in all cases unicode string data is shown properly. It is OK when we use either "Différence" or "Diff\xC3\xA9rence" in a Python script.

However, in ParaView it's no OK - with both "Différence" or "Diff\xC3\xA9rence") used in a Python script we see good text in VTK viewer but bad one in Qt widget (see paraview.png). It seems that there is missing QString::fromUtf8() somewhere.
An attempt to pass 'Différence' via "Properties" pane leads to a "vtkUnicodeString::from_utf8(): not a valid UTF-8 string" error. Missing QString::toUtf8()?

Also, it's necessary to mention that the processing of a text in the viewer seems to depend on the way we do this.
For instance, in SALOME, if we use vtkTextActor for displaying text label in a viewer, unicode text is shown correctly (as demonstrated by salome.png).
However, if we use vtkActor2D with vtkTextMapper for the same purpose, the result is bad.
Here is the corresponding code:

  // This is OK...
  myTextActor=vtkTextActor::New();
  myTextActor->SetInput(qtstring.toUtf8().constData());

  // This is KO...
  myTextMapper=vtkTextMapper::New();
  myTextActor=vtkActor2D::New();
  myTextActor->SetMapper(myTextMapper);
  myTextMapper->SetInput(qtstring.toUtf8().constData());

Regards,
Vadim.
(0032907)
Joachim Pouderoux   
2014-06-20 06:06   
(edited on: 2014-06-20 07:52)
Thanks for your detailed answer. I have just pushed a fix for this issue but so far it just fixes the Qt=>Proxy problem. I still need to fix Proxy=>Qt.


http://review.source.kitware.com/#/c/15866/ [^]

Change Ia527552c: Fix pqStringVectorPropertyWidget to support utf8 strings.

The problem is described in ticket 0014729.
UTF8 chars were not correctly converted (toLatin1()) was used
so TextSource for instance were not able to support accentuated
chars.

Change-Id: Ia527552c230351364541ddff2e9a527784bed976

(0032916)
Utkarsh Ayachit   
2014-06-25 11:22   
commit 99501ea43b10463f9fcfe251bc9e05fb35acb5c7
Author: Joachim Pouderoux <joachim.pouderoux@kitware.com>
Date: Fri Jun 20 12:01:32 2014 +0200

    Fix pqStringVectorPropertyWidget to support utf8 strings.
    
    The problem is described in ticket 0014729.
    UTF8 chars were not correctly converted (toLatin1()) was used
    so TextSource for instance were not able to support accentuated
    chars.
    
    Change-Id: Ia527552c230351364541ddff2e9a527784bed976
(0032917)
Utkarsh Ayachit   
2014-06-25 11:22   
This above fix should now address the Proxy==>Qt issue as well.
(0032938)
Utkarsh Ayachit   
2014-06-27 08:41   
SUMMARY
---------------------------------------------
Topics merged into master:
        14790_save_restore_settings_buttons
        14814_fix_cell_point_label_format
        14824_fix_reader_factory
        fix-pv-transposetable-filter
        Fix_utf8_issue_for_StringVectorPropertyWidget
        stage-push-data


---------------------------------------------
Topics reverted from next:
  14784_PythonCatalyst_better_IO_2
(0032989)
Alan Scott   
2014-07-08 15:33   
I believe this is fixed. I tested using sources/ text, and the example of Différence. Well written tests.

Tested Linux, master, local server.