Redirecting UTF-8 strings to wxTextCtrl

If you are using the main C++ distribution of wxWidgets, Feel free to ask any question related to wxWidgets development here. This means questions regarding to C++ and wxWidgets, not compile problems.
Post Reply
Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 2:05 am

In one of my projects I need to display UTF-8 string.
Some of the sample code I have available uses std::cout to do this to a console app.

Code: Select all

cout << std::setfill(' ') << std::left << std::setw(44) << md.key() << " " << std::setw(9)
                    << std::setfill(' ') << std::left << md.typeName() << " " << std::dec << std::setw(3)
                    << std::setfill(' ') << std::right << md.count() << "  " << std::dec << md.value() << std::endl;

To avoid as many transcription/translation errors as possible, I wanted to display the result on wxTextCtrl using redirection

I did find the topic discussed here:
viewtopic.php?f=1&t=46867&p=197057
but implementing the proposed class wxStreamToTextRedirectorUnicode does not seem to do the job for me.

All output, even plain text strings, just disappears.
So far I have not been able to trace through the code to track down the cause in an effort to understand and hopefully find a way to resolve the problem.
Obviously I am missing something critical, but I have no idea what.
TIA
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 4823
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by ONEEYEMAN » Sun Nov 29, 2020 5:56 am

Hi,
What code do you use to send the string to the wxTextCtrl?
Are you sure it is a UTF8 encoded one?

Thank you.

Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 5:41 pm

ONEEYEMAN wrote:
Sun Nov 29, 2020 5:56 am
Hi,
What code do you use to send the string to the wxTextCtrl?
The original code I was using, and which works well for plain ASCII strings, is:

Code: Select all

    // has problems with UTF-8 strings
      ostream stream(m_textCtrlXmpDump );
      for (const auto& md : xmpData)
      {
        //stream << 123.456 << " some text\n"; // test string - plain ASCII
        stream << std::setfill(' ') << std::left << std::setw(44) << md.key() << " " << std::setw(9)
                    << std::setfill(' ') << std::left << md.typeName() << " " << std::dec << std::setw(3)
                    << std::setfill(' ') << std::right << md.count() << "  " << std::dec << md.value() << std::endl;
      }
      stream.flush();
If I rework the code to use all wxString::Printf() code and convert all strings from UTF-8 to wxString, the display of even UTF-8 strings is correct and as expected.
However, that is a whole lot of work in this instance and, in general, if there is a way I would very much prefer to use the code as it was to avoid any gotchas due to the rewrite.
ONEEYEMAN wrote:
Sun Nov 29, 2020 5:56 am
Are you sure it is a UTF8 encoded one?
Yes very much so.
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 4823
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by ONEEYEMAN » Sun Nov 29, 2020 6:16 pm

Hi,
Why?
What is wrong with calling FromUTF8()?

Thank you.

Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 6:41 pm

ONEEYEMAN wrote:
Sun Nov 29, 2020 6:16 pm
What is wrong with calling FromUTF8()?
Nothing, but it requires a rewrite.
Some members of 'md' can be strings, numeric or boolean data and hence would need logic to deal with each type differently.
With the code as is, the stream handles all of those differences, when it goes to cout.
A rewrite might not cover all cases properly, especialy as the contents of md originate in an imported library.

If it can't be done, I'll have to deal with it. The link I posted to a wxWidgets discussion seemed to imply that it might be possible.
Then again, in some respects something important seems to be missing if the code for redirection fails for UTF-8 strings.
I can see, that there might be a need to handle UTF-8 in a special way, similar to wxString::To/FromUtf8.
That is really all I want to find out
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

Kvaz1r
Super wx Problem Solver
Super wx Problem Solver
Posts: 292
Joined: Tue Jun 07, 2016 1:07 pm

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Kvaz1r » Sun Nov 29, 2020 8:32 pm

Widgets wrote:
Sun Nov 29, 2020 2:05 am
but implementing the proposed class wxStreamToTextRedirectorUnicode does not seem to do the job for me.
Did you try to add converting that was mentioned in this message?

Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 9:15 pm

Kvaz1r wrote:
Sun Nov 29, 2020 8:32 pm
Widgets wrote:
Sun Nov 29, 2020 2:05 am
but implementing the proposed class wxStreamToTextRedirectorUnicode does not seem to do the job for me.
Did you try to add converting that was mentioned in this message?
That is the very same post I had referred to in my first post - and no, it did not.
In fact, class wxStreamToTextRedirectorUnicode was part of that very post.
Even the 'solution' referred to in the last message of that thread, did not give an actual solution using a redirected cout, but seems to refer to a solution using string conversion, without redirection.
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

User avatar
doublemax
Moderator
Moderator
Posts: 15651
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by doublemax » Sun Nov 29, 2020 9:28 pm

You need something that converts the 8bit UTF8 byte stream into a stream of (decoded) wide characters. The wxTextCtrl will not do it by itself.

Googling found this (very old) post (untested): https://stackoverflow.com/questions/148 ... ion-in-stl

https://en.cppreference.com/w/cpp/locale/codecvt_utf8
Use the source, Luke!

Kvaz1r
Super wx Problem Solver
Super wx Problem Solver
Posts: 292
Joined: Tue Jun 07, 2016 1:07 pm

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Kvaz1r » Sun Nov 29, 2020 9:37 pm

Widgets wrote:
Sun Nov 29, 2020 9:15 pm
That is the very same post I had referred to in my first post - and no, it did not.
In fact, class wxStreamToTextRedirectorUnicode was part of that very post.
Even the 'solution' referred to in the last message of that thread, did not give an actual solution using a redirected cout, but seems to refer to a solution using string conversion, without redirection.
I understood it, because you left link in first message of the topic and I can read. But Windows console hasn't support of UTF-8, so you still must use conversion if you want get UTF-8. That is exactly what written in last message of the that thread.

Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 9:49 pm

Yes, there seems to be a lot more involved with this conversion than it seemed at first.
I will have to investigate the links you provided and hopefully will be able to come up with something useful.
As always, it will likely take several tries and a fair bit of time.
For now, I think this will be put on the back burner, as it seems to involve more time than is available right now.

FWIW, meanwhile I have found some other 'solution' (read 'work-around) for windows sending UTF-8 to the DOS console in the following blog:
https://blogs.msmvps.com/gdicanio/2019/ ... mple-code/
In this case the blogger converts UTF-8 to UTF-16 - which, in a way, makes sense for Windows' MBCS/UTF-16 mentality.
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

User avatar
doublemax
Moderator
Moderator
Posts: 15651
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by doublemax » Sun Nov 29, 2020 10:35 pm

Do you actually have to output to the console? Are you developing a console application?
Use the source, Luke!

Widgets
Super wx Problem Solver
Super wx Problem Solver
Posts: 397
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: Redirecting UTF-8 strings to wxTextCtrl

Post by Widgets » Sun Nov 29, 2020 10:48 pm

No, the original sample code is/was a console app. My output needs to go to a wxWidgets GUI.
Being able to use redirection to a text control was the easy way out as long as everything was plain ASCII.
When UTF-8 strings showed up and became important, that option failed badly.
Finding a way to make the redirection handle UTF-8 or other encoding would be much simpler with fewer chances for misinterpretation of the types of data coming from within a library, than reworking & testing all the equivalent code.

But, since my last post here, I have followed some of the links you provided and concluded that, at least for the time being, there is way too much research and testing involved to try and NOT translate the cout code to something that will work 'well enough' for now.
Environment: Win 10 64-bit & Mint 10.2
MSVC Express 2010, 2015, 2017, 2019
wxWidgets 3.1.2
Fight SPAM - report it with wxSR

Post Reply