UTF8 text downloaded by wxHTTP is displayed wrong

If you are using the main C++ distribution of wxWidgets, Feel free to ask any question related to wxWidgets development here. This means questions regarding to C++ and wxWidgets, not compile problems.
Post Reply
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

Hello,

I establish a connection via wxHTTP to my server.
The program shall download some text...

My PHP-Code and Server-Output is configured to UTF-8 and the text is displayed correct in all browsers.
The text includes some letters like Ä,Ö,Ü,ä,ö,ü,ß,é... wich are displayed correct in my browser as I alredy said.
But if I display this text in my program with wxMessageBox() for example, I get those special letters displayed wrong.

Do you have any Idea how to fix this bug?

Thank you very much!
- eriX

Code: Select all

	wxHTTP get;
	get.SetHeader(_T("Content-type"), _T("text/html; charset=utf-8"));
	get.SetTimeout(10);

	for(int i=0; i<10; i++) 
	{ 
		if(!get.Connect(wxT("www.mydomain.de"))) 
		{ 
			wxSleep(2); 
		}
	} 
	 
	wxApp::IsMainLoopRunning();

	wxInputStream *httpStream = get.GetInputStream(_T("/a-dir/index.php?id="+foo));
	 
	if (get.GetError() == wxPROTO_NOERR)
	{
		// CODE
	}
User avatar
doublemax
Moderator
Moderator
Posts: 19116
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by doublemax »

You didn't post the code where you actually receive the data and how you process it.

If it's utf8 encoded, you must decode it when you create a wxString from it, e.g. with wxString::FromUTF8()
http://docs.wxwidgets.org/stable/wx_wxs ... ngfromutf8
Use the source, Luke!
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

Oops... you're right ;)

Here's the code which is important.
Maybe it is wxString/TextInputStream that causes this problem?

Code: Select all

	wxHTTP get;
	get.SetHeader(_T("Content-type"), _T("text/html; charset=utf-8"));
	get.SetTimeout(10);

	for(int i=0; i<10; i++) 
	{ 
		if(!get.Connect(wxT("www.mydomain.de"))) 
		{ 
			wxSleep(2); 
		}
	} 
	 
	wxApp::IsMainLoopRunning();

	wxInputStream *httpStream = get.GetInputStream(_T("/a-dir/index.php?id="+foo));
	 
	if (get.GetError() == wxPROTO_NOERR)
	{
		wxString res;
		wxStringOutputStream out_stream(&res);
		httpStream->Read(out_stream);
			 
		wxStringInputStream sis(ausgabe); 
		wxTextInputStream tis(sis); 
			  
		wxString line1;
		line1 << tis.ReadLine();
		
		// PROCEEDING WITH LINE1 AND LATER LINE2...
	}
User avatar
doublemax
Moderator
Moderator
Posts: 19116
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by doublemax »

That code looks unnecessarily complicated.

Code: Select all

[...]
if(httpStream!=NULL) {
  wxTextInputStream tis(*httpStream);
  while(!httpStream->Eof()) {
    wxString line=tis.ReadLine();
    wxLogMessage(line);
  }
  delete httpStream;
}
wxTextInputStream uses wxConvUTF8 as default encoding converter, so this should work for you.
http://docs.wxwidgets.org/stable/wx_wxt ... streamctor
Use the source, Luke!
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

I've tested it directly before I go on with programming, but the result is poor. Same Problem as before...

Image

This wrong char encoding should not be with UTF-8.

Code: Select all

		if (get.GetError() == wxPROTO_NOERR)
		{
            if(httpStream!=NULL)
            {
                wxTextInputStream tis(*httpStream);
                while(!httpStream->Eof()) 
                {
                    wxString line=tis.ReadLine();
                    wxMessageBox(line,"",wxOK);
                }
            }
User avatar
doublemax
Moderator
Moderator
Posts: 19116
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by doublemax »

Just to make sure: Are you using a unicode build of wxWidgets?

Can you test this code and check if it displays "ÜÖÄ"?

Code: Select all

unsigned char text[]={ 0xc3, 0x9c, 0xc3, 0x96, 0xc3, 0x84, 0x00 };
wxString s((const char *)text, wxConvUTF8);
wxMessageBox(s);
Do you have a real website that outputs utf8 that can be used for testing?
Use the source, Luke!
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

I tested it but it didn't work.
The letters are displayed wrong.
User avatar
doublemax
Moderator
Moderator
Posts: 19116
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by doublemax »

eriX wrote:I tested it but it didn't work.
The letters are displayed wrong.
Then i believe you're not using a unicode build of wxWidgets.

Which wxWidgets version and compiler/ide are you using?
Use the source, Luke!
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

I use wxDevC++ 7.3.1.3 - Windows 7 x64
WxWidgets 2.8.9

I don't know where to look up unicode... I simply used the Installer from the Website.
User avatar
doublemax
Moderator
Moderator
Posts: 19116
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by doublemax »

I don't know wxDevC++. Usually you enable unicode when you build the wxWidgets libraries.

E.g when using MinGW:
http://wiki.wxwidgets.org/Compiling_wxW ... with_MinGW

Or maybe there's an option in wxDevC++ to select which build to use.
Use the source, Luke!
eriX
Experienced Solver
Experienced Solver
Posts: 84
Joined: Wed Feb 04, 2009 2:08 pm
Location: Germany
Contact:

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by eriX »

In wxDevC++ there is an option to enable unicode-support.
If I enable it, the compiler says: "cannot find -lwxmsw28u".

I didn't try to compile wxWidgets by my own... maybe I'll try it in some days.

If I write wxString test="äöü" and display it, it is displayed correct.
Otherwise I'll replace all those letter to alternative strings like /ae /ou /ue and change it back in the program...
RainRat
I live to help wx-kind
I live to help wx-kind
Posts: 178
Joined: Thu Jan 06, 2011 11:26 pm

Re: UTF8 text downloaded by wxHTTP is displayed wrong

Post by RainRat »

eriX wrote:In wxDevC++ there is an option to enable unicode-support.
If I enable it, the compiler says: "cannot find -lwxmsw28u".
I do not use wxDevC++ either, but you generally need to compile the wxWidgets libraries to use unicode, then you need to set the compiler for your specific program to use unicode, then the linker links the correct libraries in. The "cannot find -lwxmsw28u" means that it is looking for the library file wxmsw28u where the "u" indicates unicode. "wxmsw28" would be the library compiled without unicode support. So, I'm guessing either you don't have the libraries compiled to use unicode, or the linker does not know the location of the file, but like I said, I don't use wxDevC++ so can't advise anymore than this.
Post Reply