Unicode support in component 'databaselayer'? Topic is solved

Talk here about issues with one of the components hosted at wxCode, or suggest features for it.
Post Reply
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Unicode support in component 'databaselayer'?

Post by utelle »

I would like to use the component databaselayer as a database abstraction, but I need to build my project in Unicode mode. Unfortunately databaselayer currently does not seem to support the wxWidgets Unicode build. For example opening a SQLite database is done by calling

Code: Select all

sqlite3_open(strDatabase.c_str(), &m_pDatabase);
This only works in ANSI build. In Unicode build c_str() returns a const wchar_t* string, but sqlite3_open expects const char*. Additionally in Unicode build it would be necessary to convert strings to UTF-8 before passing them to SQLite (or any of the other of the supported databases).

Is it planned to support Unicode build in a future version of databaselayer? If yes, when one may expect this to be the case?

Regards,

Ulrich
jb_coder
Super wx Problem Solver
Super wx Problem Solver
Posts: 267
Joined: Mon Oct 18, 2004 10:55 am

Post by jb_coder »

I'll see what I can do to get databaselayer Unicode compatible. Unfortunately, I can't give you a time frame since I haven't worked with wxWidgets compiled for Unicode before.

Sorry for the delayed response, but I haven't had much spare time lately to keep up with the Forum.
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Post by utelle »

jb_coder wrote:I'll see what I can do to get databaselayer Unicode compatible. Unfortunately, I can't give you a time frame since I haven't worked with wxWidgets compiled for Unicode before.
Thanks for your reply. Since Unicode builds are my primary target maybe I'm able to assist in this regard. As soon as my limited time resources allow I'll take a closer look at the databaselayer code.

Regards,

Ulrich
jb_coder
Super wx Problem Solver
Super wx Problem Solver
Posts: 267
Joined: Mon Oct 18, 2004 10:55 am

Post by jb_coder »

I've got the SQLite code working with Unicode builds in CVS. I ended up using the ConvertToUTF8 and ConvertFromUTF8 functions from the wxWidgets book. I think that I messed up something somewhere (maybe SqliteDatabaseLayer::PrepareStatement) because now the unit test fails when attempting to store a string as a blob.

I'll start working on the rest of the database backends. Please let me know if the methods that I'm using aren't comprehensive enough for regular Unicode usage.

Also, I'm wondering about switching the PreparedStatement::SetParamBlob from using an unsigned char* parameter to using a void* parameter. Would anyone object to that?
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Post by utelle »

jb_coder wrote:I've got the SQLite code working with Unicode builds in CVS.
Great!
jb_coder wrote:I ended up using the ConvertToUTF8 and ConvertFromUTF8 functions from the wxWidgets book.
When I implemented my component wxSQLite3 the wxWidgets book wasn't available yet. I wish I had known those methods then. Maybe I change my conversion code in the next release since the methods from the wxWidgets book seem to be simpler.
jb_coder wrote:I think that I messed up something somewhere (maybe SqliteDatabaseLayer::PrepareStatement) because now the unit test fails when attempting to store a string as a blob.
In case of a blob you shouldn't convert to UTF8. If the string contains non-ASCII characters the length of the UTF8 string is greater than that of the original string. Ok, this is not the case in your test code, but you add 1 to the length. Since you have no longer a C string as before introducing UTF8 conversion you don't have the terminating null character in the converted string but probably a random character. This may cause an invalid UTF8 string which is not converted at all when you read back from the database. But even if the UTF8 string is valid your result will be one character longer (since you added 1) causing the comparison to fail.
jb_coder wrote:I'll start working on the rest of the database backends. Please let me know if the methods that I'm using aren't comprehensive enough for regular Unicode usage.
Except for the blob case the code seems to be ok.
jb_coder wrote:Also, I'm wondering about switching the PreparedStatement::SetParamBlob from using an unsigned char* parameter to using a void* parameter. Would anyone object to that?
For me it would be ok to use void* parameter.

Regards,

Ulrich
jb_coder
Super wx Problem Solver
Super wx Problem Solver
Posts: 267
Joined: Mon Oct 18, 2004 10:55 am

Post by jb_coder »

Thanks for the feedback.

All four database backends now compile and pass the unit tests under Unicode builds now. One funny thing that I found was that I had to fix a memory leak under MySQL to get the Unicode build to pass the unit tests, but that memory leak fix caused a segmentation fault under the non-Unicode build so I had to wrap it in a "#if wxUSE_UNICODE" block.

I'll try to get a wxCode release out by the end of this weekend with the Unicode changes in place.
utelle
Moderator
Moderator
Posts: 1125
Joined: Tue Jul 05, 2005 10:00 pm
Location: Cologne, Germany
Contact:

Post by utelle »

jb_coder wrote:All four database backends now compile and pass the unit tests under Unicode builds now.
Ah, good news! :D

In which state is the Oracle backend? Already usable?
jb_coder wrote:One funny thing that I found was that I had to fix a memory leak under MySQL to get the Unicode build to pass the unit tests, but that memory leak fix caused a segmentation fault under the non-Unicode build so I had to wrap it in a "#if wxUSE_UNICODE" block.
Strange. I'll take a look at the code later, maybe I have an idea what's causing the problem.

Regards,

Ulrich
jb_coder
Super wx Problem Solver
Super wx Problem Solver
Posts: 267
Joined: Mon Oct 18, 2004 10:55 am

Post by jb_coder »

I just checked in some more Oracle code to try to fill in the actual database connection functionality. A lot of the details seem to be specified in the connection string parameter of the occi::Environment::createConnection function. Hopefully this page (http://media.datadirect.com/download/do ... l#wp947701) is correct in listing the possible parameters though. I need to learn a little more abor Oracle database administration before being able to run the unit tests though. It probably doesn't compile in Unicode mode, because I want to at least make sure that it passes the unit tests in a non-Unicode build before I start making any more changes.
Post Reply