help wxTextCtrl with very large files

If you are using the main C++ distribution of wxWidgets, Feel free to ask any question related to wxWidgets development here. This means questions regarding to C++ and wxWidgets, not compile problems.
Post Reply
jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

help wxTextCtrl with very large files

Post by jsheets » Wed Dec 21, 2005 5:04 pm

Hello everyone,

We are working on a project that translates very large log files into more human readable form, these log files range from 15 MB to over 1 GB depending on who created it and what information is logged.

We have two wxTextCtrls, one that has raw log file output and another that has translated text. We want to allow the user to view the full untranslated log file by scrolling through the raw log wxTextCtrl but to conserve memory we also want to limit the currently displayed text to about 50 lines.

We have been unable to implement this in such a way that does not severely impact performance (currently every time a line is added to the control we check to see if there is 50 lines, if so we drop the first line and add the new content).

The last bit of information is the raw log file can be a live file, meaning we are parsing it at the same time information is being added to it.

I'd sincerely appreciate any help or suggestions.

Jason

benedicte
wxWorld Domination!
wxWorld Domination!
Posts: 1409
Joined: Wed Jan 19, 2005 3:44 pm
Location: Paris, France

Post by benedicte » Wed Dec 21, 2005 5:07 pm

Maybe you should use a widget more like a "text editor", such as wxStyledTextControl (stc in contrib folder), or wxRichTextEditor (http://forums.wxwidgets.org/viewtopic.php?t=4663).

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets » Wed Dec 21, 2005 5:14 pm

Thanks for the reply,

I'm open to using a different editor, such as wxStyledTextControl or wxScintilla but unless I've missed something in the documentation (which is entirely possible) I don't see anything in those editors that would enable it to handle the larger file differently than wxTextCtrl.

I'm assuming I'll need to somethimg similar to attaching the control to the file stream and adjust it on a scroll event?

benedicte
wxWorld Domination!
wxWorld Domination!
Posts: 1409
Joined: Wed Jan 19, 2005 3:44 pm
Location: Paris, France

Post by benedicte » Wed Dec 21, 2005 5:22 pm

I've heard that system text control widgets may not handle large buffers... (you can probably do a search with wxTextCtrl here and in the wxWidgets newsgroup).
As most wxWidgets controls are just a layer on system-dependant objects, you may face troubles with that point.

Text editors are able to load larger buffers, although I don't know if they have a limit, too.

jsheets
Earned a small fee
Earned a small fee
Posts: 22
Joined: Wed Oct 19, 2005 1:21 pm

Post by jsheets » Wed Dec 21, 2005 5:38 pm

Unfortunately from what I've read even Scintilla is designed only for reasonably small files (in the hundreds of KB) and scales to megabytes but doesn't scale well into hundreds of MB.

I'm not so much wanting to load the entire file into the control at one time as much as make it appear that I have (much like vi and vim does) so memory usage is constant at say the amount of memory required to load 50 lines into the control.

travfar
Earned some good credits
Earned some good credits
Posts: 132
Joined: Mon Apr 04, 2005 12:54 am
Location: California
Contact:

Post by travfar » Wed Dec 21, 2005 9:52 pm

Hi!

Do the users have to edit the large file?

If they don't, there really is no need for a wxTextCtrl. You could use a wxDC and draw the text onto a wxClientDC which is inside a wxScrolledWindow.
I see said the blind man to the deaf girl who was mute.

Visit my project at: http://vwp.sourceforge.net

wx version:2.6.2 and 2.6.3 on linux
OS's:Windows XP, Gentoo is the best!!!!!!
compiler:Codeblocks(MingW32) and gcc on linux

Post Reply