Is there any way to speed up loading grids in wxpython??? Topic is solved

This forum can be used to talk about general design strategies, new ideas and questions in general related to wxWidgets. If you feel your questions doesn't fit anywhere, put it here.
Post Reply
User avatar
pikkip
Knows some wx things
Knows some wx things
Posts: 37
Joined: Mon Sep 26, 2016 6:08 am

Is there any way to speed up loading grids in wxpython???

Post by pikkip » Fri Oct 21, 2016 5:06 am

I have a text file of about 2,50,000 lines. I have to process these lines and insert them to a database. It is then retrieved from there to be displayed on a grid(wx widget). I am using Python for this. It took around one hour and thirty minutes for this. Could anyone suggest a way by which this time can be reduced??? :?

User avatar
doublemax
Moderator
Moderator
Posts: 13992
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Is there any way to speed up loading grids in wxpython???

Post by doublemax » Fri Oct 21, 2016 6:38 am

There a three different issues to consider:

1) Parsing the text file: Usually there is not much you can do wrong here, but it depends on which class you're using and how exactly you do it. You could try using a wxBufferedInputStream with a big buffer size, but usually the OS does a proper file caching itself. Or you could use a wxTextFile which will load the whole text into memory.

2) Inserting data into the database: Of course if the textfile doesn't change, you shouldn't do this process every time. Also, if only a few lines change, try to just update these lines and not the whole database. Also make sure to wrap the whole process in a SQL transaction which will speed up things a lot. Sometimes it even helps to drop all database indices before inserting data and then recreate them afterwards.

3) Displaying the data in wxGrid: If you already have a database that contains all data, you shouldn't insert all data into the wxGrid. Instead derive your own class from wxGridTableBase that reads the data directly from the database. If that's too much effort and you still want to insert the data into wxGrid, make sure to wrap the import process with wxGrid::BeginBatch/EndBatch.
Use the source, Luke!

User avatar
pikkip
Knows some wx things
Knows some wx things
Posts: 37
Joined: Mon Sep 26, 2016 6:08 am

Re: Is there any way to speed up loading grids in wxpython???

Post by pikkip » Fri Oct 21, 2016 7:15 am

Thank you so much :D

ONEEYEMAN
Part Of The Furniture
Part Of The Furniture
Posts: 3409
Joined: Sat Apr 16, 2005 7:22 am
Location: USA, Ukraine

Re: Is there any way to speed up loading grids in wxpython???

Post by ONEEYEMAN » Fri Oct 21, 2016 1:47 pm

Hi,
Just a suggestion:

You could try to write a Perl script that will read and parse the text file and save the data somewhere.
Then you wxPython script can read that data and put it in the wx.Grid().

Perl is highly optimized to process text files so you could save some time here as well.

Thank you.

User avatar
pikkip
Knows some wx things
Knows some wx things
Posts: 37
Joined: Mon Sep 26, 2016 6:08 am

Re: Is there any way to speed up loading grids in wxpython???

Post by pikkip » Mon Oct 24, 2016 5:36 am

Thank you for the suggestion!!! :D

Post Reply