Is there any way to speed up loading grids in wxpython??? Topic is solved
Is there any way to speed up loading grids in wxpython???
I have a text file of about 2,50,000 lines. I have to process these lines and insert them to a database. It is then retrieved from there to be displayed on a grid(wx widget). I am using Python for this. It took around one hour and thirty minutes for this. Could anyone suggest a way by which this time can be reduced???
Re: Is there any way to speed up loading grids in wxpython???
There a three different issues to consider:
1) Parsing the text file: Usually there is not much you can do wrong here, but it depends on which class you're using and how exactly you do it. You could try using a wxBufferedInputStream with a big buffer size, but usually the OS does a proper file caching itself. Or you could use a wxTextFile which will load the whole text into memory.
2) Inserting data into the database: Of course if the textfile doesn't change, you shouldn't do this process every time. Also, if only a few lines change, try to just update these lines and not the whole database. Also make sure to wrap the whole process in a SQL transaction which will speed up things a lot. Sometimes it even helps to drop all database indices before inserting data and then recreate them afterwards.
3) Displaying the data in wxGrid: If you already have a database that contains all data, you shouldn't insert all data into the wxGrid. Instead derive your own class from wxGridTableBase that reads the data directly from the database. If that's too much effort and you still want to insert the data into wxGrid, make sure to wrap the import process with wxGrid::BeginBatch/EndBatch.
1) Parsing the text file: Usually there is not much you can do wrong here, but it depends on which class you're using and how exactly you do it. You could try using a wxBufferedInputStream with a big buffer size, but usually the OS does a proper file caching itself. Or you could use a wxTextFile which will load the whole text into memory.
2) Inserting data into the database: Of course if the textfile doesn't change, you shouldn't do this process every time. Also, if only a few lines change, try to just update these lines and not the whole database. Also make sure to wrap the whole process in a SQL transaction which will speed up things a lot. Sometimes it even helps to drop all database indices before inserting data and then recreate them afterwards.
3) Displaying the data in wxGrid: If you already have a database that contains all data, you shouldn't insert all data into the wxGrid. Instead derive your own class from wxGridTableBase that reads the data directly from the database. If that's too much effort and you still want to insert the data into wxGrid, make sure to wrap the import process with wxGrid::BeginBatch/EndBatch.
Use the source, Luke!
Re: Is there any way to speed up loading grids in wxpython???
Thank you so much 
Re: Is there any way to speed up loading grids in wxpython???
Hi,
Just a suggestion:
You could try to write a Perl script that will read and parse the text file and save the data somewhere.
Then you wxPython script can read that data and put it in the wx.Grid().
Perl is highly optimized to process text files so you could save some time here as well.
Thank you.
Just a suggestion:
You could try to write a Perl script that will read and parse the text file and save the data somewhere.
Then you wxPython script can read that data and put it in the wx.Grid().
Perl is highly optimized to process text files so you could save some time here as well.
Thank you.
Re: Is there any way to speed up loading grids in wxpython???
Thank you for the suggestion!!!