Reading large files on Windows

If you are using the main C++ distribution of wxWidgets, Feel free to ask any question related to wxWidgets development here. This means questions regarding to C++ and wxWidgets, not compile problems.
Post Reply
er2joe
In need of some credit
In need of some credit
Posts: 9
Joined: Sun Mar 03, 2013 5:55 pm

Reading large files on Windows

Post by er2joe » Sat Aug 29, 2020 12:05 am

I'm trying to recover data from an SD card that got the FAT corrupted. I am able to open the raw drive using wxFile using the syntax, "\\\\.\\H:" as I found in a Microsoft post.

However, The SD card is 16GB, and wxFile::Read fails when I try to read 16GB (I have 48GB RAM, so that's not an issue; I have already created the 16GB vector by this time).

The assertion failure is, "buffer_size <= INT_MAX." Reading the documentation, it explicitly states, "If the platform supports it, wxFileOffset is a typedef for a native 64 bit integer, otherwise a 32 bit integer is used for wxFileOffset." I'm compiling 64-bit (as evidenced by the successful creation of a 16GB vector), but this assertion is failing. Should I just comment out the assertion?

I could read less, but I want to ensure I don't cut off a file prematurely when I hit the end of the data that was read. I didn't see an easy way to read chunks and add to the vector afterwards. I guess I could read into one vector and append to another post-read, but it would be simplest if I could just read 16GB.

Joe

User avatar
doublemax
Moderator
Moderator
Posts: 15283
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: Reading large files on Windows

Post by doublemax » Sat Aug 29, 2020 6:33 am

What's the exact assert message, because i can't find anything like "buffer_size <= INT_MAX" in the sources?

I don't see any problem reading in chunks, but i wouldn't use a vector, just a plain buffer allocated with malloc.

You could also try wxFFile instead of wxFile, it's supposed to be more low-level.
Use the source, Luke!

PB
Part Of The Furniture
Part Of The Furniture
Posts: 2675
Joined: Sun Jan 03, 2010 5:45 pm

Re: Reading large files on Windows

Post by PB » Sat Aug 29, 2020 8:26 am

wxFile is just a wrapper around CRT file functions and on MSW, unlike size_t, int is 32-bit even in 64-bit builds. wxFile uses wxRead() which I think with MSVC translates to _read() and the assert comes from there.

FWIW, wxFFile uses fread() which may not have the problem.

TBH, I find attempting to read several GBs at once into memory instead of reading in reasonably-sized chunks rather odd. But I guess your scenario somehow requires it?

Similarly to doublemax, I do not understand why you need a vector for this. You should just allocate a char* with new[] and read into it. This may be one of the few actual uses for a raw pointer in modern C++.

Post Reply