Page 1 of 1

How can the best output and storage of large amounts of data

Posted: Sun Jun 28, 2020 3:16 am
by Andy
My project creat a struct,As follows:
struct rawdatdata {
uint8_t record[24];
uint8_t data[469];
};
I try to save into txt file,then some strange symbols and blank appear in the file.Why????????
" TU4SECU4査?BD#[email protected]?0?"
Worse more,The data is too much to result to the txt file is stuck.
I use wxFile class and the function Write。

wxFile m_file("H:\\rdata.txt", wxFile::write_append);
m_file.Write(rawdatdata->data, sizeof(rawdatdata->data));

Any idea or suggestion? plz! Thank you :D

Re: How can the best output and storage of large amounts of data

Posted: Sun Jun 28, 2020 5:14 am
by ONEEYEMAN
Hi,
Couple of standard questions:
1. wx version?
2. OS version?
3. Toolkit used and version?

And more:
4. How do you populate the variable of that structure? Please post some code?
5. How do you check the content of the file?

Thank you.

Re: How can the best output and storage of large amounts of data

Posted: Sun Jun 28, 2020 6:26 am
by PB
I would say: Do not write binary data to a text file which is for text strings. Use a binary file to store binary data. If you still want to store binary data as if they were strings, use wxBase64{De|En}code.

See here for an example of writing binary data using wxWidgets: viewtopic.php?f=1&t=47221#p199131

BTW, I would be vary of writing the entire structs because of padding, unless you explicitly tell the compile to pack the struct. I would also consider using wxFFile instead of wxFile, should be faster.

Re: How can the best output and storage of large amounts of data

Posted: Tue Jun 30, 2020 10:02 am
by Nunki
Hi Andy,

Working with data in your application, you should always ask yourself what am I going to do with the data besides writing it to a file. Are you going to read it, alter it, search in it. If so you should do the little overhead now to introduce storage into an SQL database. So that for future enhancements you may have to do less work than with the approach your choosing now. Besides using SQL in your application simplifies a lot. Especially problems like you are facing now. On top of that, SQL databases are tuned to give optimal performance. Allows you to even put the data on a separate machine (server).

Just a thought,
Nunki