How to store a double xplatform? Topic is solved

[Jorg]

Hi guys,

I am busy with a rewrite from my old serializer (SIO) to wxArchive, which will be the counter part of MFC's CArchive. I want to store all values as x-platform as possible meaning binary compatible so that serialized files can be opened on linux, MAC and Windows without binary incompatibility.

For a double, this is not so easy. One solution is convert to a string and store the double, another solution is convert it to two long or int64 values, there is a function that splits up the fraction and integer part.

However, with a splitted double I get back for example:

243.1434546784578

Integer: 243
Fraction: 0.1434546784578

Where I rather have two integers back which I can serialize in a binary compatible way.

The lamest method is:

Save the integer part
TmpInt = (Faction * 1000000000)
Save the fraction part

This is limiting.. The precision of the double can be more precise then this. So whatever method I choose, there is a precision error. The string conversion is still the most precise as you can specify how many digits you would like.

Any ideas anyone?

- Jorgen

[Jamie]

Have a look at src/common/extended.c which is used in src/common/datstrm.cpp. Add these if you want to use them:

extern "C" void ConvertToIeeeExtended(double num, unsigned char *bytes);

extern "C" double ConvertFromIeeeExtended(const unsigned char *bytes);

Google for IEEE 754.

[ssigala]

Hi guys,
I am busy with a rewrite from my old serializer (SIO) to wxArchive, which will be the counter part of MFC's CArchive. I want to store all values as x-platform as possible meaning binary compatible so that serialized files can be opened on linux, MAC and Windows without binary incompatibility.

Have you considered using wxDataInputStream/wxDataOutputStream?

http://www.wxwidgets.org/manuals/2.6.1/ ... nputstream

They already handle doubles

[Jorg]

Thanks for the hint on the IEEE standard! I will have to look at that.

As for the original stream objects, I was not aware of them but I still believe they seriously lack in API and reliability. They probably do not do any type checking of what you read back, if you make a mistake in reading back.

See, wxArchive (and CArchive) handle this differently. I implemented a version check that will give errors if the stream contains a higher version then the software expected, but not if a lower version is present. This will make the serializer accept old streams and save them as newer streams. Also the header check will assure you that no bogus formats will be read back, and give errors on that.

This is a snippet of my code as I used this in many big projects:

Code: Select all

wxArchive a(in_stream, _("Header"), 1001);

// in your serializing routines;

if(a.IsStoring())
{
  a << m_name << m_age << m_txt;
}
else
{
  a >> m_name >> m_age;

  // when version of stream is 1001 and up 
  // we need to read extra 
  if(a.GetVersion() > 1000)
  {
    a >> m_txt;
  }
}

Also, when an error is made in the code, e.g. an int is expected but a string is read back, the stream class returns an error and subsequent calls that might follow like;

Code: Select all

  a >> m_someText >> m_someMore;

Will not have any effect as the internal state is an error, the variables will not be affected..

Also, my class has:

Code: Select all

a.ReadString(str, true, _T("Default string"));

In case of an error, the string will be filled with a default value.

There's more... I have an IsPartial flag that can have the benefit of distinguishing in your code if your object needs a partial restore/save or a full one. For example;

Code: Select all

if(a.IsStoring())
{
    a << m_name << m_age;
    
    // serialize our children
    if(!a.IsPartial())
    {
       a << m_items.Count();
       for(size_t i = 0; i < m_items.Count(); i++)
          m_items[i]->Serialize(a);
    }
}
else
{
    a >> m_name >> m_age;
    
    // serialize our children
    if(!a.IsPartial())
    {
       size_t count = 0; 
       a >> count;
       for(size_t i = 0; i < count; i++)
       {
          MyObj *ptr = new MyObj();
          if(ptr->Serialize(a))
            m_items.Add(ptr);
          else
            delete ptr;
       }
    }
}

And third, when this object is finished I will add a concept that will allow extending the serializing stream. People who serialized before know that once a stream is saved, you need to read it back exactly like that (hence the version and header check I built in). I will use markers that can be used to skip parts of the object, and fill the data with defaults or leave them unchanged. For example:

Code: Select all

a.EnterObject();

if(a.IsStoring())
{
    // save all data
   a << m_age << m_name << m_address;
}
else
{
  a >> m_age >> m_name;

  // the address is added in a later stream. At this point in the stream
  // the marker is reached for the end of the object it is currently
  // serializing. All data being read back actually doesn't do much
  // when the data is not present, because the end of the object is
  // at the end of this code

  a >> m_address; // will be left default
}

a.LeaveObject(); // now we can continue with the next object

This way the serializing has become a lot more appealing. The wxWidgets class I was not aware of, but not to my surprise it is a very poor class, with all problems I mentioned earlier still being present.

This is the only class I am particulary fond of, because througout my project it always proved very useful, robust in both handling data corruption, recovery etc.

I even let a collegue of mine deliberately destroy serialized data and showed him the data would never be read back and let the app crash, because of the safe type checking and reliable return of variables

Thanks all, I will finish this class and offer it to the community. For all who wants to use the original ones, they will miss out all the benefits I just mentioned..

With regards,
- Jorgen

[Jorg]

I hope I wasn't sounding to patronizing (sp?) towards the original classes, I decided to wrap my class logic around the existing streams where needed as they provide a good low level streaming method. The higher level logic will be added to provide MFC porters with a CArchive -> wxArchive conversion plus the reliability of the stream.

- Jorgen