exec example - DoGetFromStream timeout Topic is solved

If you are using the main C++ distribution of wxWidgets, Feel free to ask any question related to wxWidgets development here. This means questions regarding to C++ and wxWidgets, not compile problems.
Post Reply
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

exec example - DoGetFromStream timeout

Post by Widgets »

In my current project, I am using the piped frame code from the exec example ( 3.1.4) to get piped data to & from an external program (Exiftool - ET)

An image (*.jpg) is passed to ET via an input pipe, along with some selected command line options and it returns its output via a piped output stream back to my app. In some ways, ET looks like a memory resident application.

For some cases, the code as taken from the example drops the last partial buffer. But I cannot be sure whether it is related to output length or system load.

The code I am using looks like

Code: Select all

bool MyFrame::DoGetStream( wxArrayString &ar_wasIn, wxInputStream& in )
{
  wxString wsBuffer;
  char buffer[4096 + 1];
  ar_wasIn.clear();
  in.Reset();

  while ( in.CanRead() )
  {
    memset( buffer, '\0', WXSIZEOF(buffer) - 1 );
    buffer[in.Read(buffer, WXSIZEOF(buffer) - 1).LastRead()] = '\0';
    wsBuffer += buffer; 
#if defined( TEST_STREAM )
    ::wxMilliSleep( 5);
#endif
  }
  wxStringTokenizer tokenizer( wsBuffer, "\r\n");
  while ( tokenizer.HasMoreTokens() )
  {
    wxString wsToken = tokenizer.GetNextToken();
    ar_wasIn.Add( wsToken );
  }
  return true;
}
The major difference is that I am using a string tokenizer to split the output into a string array. I have included that part for the sake of completeness, though I have tracked the issue to the in.CanRead loop.

If I include the line ::wxMilliSleep( 5); in the loop, the code works as expected for the cases I have tested.
Without that line, there is at least one image for which the last 0x260 characters (2-bytes wide) out of a total of 0x2260 are lost and are not appended to wsBuffer. IOW, it looks like the CanRead() timeout somewhere during the last few bytes and drops out without appending them.

My problem: How can I ensure that this works as expected in general.
Using a hardwired sleep of 5 milliseconds may work on my current machine with the current normal work load, but it is not a good solution at all.
And, even if I could change the time out value for the input stream, how can I ensure it works for all cases, other than possibly a user controlled wxMilliSleep delay?
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: exec example - DoGetFromStream timeout

Post by doublemax »

Can you try this code for reading the data from the stream and check if it makes any difference?

Code: Select all

#include <wx/mstream.h>  // for wxMemoryOutputStream

wxMemoryOutputStream data();
data.Write( in );
wxStreamBuffer *buffer = data.GetOutputStreamBuffer();
wxString wsBuffer = wxString::FromUTF8( buffer->GetBufferStart), buffer->GetBufferSize());
Use the source, Luke!
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: exec example - DoGetFromStream timeout

Post by Widgets »

Where & how should I insert this code or what part should it replace?
My understanding of these stream is rather limited.
plus, this is Windows :-)

Code: Select all

for   while ( in.CanRead() )
  {
    wxMemoryOutputStream data();
     data.Write( in );
   ....
}
it complains:
1>D:\pkg\wx\MSVC2019\_3.1.4_2019\wxET\wxEtReadWasStream.cpp(78,11): error C2228: left of '.Write' must have class/struct/union
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
PB
Part Of The Furniture
Part Of The Furniture
Posts: 4193
Joined: Sun Jan 03, 2010 5:45 pm

Re: exec example - DoGetFromStream timeout

Post by PB »

Widgets wrote: Thu Oct 28, 2021 5:46 pm it complains:
1>D:\pkg\wx\MSVC2019\_3.1.4_2019\wxET\wxEtReadWasStream.cpp(78,11): error C2228: left of '.Write' must have class/struct/union
I think the parentheses after the declaration of "data" variable do not belong there, it is variable declaration, not a function one.

And I would guess the code was meant to replace your while cycle reading from the stream?
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: exec example - DoGetFromStream timeout

Post by Widgets »

I now have

Code: Select all

//  while ( in.CanRead() )
  {
    wxMemoryOutputStream data;
    data.Write( in );   // <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< hangs -- waiting for data??
    wxStreamBuffer *buffer = data.GetOutputStreamBuffer();
    data.CopyTo( buffer, data.GetSize() );
}
With the 'while', the code simply returns - no data read
without it, it hangs at the first call, waiting for data, though at this time, ET has been asked to return data
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: exec example - DoGetFromStream timeout

Post by doublemax »

Sorry for the error, i wrote that code blindly.

The whole reading code was supposed to look like this:

Code: Select all

  wxMemoryOutputStream data;
  data.Write( in );
  const wxStreamBuffer *buffer = data.GetOutputStreamBuffer();
  wxString wsBuffer = wxString::FromUTF8( (const char *)buffer->GetBufferStart(), buffer->GetBufferSize());
But if it hangs at the Write call, it doesn't make any difference.
Without that line, there is at least one image for which the last 0x260 characters (2-bytes wide) out of a total of 0x2260 are lost and are not appended to wsBuffer.
What do you mean with "2-bytes wide"? The way i see it your code can only work if the input contains 8 bit text.

Anyway, if you can't know in advance how many bytes there will be to read, i have no idea how to reliably detect when the output is finished. Maybe it would be easier to let ET write into a temporary file and then read that.
Use the source, Luke!
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: exec example - DoGetFromStream timeout

Post by Widgets »

Edit: after I had finished composing my response, something prompted me to explore the -stay_open documentation some more. and there seems to be a way to force a terminating output to the stderr stream as well.
I will have to explore that issue next, but I wanted to keep the rest of my comments here for feedback to other users, who might need or want to use the piped frame example.

With the new code, it hangs at the data.Write() as it did before.
doublemax wrote: Thu Oct 28, 2021 8:53 pm
Without that line, there is at least one image for which the last 0x260 characters (2-bytes wide) out of a total of 0x2260 are lost and are not appended to wsBuffer.
What do you mean with "2-bytes wide"? The way i see it your code can only work if the input contains 8 bit text.
All I know the code did work (mostly ;-))
As I understand this, the stream doesn't/should not care what format the data will eventually be cast/transformed into. The output from Exiftool is guaranteed to be UTF8 (though in this specific case it was all plain ASCII - but wide chars).
When I forced ET to output to a text file via command line redirection, I also got a text file of the exact same length, which was my initial test to try and track down the problem.

At one time, I suspected some optimization issue, but optimization was supposedly turned off for the debug version.
To check that possibility. I redirected the stream output to a log text control and it all worked. (the extra processing time seems to have made the difference.
Hence I tried the short delay to give the system a chance to catch up. Since that worked, I suspected that there must be some sort of time-out set for the CanRead() though I don't know where - or how to fiddle its value.
FWIW, I have had to add delays between my issuing the commands and attempting to read the streams much earlier to allow the system to process the requests.
doublemax wrote: Thu Oct 28, 2021 8:53 pm Anyway, if you can't know in advance how many bytes there will be to read, i have no idea how to reliably detect when the output is finished. Maybe it would be easier to let ET write into a temporary file and then read that.
For one specific operation, I have already resorted to that workflow, but using a separate wxExecute process to avoid this very issue.
When I first ported this example to my app, I had been wondering about the very issue, but assumed that the 'magic' behind the CanRead() call would be sufficient.
For the data stream there actually is a solution provided for by ET's stay_open protocol. It will terminate is text output with a special string which starts with '{execute...}
For the stderr stream however, that option is not available.
I have not used the option to wait for the terminating string, mainly because initially it did not seem necessary and secondly, because I was hoping to be able to use the same code for stderr streams as well.

My options seem to be:
1) make things work as is
2) persuade/convince the author of ET to add a similar terminator for the sterr stream.

Not at all sure which would be easier. :-)
Then again, I wanted to make sure that I understood the piped frame process example well enough and did not make fundamental mistakes in porting it to my project
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: exec example - DoGetFromStream timeout

Post by doublemax »

Do you read the input stream one last time when wxProcess::OnTerminate() is called?
Use the source, Luke!
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: exec example - DoGetFromStream timeout

Post by Widgets »

Not really; the process stays alive until the app exits.
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
User avatar
doublemax
Moderator
Moderator
Posts: 19114
Joined: Fri Apr 21, 2006 8:03 pm
Location: $FCE2

Re: exec example - DoGetFromStream timeout

Post by doublemax »

Widgets wrote: Fri Oct 29, 2021 12:45 am Not really; the process stays alive until the app exits.
How does that work? Can you just send another command to the process without restarting it?
Use the source, Luke!
Widgets
Ultimate wxWidgets Guru
Ultimate wxWidgets Guru
Posts: 534
Joined: Thu Jun 01, 2006 4:36 pm
Location: Right here!

Re: exec example - DoGetFromStream timeout

Post by Widgets »

Yes, the 'working part' of ET stays resident and is supplied command 'commands' via its input stream and then provides the results via its output stream. This usually means the ET has to scan the input image on disk to parse it and output the requested data.
This can and is repeated several times throughout a normal session.

There is also a C++ interface available by the author of ET, but it is based on and requires a Posix compliant compiler and libraries. My attempts at trying to port it to Windows proved too much work and hence I picked the piped frame as a base.

At this point, I also want to clarify the the reason I found the code you proposed to 'hang'. It was because, as part of my workflow, there are times when I called the input routine just to make sure there are no leftover bytes in the stream.

The -stay_open interface does have a way to signal when the stdin stream - to my app - is done by sending a unique string to signal the end of data and my latest code now uses that, rather than just waiting for a timeout from CanRead(). ( but still needs some refining for buffer size border cases)
Up to recently, things had worked with the code as it was and I was too focused on getting some of the other features of the app to work as expected. As I explored files with more metadata, off and on, things went wrong. Initially, I did not even notice the missing data. But eventually, the problem became too conspicuous and consistent and I investigated.
I had always been aware of the stdin protocol but never paid much attention to stderr - it hardly ever produced any output, and when it did, it was not to hard to figure out why and how to fix it.

When our conversation here convinced me that using CanRead() by itself would not be workable for the general case, my big concern was that while I knew about the terminating string for stdin, I was concerned that I would still need to work out something for stderr.
Finally I went back to the documentation for ET and I realized that there was a similar protocol for stderr.
Right now I have worked out a protocol for stdin, waiting for the terminating string and my next step will be to modify the code to also handle stderr
After all, there is no need to ask the author of ET to 'fix'; it is well enough thought out, all I needed to do was RTFM better :-)
Environment: Win 10/11 64-bit & Mint 21.1
MSVC Express 2019/2022
wxWidgets 3.2.2
Post Reply