Fatal exceptions logging

[Big Muscle]

Hello,

I am trying to implement fatal exceptions logging into our C++ application that runs on Raspberry Pi (Raspbian Lite Buster, g++ 8.3.0, wxWidgets 3.1.3.0, GTK+2.0), so the application generates the log with stack anytime it crashes.

This is the main part of the code:

Code: Select all

class COurApp: public wxApp
{
public:
	COurApp() {
		wxHandleFatalExceptions();
	}
	
	...
	
	virtual void OnFatalException() {
		wxLogError("Application crashed!");

		wxString strDirCrashLogs = "crashlogs";
		wxMkDir(strDirCrashLogs, wxS_DIR_DEFAULT);

		// log information to debug report
		wxDebugReportCompress report;
		report.SetCompressedFileDirectory(strDirCrashLogs);
		report.AddAll();

		// calling Show() is not mandatory, but is more polite
	//	if (wxDebugReportPreviewStd().Show(report))
		{
			if (report.Process()) {
				wxLogMessage("Report generated in \"%s\".", report.GetCompressedFileName());
				report.Reset();
			}
		}
	}	
}

The code works. The ZIP file with XML crash log is correctly generated but the included callstack is absolutely useless. It contains only the information about this exception handler. And nothing more.

Code: Select all

<stack>
<frame level="0" function="COurApp::OnFatalException()" offset="0" address="0x96388" file="/home/pi/OurApp/src/main.cpp" line="88"/>
<frame level="1" function="wxFatalSignalHandler" offset="0" address="0x38be30"/>
<frame level="2" function="__default_sa_restorer" offset="0" address="0x76024120" module="/lib/arm-linux-gnueabihf/libc.so.6"/>
</stack>

These are our compiler flags

Code: Select all

CFLAGS  := -ggdb3 `wx-config --cxxflags` `pkg-config --cflags gtk+-2.0` -std=c++14 -pipe
LIB         := `wx-config --libs` `pkg-config --libs gtk+-2.0`

My question is how to generate usable callstack with the location where the code actually crashes? Are some other compiler flags needed?
Thank you.

[alys666]

cant solve the problem just now, but can explain a reason.
when c++ exception is thrown, c++ runtime will search closest exception handler and unroll the stack. if there is no handler in current frame(try ..catch statement), it will call destructors and close the frame, if there is no handler in previous frame - it will close it...etc.
in your case it has found the handler(catch statement) in topmost frame.
and this virtual method OnFatalException is just invoked from this catch statement. But all frames are closed, and you have them lost.
To have not them lost, you must invoke stack dump function at a moment of throw statement. Just now don't know how to do it.
or may be C++ runtime has this functionality - to get stack dump at throw statement.
I used to work with small C++ custom runtimes, and cannot just now answer this question.
at least this is not about wxWidgets, but some common features of c++ implementation. imo.

try to google something like - "c++ dump stack at exception"

[Big Muscle]

Thank you for the explanation. Has it something to do with stack unwinding? I have not much experience withi this, I have seen some compiler switch -fno-exceptions. Could it be connected?

I just made a simple experiment with this code:

Code: Select all

int* crash = NULL;
*crash = 7;

When this code executes in main thread, exception handler generates complete stack trace:

Code: Select all

<stack>
<frame level="0" function="COurApp::OnFatalException()" offset="0" address="0x96370" file="/home/pi/OurApp/src/main.cpp" line="88"/>
<frame level="1" function="wxFatalSignalHandler" offset="0" address="0x38be18"/>
<frame level="2" function="__default_sa_restorer" offset="0" address="0x75fb2120" module="/lib/arm-linux-gnueabihf/libc.so.6"/>
<frame level="3" function="CSettings::Open(ICamera*)" offset="0" address="0x75c38" file="/home/pi/OurApp/src/Settings.cpp" line="261"/>
<frame level="4" function="wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&)" offset="0" address="0x3a6e50"/>
<frame level="5" function="wxEvtHandler::SearchDynamicEventTable(wxEvent&)" offset="0" address="0x3a732c"/>
...

When this code executes in different thread (run by std::thread), stack trace contains only 3 lines:

Code: Select all

<frame level="0" function="COurApp::OnFatalException()" offset="0" address="0x96104" file="/home/pi/OurApp/src/main.cpp" line="88"/>
<frame level="1" function="wxFatalSignalHandler" offset="0" address="0x38bbb0"/>
<frame level="2" function="__default_sa_restorer" offset="0" address="0x7604c120" module="/lib/arm-linux-gnueabihf/libc.so.6"/>

[Big Muscle]

And currently we're hunting for one specific crash. When attaching GDB, it randomly and very rarely crashes with the following stack trace:

Code: Select all

Thread 1 "Thermal" received signal SIGSEGV, Segmentation fault.
0x7692c940 in ?? () from /usr/lib/arm-linux-gnueabihf/libcairo.so.2
(gdb) bt
#0  0x7692c940 in  () at /usr/lib/arm-linux-gnueabihf/libcairo.so.2
#1  0x76961d88 in  () at /usr/lib/arm-linux-gnueabihf/libcairo.so.2

Since we don't use libcairo in our code, it must be wxWidgets calling this library. But due to only these 2 lines are contained in the strack trace, we are unable to debug it properly.

[alys666]

from some docs:

Compiling with -fno-exceptions disables exceptions support and uses the variant of C++ libraries without exceptions. Use of try, catch, or throw results in an error message.

Linking objects that have been compiled with -fno-exceptions automatically selects the libraries without exceptions. You can use the linker option --no_exceptions to diagnose whether the objects being linked contain exceptions.

because all this C++ exceptions kitchen is quite heavy and brings more code in your exe file, people like to write code without exceptions(me too ) and compiler helps them. You just have to mention this option -fno-exceptions on your compiler options list.

Yes. it is "stack unwinding"(I m not a native english speaker )
Anyway - OnFatalException - is not that method, you need to dump the stack. When it is called by wxWidgets, all your stack is already unwound.

[alys666]

if "Thermal" is not a main thread then....
never worked with cairo... bit this is some drawings library.
and if it is called in some secondary thread -> then you are doing drawings from secondary thread...
but wxWidgets drawing cannot be issued from secondary thread, only from main.
do not call wxWidgets functionality, except of mentioned as thread safe, from secondary thread. Look at forum discussions about it.

added...
I checked - gdb reports main thread as thread 1.
so it looks like fault inside main thread... anyway it could be produced by using wxWidgets from come other thread(at least i cannot exclude this case...).

[alys666]

and quite usual advice from me - i write code with highest warnings level (absolutely recommended)
"-Wall" "-Wextra"
also i use additional warnings in my projects -
"-pedantic" "-pedantic-errors" "-Wredundant-decls" "-Wcast-align" "-Wundef" "-Wfloat-equal" "-Wunreachable-code" "-Wmissing-include-dirs" "-Wpointer-arith" "-Wwrite-strings" "-Wswitch" "-Wswitch-default" "-Wswitch-bool".
So first of all, add "-Wall" "-Wextra" and rebuild your project. Correct all places where warning is.

[Big Muscle]

if "Thermal" is not a main thread then....
never worked with cairo... bit this is some drawings library.
and if it is called in some secondary thread -> then you are doing drawings from secondary thread...
but wxWidgets drawing cannot be issued from secondary thread, only from main.
do not call wxWidgets functionality, except of mentioned as thread safe, from secondary thread. Look at forum discussions about it.

added...
I checked - gdb reports main thread as thread 1.
so it looks like fault inside main thread... anyway it could be produced by using wxWidgets from come other thread(at least i cannot exclude this case...).

According to the debugger, it seems to be the main thread. I never call wxWidgets functions from the other threads. I don't use cairo, is it probably invoked from wxWidgets somewhere? But it is impossible to say due to the missing frames in the stack trace.

The application is just a wxPanel where wxImage and short text are drawn on and wxTimer calls Refresh(false) every 100 ms.

[alys666]

gdb shows strange things, there must be upper frames...if you have not damaged the stack by something.
what about stack size? what about return of address of some stack variable as pointer?
have you verified your code with -Wall - Wextra?

added. such small devices as rusberry, have limited resources and you must take care about size of stack and size of free memory for memory manager.
else behavior of your program would be unpredictable and of gdb too.
I do not see another reason for now - why gdb cannot backtrace frames. but he definitely can not.
for me it looks like the damaged stack

[Big Muscle]

After long experiments, I finally discovered the bug. You were probably right with the stack corruption. The bug was caused by memcpy copying data to invalid location, thus behaviour was unpredictable and it crashed randomly - probably when it overwrote some other important data on the stack. Now we run several days without any crashes.

But my initial question is still valid. I just tried to put the following code to initiate crash in the wxEVT_LEFT_UP handler (in the main thread).

Code: Select all

int* crash = NULL;
*crash = 1;

Yes, this crashes immediately but the crash log contains only:

Code: Select all

<stack>
<frame level="0" function="COurApp::OnFatalException()" offset="0" address="0x5c360"/>
<frame level="1" function="wxFatalSignalHandler" offset="0" address="0x3eb1a0"/>
<frame level="2" function="__default_sa_restorer" offset="0" address="0x75fea120" module="/lib/arm-linux-gnueabihf/libc.so.6"/>
</stack>

Although GDB displays the stack correctly:

Code: Select all

Thread 1 "Thermal" received signal SIGSEGV, Segmentation fault.
0x000654c4 in CMainWindow::OnThermalMouseUp(wxMouseEvent&) ()
(gdb) bt
#0  0x000654c4 in CMainWindow::OnThermalMouseUp(wxMouseEvent&) ()
#1  0x00407b60 in wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&) [clone .part.49] ()
#2  0x004096cc in wxEvtHandler::SearchDynamicEventTable(wxEvent&) ()
#3  0x00409a80 in wxEvtHandler::TryHereOnly(wxEvent&) ()
#4  0x00409bf0 in wxEvtHandler::ProcessEvent(wxEvent&) ()
#5  0x0040b59c in wxEvtHandler::SafelyProcessEvent(wxEvent&) ()
#6  0x000c3234 in gtk_window_button_release_callback ()
#7  0x76b27280 in  () at /usr/lib/arm-linux-gnueabihf/libgtk-x11-2.0.so.0

And after "continue":

Code: Select all

(gdb) bt
#0  0x75f66f24 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x75f52230 in __GI_abort () at abort.c:79
#2  0x003eb1a4 in wxFatalSignalHandler ()
#3  0x75f68120 in <signal handler called> () at ../sysdeps/unix/sysv/linux/arm/sigrestorer.S
#4  0x000654c4 in CMainWindow::OnThermalMouseUp(wxMouseEvent&) ()
#5  0x00407b60 in wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&) [clone .part.49] ()
#6  0x004096cc in wxEvtHandler::SearchDynamicEventTable(wxEvent&) ()
#7  0x00409a80 in wxEvtHandler::TryHereOnly(wxEvent&) ()
#8  0x00409bf0 in wxEvtHandler::ProcessEvent(wxEvent&) ()
#9  0x0040b59c in wxEvtHandler::SafelyProcessEvent(wxEvent&) ()
#10 0x000c3234 in gtk_window_button_release_callback ()
#11 0x76ae8280 in  () at /usr/lib/arm-linux-gnueabihf/libgtk-x11-2.0.so.0

[alys666]

Yes, this crashes immediately but the crash log contains only:

Code: Select all

<stack>
<frame level="0" function="COurApp::OnFatalException()" offset="0" address="0x5c360"/>
<frame level="1" function="wxFatalSignalHandler" offset="0" address="0x3eb1a0"/>
<frame level="2" function="__default_sa_restorer" offset="0" address="0x75fea120" module="/lib/arm-linux-gnueabihf/libc.so.6"/>
</stack>

Although GDB displays the stack correctly:

it's because OnFatalException handler is called after stack unwinding. and using it you CANNOT dump stack state for error.
but debugger gets stack frames information before(if any) stack unwinding. debugger makes it correctly. but C++ runtime not.
if you want to implement c++ stack unwinding you must correct somehow exceptions handling code in C++ runtime sources. i do not say it's impossible, but i think it is expensive and not needed for usual design.

[alys666]

BTW, your gdb dump "after continue" shows that your frames are not unwound inside OnFatalException.
and it looks like stack frames info must be available inside it.
i never used this functionality of wxWidgets...you're getting info using addAll...but documentation for this function is not clear.
for me addCurrentContext() must give a stack.
or may be try to add both info:
addCurrentContext();
addExceptionContext();

[Big Muscle]

I replaced AddAll() with these both two and now generated ZIP file contains two same XML files with same "bad" stack trace as before.

If I have time, I will look deeply into src/unix/stackwalk.cpp, maybe there is something that stops other frames to be written into exception log.

[alys666]

I added to my app this crash logs, and in my case everything works fine. And all the stack is logged.
but my stack is a way deeper than yours.

Code: Select all

<frame level="0" function="CodeBeeApp::OnFatalException()" offset="00000000" file="...." line="3048"/>
<frame level="1"/>
<frame level="2"/>
<frame level="3" function="IdeFrame::__onMenuItem(wxCommandEvent&)" offset="00000000" file="...." line="2719"/>
<frame level="4" function="wxAppConsoleBase::CallEventHandler(wxEvtHandler*, wxEventFunctor&, wxEvent&) const" offset="0000003e"/>
<frame level="5" function="wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&)" offset="00000057"/>
<frame level="6" function="wxEvtHandler::SearchDynamicEventTable(wxEvent&)" offset="0000005e"/>
<frame level="7" function="wxEvtHandler::TryHereOnly(wxEvent&)" offset="0000001f"/>
<frame level="8" function="wxEvtHandler::DoTryChain(wxEvent&)" offset="00000043"/>
<frame level="9" function="wxEvtHandler::ProcessEvent(wxEvent&)" offset="00000045"/>
<frame level="10" function="wxEvtHandler::SafelyProcessEvent(wxEvent&)" offset="00000007"/>
<frame level="11" function="wxMenuBase::SendEvent(int, int)" offset="00000130"/>
<frame level="12"/>
<frame level="13" function="g_closure_invoke" offset="00000145"/>
<frame level="14"/>
<frame level="15" function="g_signal_emit_valist" offset="00000fbc"/>
<frame level="16" function="g_signal_emit" offset="0000008f"/>
<frame level="17" function="gtk_widget_activate" offset="0000006e"/>
<frame level="18" function="gtk_menu_shell_activate_item" offset="000000fd"/>
<frame level="19"/>
<frame level="20"/>
<frame level="21" function="g_closure_invoke" offset="00000145"/>
<frame level="22"/>
<frame level="23" function="g_signal_emit_valist" offset="00000a59"/>
<frame level="24" function="g_signal_emit" offset="0000008f"/>
<frame level="25"/>
<frame level="26" function="gtk_propagate_event" offset="000000c4"/>
<frame level="27" function="gtk_main_do_event" offset="000002cb"/>
<frame level="28"/>
<frame level="29" function="g_main_context_dispatch" offset="000002a7"/>
<frame level="30"/>
<frame level="31" function="g_main_context_iteration" offset="0000002c"/>
<frame level="32" function="gtk_main_iteration" offset="00000021"/>
<frame level="33" function="wxWindow::DoPopupMenu(wxMenu*, int, int)" offset="000000ed"/>
<frame level="34" function="wxWindowBase::PopupMenu(wxMenu*, int, int)" offset="00000043"/>
<frame level="35" function="wxWindowBase::PopupMenu(wxMenu*, wxPoint const&)" offset="00000000" file="/usr/include/wx-3.0/wx/window.h" line="1216"/>
<frame level="36" function="IdeFrame::popupDynamicMenu()" offset="00000000" file="..." line="2477"/>
<frame level="37" function="IdeFrame::__onMenuItem(wxCommandEvent&)" offset="00000000" file="..." line="2513"/>
<frame level="38" function="wxAppConsoleBase::CallEventHandler(wxEvtHandler*, wxEventFunctor&, wxEvent&) const" offset="0000003e"/>
<frame level="39" function="wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&)" offset="00000057"/>
<frame level="40" function="wxEvtHandler::SearchDynamicEventTable(wxEvent&)" offset="0000005e"/>
<frame level="41" function="wxEvtHandler::TryHereOnly(wxEvent&)" offset="0000001f"/>
<frame level="42" function="wxEvtHandler::DoTryChain(wxEvent&)" offset="00000043"/>
<frame level="43" function="wxEvtHandler::ProcessEvent(wxEvent&)" offset="00000045"/>
<frame level="44" function="wxWindowBase::TryAfter(wxEvent&)" offset="00000068"/>
<frame level="45" function="wxAuiToolBar::OnLeftUp(wxMouseEvent&)" offset="00000128"/>
<frame level="46" function="wxAppConsoleBase::CallEventHandler(wxEvtHandler*, wxEventFunctor&, wxEvent&) const" offset="0000003e"/>
<frame level="47" function="wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&)" offset="00000057"/>
<frame level="48" function="wxEventHashTable::HandleEvent(wxEvent&, wxEvtHandler*)" offset="0000008b"/>
<frame level="49" function="wxEvtHandler::TryHereOnly(wxEvent&)" offset="0000004b"/>
<frame level="50" function="wxEvtHandler::ProcessEventLocally(wxEvent&)" offset="00000033"/>
<frame level="51" function="wxEvtHandler::ProcessEvent(wxEvent&)" offset="00000045"/>
<frame level="52" function="wxEvtHandler::SafelyProcessEvent(wxEvent&)" offset="00000007"/>
<frame level="53"/>
<frame level="54"/>
<frame level="55" function="g_closure_invoke" offset="00000145"/>
<frame level="56"/>
<frame level="57" function="g_signal_emit_valist" offset="00000a59"/>
<frame level="58" function="g_signal_emit" offset="0000008f"/>
<frame level="59"/>
<frame level="60" function="gtk_propagate_event" offset="000000c4"/>
<frame level="61" function="gtk_main_do_event" offset="000002cb"/>
<frame level="62"/>
<frame level="63" function="g_main_context_dispatch" offset="000002a7"/>
<frame level="64"/>
<frame level="65" function="g_main_loop_run" offset="000000c2"/>
<frame level="66" function="gtk_main" offset="000000b7"/>
<frame level="67" function="wxGUIEventLoop::DoRun()" offset="00000025"/>
<frame level="68" function="wxEventLoopBase::Run()" offset="000000a3"/>
<frame level="69" function="wxAppConsoleBase::MainLoop()" offset="00000056"/>
<frame level="70" function="wxEntry(int&, wchar_t**)" offset="00000070"/>
<frame level="71" function="main" offset="00000000" file="..." line="3219"/>
<frame level="72" function="__libc_start_main" offset="000000f0"/>
<frame level="73" function="_start" offset="00000000"/>

[Big Muscle]

I see your frames 1 and 2 are empty, there is even no __default_sa_restorer. Is it Raspberry/ARM with g++ ?

Did you compile wxWidgets library on your own? If yes, what settings did you use during the building?