I'm writing small utility (VC 2010, no clr) that does one simple task (rasterizing) using 3rd party library. Later utility will be used by bigger application. Sometimes the utility crashes because of some heap corruption in 3rd party library. That is OK, but Windows (Vista/2008) shows well known dialog "Program has stopped working ... Close/Debug program." which is not appropriate in my case (Server side). Utility should crash/terminated silently w/o any visible effects.
To do that I installed SEH for unhandled exception (SetUnhandledExceptionFilter). The handler is perfectly invoked for exceptions like AV ( *(PDWORD)0 = 0 ), but for some reason it is not invoked in the case of heap corruption. Corruption happens in dllmain of one of 3rd party library dlls while it is being unloaded.
Couple of questions. Can anybody explain why the handler is not invoked? Is there any another way to prevent that dialog?
but for some reason it is not invoked in the case of heap corruption. Corruption happens in dllmain of one of 3rd party library dlls while it is being unloaded.
Heap corruption is undefined behavior. It may throw exceptions, it may do otherwise. If a buggy third party library is messing up your heap, then the question is "why are you allowing them to mess with your heap in the first place?"
The "program has stopped working" dialog is shown whenever a process terminates abnormally. Not all abnormal process terminations result from exceptions. Many errors (such as stack overflow, misaligned stack, etc.) cause instant termination of the process, which may show that message but will not give you a chance to handle the error.
(Also, see Hans' awesome comment above)