AppDomain.CurrentDomain.UnhandledException is Not Called

I've got a similar problem as described in thread https://forum.photonengine.com/discussion/9955/unhandled-exception-handler-not-called.

The difference is that we would like to do more than just logging: we would like to create a memory dump so any crash could be analyzed.
public class MyApplication : ApplicationBase
{
protected override void Setup()
{
// ...
AppDomain.CurrentDomain.UnhandledException += OnUnhandledException;
// ...
}

private static void AppDomain_OnUnhandledException(object sender, UnhandledExceptionEventArgs args)
{
_log.Fatal(args.ExceptionObject);
MemoryDump.MiniDump.Create(Process.GetCurrentProcess(), Path.Combine(Path.GetTempPath(), "MyApplication.dmp"));
}
}
I made sure the the Unhandled Exception Policy is configured as "TerminateProcess" so Photon will not swallow all the exceptions. This information was available here: https://doc.photonengine.com/en-us/server/current/app-framework/exception-handling#configuring_the_unhandledexception_policy

The server application now closes if an exception happens (opposed to the "Ignore" option), but the UnhadnledException callback is still not called.

The version of the Photon Server I'm running is 4.0.29.11263

Any help would be appreciated.

Comments

  • hi, @Shakaron
    did you see there setting about memory dump? it should be in bin_win64/log folder. do you get them?

    best,
    Ilya
  • Shakaron
    Shakaron
    edited September 2019
    Hi Ilya,

    thanks for the quick response.

    I've added a simple throw new Exception("ExceptionForTesting"); into the above mentioned Setup() as the last line.

    The application crashes as expected, as the exception handling policy is still "TerminateProcess". However, I didn't see any crash dump file created in Photon-OnPremise-Server-SDK_v4-0-29-11263\deploy\bin_Win64\log\ folder. Only two log files there: PthotonCLR.log and Photon-MyApplication-20190906.log

    The latter contains lines, such as
    "29468: 20:28:43.051 - Will produce at most: 10 crash dumps"
    but apart from that, I didn't see any trace of memory dumps. And after this line, there was no info, where the dump is created, if any.

    29468: 20:28:52.039 - PulseIQ:2 - PhotonRunning() failed.
    Exception:
    System.Exception: ExceptionForTesting
    at MyApplication.Setup() in C:\_______\MyApplication.cs:line 196
    at Photon.SocketServer.ApplicationBase.PhotonHostRuntimeInterfaces.IPhotonControl.OnPhotonRunning() in h:\svncontent\photon-socketserver-sdk_cloud\src\Photon.SocketServer\ApplicationBase.cs:line 1189
    at PhotonHostRuntime.PhotonDomainManager.PhotonPlainAppDomainBehavior.PhotonRunning()
    at PhotonHostRuntime.PhotonDomainManager.PhotonRunning()
    29468: 20:28:52.040 - CService::OnException() - Exception: CManagedHost::PhotonRunning() - Failed in AppDomain: 2 - ExceptionForTesting
  • @Shakaron does it work if you set `Ignore` policy

    best,
    ilya
  • Hi Ilya,

    I think I was in a bit of a rush on Friday when I posted my original post. Let me share my update on the subject.

    Update:

    I've done a bit more testing, at it seems like no matter what I set for UnhandledExceptionPolicy (ReloadAppDomain, Ignore, or TerminateProcess), the application always closes and calls Photon.SocketServer.ApplicationBase.TearDown().

    This only happens if I throw an exception in Photon.SocketServer.ApplicationBase.Setup().

    If I try to throw an exception in Photon.SocketServer.Rpc.Peer.OnOperationRequest(), the exception is always swallowed, regardless what the UnhandledExceptionPolicy setting is.

    I've also found settings how to set up the Memory Dump: https://doc.photonengine.com/en-us/server/current/app-framework/exception-handling#photon_core_debugging

    Unfortunately, this doesn't work either. I've tried it with the above variations: all 3 UnhandledExceptionPolicy policy, throwing exception in Photon.SocketServer.ApplicationBase.Setup() and Photon.SocketServer.Rpc.Peer.OnOperationRequest(). Never saw any memory dump in the Photon-OnPremise-Server-SDK_v4-0-29-11263\deploy\bin_Win64\log\ folder.

    Note #1:

    The SDK says version 4.0.29.11263, but if I run PhotonSocketServer.exe /? it says
    Photon Socket Server - 4.0.28.2962

    Usage: PhotonSocketServer.exe Followed by the following command line options.
    /help Display this list.
    /noMessages Must be first. Do not display status messages.
    /install name Installs an instance of the service named 'name'.
    /installCounters Installs performance counters for this service.
    /remove name Removes the instance of the service named 'name'.
    /removeCounters Removes performance counters for this service.
    /run name Runs the service as a normal process for debugging.
    /stop Request all instances of Photon to shutdown and exit.
    /stop1 name Request instance 'name' of Photon shutdown and exit.
    /configPath path Use the supplied path to locate the config file.
    /config file Use the supplied config file.

    Note that /config and /configFile can be used with /install and /run

    Note that only /noMessages can be combined with all other command line options.
    So it's a slightly different version that what the SDK indicates.

    Note #2:

    I'm pretty sure my configuration is all right, as I've noticed if I mess up something in the configuration file (e.g. wrong value), Photon doesn't start.

    So the configs seem to be correct(as Photon starts), but neither the MemoryDump nor the UnhandledException settings work, nor does AppDomain catch any unhandled exception (as Photon handles them anyway), so I'm running out of options. Anything else I could try? Any idea why these settings make no difference?

    Is there a sample project that shows that these settings work as intended?
  • hi, @Shakaron

    There is one feature related to fibers. By default, peers are using fiber with FailSafeExecutor. So, if some exception happens during fiber's action execution, it is, yes, swallowed to not break room/fiber workflow, but it is also logged. Almost everything is going on in fibers, so that is the reason why you do not see unhandled exceptions. They all are handled.

    This is how we work with all that stuff.

    if you want to change this, you have to override ApplicationBase.CreatePeerFiber. By default, it creates PoolFiber with FileSafeExecutor. You may use DefaultExecutor or your own executor.

    best,
    ilya
  • Hi, Ilya.

    I've trying to use the DefaultExecutor to customize the IFiber creation. Still not exception caught/logged coming for Photon.SocketServer.Rpc.Peer.OnOperationRequest().

    Then I created my own IExecutor:
    public class MyApplication : ApplicationBase
    {
    private class MyExecutor : IExecutor
    {
    public void Execute(List toExecute)
    {
    foreach (var action in toExecute)
    Execute(action);
    }

    public void Execute(Action toExecute)
    {
    try
    {
    toExecute();
    }
    catch (Exception e)
    {
    log.Fatal($"Exception in ThreadFiber: {e}");
    throw;
    }
    }
    }

    protected override IFiber CreatePeerFiber(InitRequest request) => new PoolFiber(new MyExecutor());
    }
    I can see a new PoolFiber with MyExecutor being created. I can see toExecute() being called.

    However, I never catch the exception, never get to the line log.Fatal(...)

    A quick glance at the call stack can probably explain why:
    >	MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.CompWorld.OperationHandlerWorld.OnHandlerMessage(MyApplication.Local.Common.SubCode subcode, Photon.SocketServer.OperationRequest request, Photon.SocketServer.OperationResponse response, MyApplication.Local.ControlServer.CompClientPeer.BaseClientPeer peer, Photon.SocketServer.SendParameters sendParameters) Line 26	C#
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.CompClientPeer.BaseClientPeer.OnOperationRequest(Photon.SocketServer.OperationRequest operationRequest, Photon.SocketServer.SendParameters sendParameters) Line 61 C#
    [External Code]
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.ControlServerApplication.MyExecutor.Execute(System.Action toExecute) Line 196 C#
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.ControlServerApplication.MyExecutor.Execute(System.Collections.Generic.List toExecute) Line 188 C#
    [External Code]
    Please, notice the [External Code] between OnOperationRequest and MyExecutor.Execute.

    That external code (probably Peer) can easily handle the exception, preventing it to leave the execution, preventing it to be caught on the level of MyExecutor.

    Please, find the logs here
  • Hi Iilya,

    I've tried to use the DefaultExecutor, but nothing changed. I even created my own executor:
    public class MyApplication : ApplicationBase
    {
    private class MyExecutor : IExecutor
    {
    public void Execute(List toExecute)
    {
    foreach (var action in toExecute)
    Execute(action);
    }

    public void Execute(Action toExecute)
    {
    try
    {
    toExecute();
    }
    catch (Exception e)
    {
    log.Fatal($"Exception in ThreadFiber: {e}");
    throw;
    }
    }
    }

    protected override IFiber CreatePeerFiber(InitRequest request) => new PoolFiber(new MyExecutor());
    }
    I can see the CreatePeerFiber being called. Execute being called. toExecute() being called. Yet I never get into the catch branch, when I throw an exception in Photon.SocketServer.Rpc.Peer.OnOperationRequest.

    Looking at the call stack may reveal the culprit here:
    >	MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.CompWorld.OperationHandlerWorld.OnHandlerMessage(MyApplication.Local.Common.SubCode subcode, Photon.SocketServer.OperationRequest request, Photon.SocketServer.OperationResponse response, MyApplication.Local.ControlServer.CompClientPeer.BaseClientPeer peer, Photon.SocketServer.SendParameters sendParameters) Line 26	C#
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.CompClientPeer.BaseClientPeer.OnOperationRequest(Photon.SocketServer.OperationRequest operationRequest, Photon.SocketServer.SendParameters sendParameters) Line 61 C#
    [External Code]
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.ControlServerApplication.MyExecutor.Execute(System.Action toExecute) Line 196 C#
    MyApplication.Local.ControlServer.dll!MyApplication.Local.ControlServer.ControlServerApplication.MyExecutor.Execute(System.Collections.Generic.List toExecute) Line 188 C#
    [External Code]
    As you can notice there is an [External Code] call between OnOperationRequest and MyExecutor.Execute. That must be Peer invoking OnOperationRequest and handling the exception. So it seems like I will never be able to have a convenient top level class to catch and handle the exceptions.

    It seems like I have to push down the exception handling, logging and dump file creation to every individual call when I may have an exception. :(

    If it is of any help, please, find the logs here

    No exception logged there. No dump file created (Dump and Exception settings are same as before).
  • hi, @Shakaron

    log you have provided is native log. it does not contain anything about managed exception. It can be useful if you have stackoverflow

    please check PhotonCLR.log in same folder or 'your_root_folder/log'

    your exceptions are caught and logged here
    ```
    protected void OnOperationRequestInternal(OperationRequest operationRequest, SendParameters sendParameters)
    {
    try
    {
    this.OnOperationRequest(operationRequest, sendParameters);
    }
    catch (Exception e)
    {
    log.Error("Exception during handling of operation", e);
    }
    }
    ```
    so, make sure that logging is correctly setup in your application.

    best,
    ilya
  • Hi Iilya,

    I had a break point in the catch block on line log.Fatal($"Exception in ThreadFiber: {e}"); and that was never caught.

    The RhotonCLR.log is located here

    Kind regards,
    --
    Shakaron
  • Hi, @Shakaron
    That is totally ok because we intercept exception earlier. See the example I gave in the previous post.

    PhotonCLR will contain only unhandled exceptions. You have to setup logging for your application to be able to see logged errors. Just copy log setup from LoadBalancing applications code.

    best,
    ilya