SIGSEGV: egaes256_init

Hi,

recently I've got some crashes on Crashlytics:
Crashed: Thread: SIGSEGV  0x0000000000000000
       at egaes256_init + 305(aes256.c:305)
       at ExitGames::Photon::Internal::Encryption::decrypt(unsigned char const*, int, unsigned char const*, unsigned char**, int*) + 74(Encryption.cpp:74)
       at ExitGames::Photon::Internal::Encryption::decrypt(unsigned char const*, int, unsigned char const*, unsigned char**, int*) + 74(Encryption.cpp:74)
       at ExitGames::Photon::Internal::PeerBase::deserializeOperationResponse(unsigned char*, bool, int, unsigned char) + 569(PeerBase.cpp:569)
       at ExitGames::Photon::Internal::PeerBase::deserializeOperation(unsigned char*, int) + 544(PeerBase.cpp:544)
       at ExitGames::Photon::Internal::TPeer::dispatchIncomingCommands() + 158(TPeer.cpp:158)
       at ExitGames::Photon::Internal::PeerBase::service(bool) + 206(PeerBase.cpp:206)
       at ExitGames::Photon::PhotonPeer::service(bool) + 169(PhotonPeer.cpp:169)
       at ExitGames::LoadBalancing::Client::service(bool) + 700(Client.cpp:700)

Right before each and every crash there are some failed authentication operations (we have a custom token-based authentication). I wasn't being able to reproduce the issue. It happens on Android, Realtime SDK v4.1.15.0. Hope you guys might have some ideas.

Thank you,
Alex

Comments

  • Hi @abelthefirst.

    "Right before each and every crash there are some failed authentication operations"
    I assume you know this because you have log files that indicate this?
    Can you provide the Photon log output?
  • Hi @Kaiserludi ,

    thank you for your quick response.
    I assume you know this because you have log files that indicate this?
    Can you provide the Photon log output?
    That's not entirely the case. Every failed authentication operation is reported to the Crashlytics, as a non-fatal error, like this:
    Non-fatal Exception: ******.PhotonManagerImpl$PhotonManagerException: Photon connection error: CUSTOM_AUTHENTICATION_FAILED The token has expired
    
    Thus, for a given user I can see a list of events, but not the complete Photon log output. That would only be possible, if I reproduce the issue locally, which as I mentioned is not the case unfortunately.

    Thank you,
    Alex
  • Hi @abelthefirst.
    CUSTOM_AUTHENTICATION_FAILED The token has expired

    That might actually be an extremely helpful piece of information on how to reproduce this issue locally.

    We are looking into this. I will update you when I have more information on this issue.
  • Hi @abelthefirst.

    It looks like for some unknown reason the client receives an encrypted message from the server in a state in which encryption is not initialized on the client yet. This should not actually be possible as the server should not send encrypted messages until encryption has been established.
    Unfortunately the current version of the client relies on this no being possible and hence when it receives an encrypted message from the server, it assumes that encryption has been established and blindly attempts to decrypt the messages, without verifying first that encryption actually has been established. This leads to the client attempting to de-reference a NULL pointer. We will add a check in the client that encryption has actually been established before attempting to decrypt in the next release.

    However this does not explain WHY the server is sending an encrypted messages in this state.

    I have checked the Photon Cloud admin panel for the company that I think you belong to and could see that you are running custom server side code via the Photon Enterprise Cloud Plugin interface.

    Could it be the case that your custom server side code sends encrypted messages to the client?
    Maybe it somehow gets into a state in which it thinks that encryption has been established, but either encryption has not been established yet, or the connection is already getting cleaned up and for an unknown reason there is still an encrypted message getting sent, when the client is no in a state anymore to decrypt it.
    Both should not actually been possible, but I don't see another way how this could happen.

    I think you need to find a way to reproduce this issue for you/us to be able to track down the cause.
  • edited January 15
    Hi @Kaiserludi ,

    I work closely with the backend team right now, but we didn't find anything particularly weird just yet.

    In the meantime I was doing a little bit more debugging on the client side, while my authentication token is expired. As far as I can tell (please correct me if I'm wrong), it's a disconnect call which destroys an established encryption. The only encrypted incoming message, that I encountered, was an "GET_REGIONS"" operation result. Since disconnect might be initiated by the server, is it a possibility, that the "incoming queue" contains both "server disconnect" AND "get regions" messages? In that case first message might cause encryption destruction and second message will actually crash the client.

    Again, that is not something I was able to reproduce, just some assumptions.

    Thank you,
    Alex
  • Hi @abelthefirst.
    As far as I can tell (please correct me if I'm wrong), it's a disconnect call which destroys an established encryption.
    Note that under the hood the client disconnects from the name server when it connects to a regions master server, disconnects from a master server, when it enters a room and disconnects from a game server when it leaves a room. Whenever the client connects to a different server under the hood, it needs to reestablish the encryption, as that new server does not have a matching key for the encryption that the client has established to the server to which it was previously connected.
    Since disconnect might be initiated by the server, is it a possibility, that the "incoming queue" contains both "server disconnect" AND "get regions" messages? In that case first message might cause encryption destruction and second message will actually crash the client.
    In that case the servers message will have been sent before the disconnect and hence get processed by the client before the disconnect, when encryption still exists.
    Neither will the server send any more messages to client after having sent it a disconnect, nor will the client process any more incoming messages on that connection once it has received the disconnect.
Sign In or Register to comment.