[PUN2] Best Region: investigation and improvements

JohnTube
JohnTube ✭✭✭✭✭
Hi,

We have been getting various reports about clients (Unity Editor or build) unexpectedly not connecting to the same region when "Best Region" is used. By "unexpectedly", I meant clients in the same geographical area, connected to the same network or even running on the same machine.

We started addressing this issue on documentation, see "Best Region Considerations".

However, we think that this is not enough and we want to improve the heuristics we use to determine the "Best Region".
But in order to do this, we need your help. We need you to provide us with some logging entries which will help us investigate and decide what to do next.

To know which region you are connected to, one way is to log it in OnConnectedToMaster callback, something like this:
        public void OnConnectedToMaster ()
	{
              Debug.LogFormat("PUN2 client connected to region: {0}", PhotonNetwork.CloudRegion);
	}
First of all, you need to update to latest PUN2.

Logging instructions:

The region pinging results can be logged very easily:

In the PhotonServerSettings, set the "PUN Logging" level to Informational at least.
For the editor, you can force the "Best Region" pinging. Use the "Reset" button in the PhotonServerSettings to clear the "Best Region Preference".
Via code, you can also clear the setting:
PhotonNetwork.PhotonServerSettings.ResetBestRegionCodeInPreferences();

Log entries needed:

Two log entries are of interest:

"PUN got region list. [...]" and "Region Pinging Result [...]" which contains multiple lines.
We need those from the clients that have different opinions of which region to choose.
You can copy them directly from Unity Editor Console window or from Editor log or build log file (log files).

When you manage to reproduce the issue of two PUN2 clients, geographically located next to each other or connected to the same network or running on the same machine but were connected to different regions using "Best Region" (default), get the logs mentioned above using the instructions explained above as well.
You can copy those and send them to us via developer@photonengine.com, preferably use the same subject as this discussion: "[PUN2] Best Region: investigation and improvements". Do not forget to send us PUN2 version, Unity version and other details about the report that you could consider useful (if clients are connected to the same network, Unity Editor or build, which device, which OS, running on the same machine or not, etc.).

We appreciate your efforts and all your help.
Thank you for your time and understanding.

NOTE: PUN Classic is in maintenance or feature freeze mode. So we will focus on PUN2 only.

--

UPDATE / EDIT: We now have a new feature: "Development Region".
«1

Comments

  • Jebel
    Jebel
    edited February 2020
    Unfortunately, I have the problem. I have the same code and running the same machine, but the Clients connect to different regions. Unity Editor connect to "asia" but the Mac Build Client connect to "rue".
    My env:
    Mac OS 10.13.6
    Unity Editor: 2019.3
    Photon PUN: v2.16

    Unity Editor Log:
    PUN got region list. Going to ping minimum regions, based on this previous result summary: asia;134;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za
    Region Pinging Result: asia[92.38.183.13:5055]: 129ms
    Previous summary: asia;134;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za

    Unity Editor Running Image
    Mac Build Client Running Image
  • I'm new to Photon and follow up the office tutorial "PUN Basics Tutorial". I can't join the same room with the Unity Editor and Mac Build Client as the problem. How could i deal with the problem? Wait for your help. Thank you!
  • JohnTube
    JohnTube ✭✭✭✭✭
    edited February 2020
    Hi @Jebel,

    Thank you for choosing Photon and for your report!

    We would like to get the logs (ping summary) from the macOS build also to know the results of pinging each region and why it led to "rue".

    Otherwise the solution in this case is to reset best region preference from PhotonServerSettings OR just force an explicit unique fixed region at least for development purposes, also from PhotonServerSettings.

    May we know where are you located exactly?
  • Hi @JohnTube,
    Thank you for your response.
    I have tested again and got log as below:
    macOS build:
    PUN got region list. Going to ping minimum regions, based on this previous result summary: rue;110;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za
    Region Pinging Result: rue[92.38.191.5:5055]: 103ms
    Previous summary: rue;110;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za

    Unity Editor:
    PUN got region list. Going to ping minimum regions, based on this previous result summary: asia;128;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za
    Region Pinging Result: asia[92.38.183.13:5055]: 105ms
    Previous summary: asia;128;asia,au,cae,eu,in,jp,kr,ru,rue,sa,us,usw,za

    My IP: 125.73.202.255
    My Location: Guangxi, China
  • I have tested another case. I builded my project to an apk file and installed to my two android mobile phone. They are connected to the same wifi, but one is connected to region "asia" and the other one is connected to region "rue".
  • JohnTube
    JohnTube ✭✭✭✭✭
    edited February 2020
    Hi @Jebel,

    Thank you for your report!
    We will look into it.

    Of course it always help if you keep sending us pinging summary results from the logs, for instance from the two Android devices you mentioned in the last post.

    You could set a fixed region now for development purposes.
  • JohnTube
    JohnTube ✭✭✭✭✭
    edited February 2020
    Hey @Jebel,

    If it's not asking too much :)

    My colleague @Tobias asked for this:

    First reset best region selection (either from Unity Editor or using code: ServerSettings.ResetBestRegionCodeInPreferences()).
    Then set logging level to INFO.
    When you connect you will get the result of the 5 ping time per region and not just the average per region.
    We need those pings for all the regions and especially "asia" and "rue" here.

    It might be that your location is at the center of two regions.
    So since you are close to two regions, clients might choose one or the other.
    In production you could contact us to give you access to China region.

    Thank you!
  • Hi @JohnTube,
    That's all right. I have tested and got the logs as below by ten testing. It's almost connect to different regions. :D

    ns.exitgames.com resolved to these addresses: 137.116.144.239 (InterNetwork)

    1
    Region Pinging Result: rue[92.38.191.5:5055]: 110ms
    eu: 361 (800, 800, 349, 374, 800)
    us: 242 (251, 243, 243, 242, 242)
    usw: 198 (204, 200, 198, 198, 800)
    cae: 288 (289, 281, 300, 290, 281)
    asia: 126 (800, 127, 127, 126, 126)
    jp: 137 (135, 135, 138, 140, 800)
    au: 266 (271, 267, 268, 800, 265)
    sa: 368 (376, 368, 370, 368, 368)
    in: 348 (352, 800, 348, 348, 800)
    ru: 322 (800, 319, 800, 325, 800)
    rue: 110 (117, 800, 800, 800, 110)
    kr: 164 (172, 164, 165, 800, 800)
    za: 536 (800, 536, 536, 536, 800)

    2
    Region Pinging Result: asia[92.38.183.13:5055]: 103ms
    eu: 271 (278, 271, 272, 800, 272)
    us: 245 (249, 245, 245, 245, 245)
    usw: 195 (203, 195, 195, 195, 195)
    cae: 290 (298, 290, 290, 291, 291)
    asia: 103 (112, 103, 103, 103, 104)
    jp: 132 (138, 132, 132, 800, 132)
    au: 267 (274, 801, 267, 268, 268)
    sa: 396 (401, 397, 396, 800, 396)
    in: 334 (341, 334, 800, 334, 334)
    ru: 395 (800, 800, 800, 392, 399)
    rue: 110 (119, 110, 111, 800, 110)
    kr: 189 (190, 189, 189, 189, 800)
    za: 438 (444, 438, 438, 438, 438)

    3
    Region Pinging Result: rue[92.38.191.5:5055]: 111ms
    eu: 273 (279, 275, 800, 273, 273)
    us: 244 (252, 245, 245, 245, 244)
    usw: 193 (197, 193, 193, 196, 193)
    cae: 292 (299, 293, 293, 291, 291)
    asia: 129 (136, 129, 129, 129, 130)
    jp: 137 (150, 135, 134, 143, 800)
    au: 267 (273, 268, 266, 267, 267)
    sa: 383 (392, 382, 385, 384, 383)
    in: 415 (800, 800, 800, 415, 800)
    ru: 328 (334, 328, 334, 326, 326)
    rue: 111 (119, 111, 800, 111, 800)
    kr: 169 (800, 169, 800, 169, 169)
    za: 440 (447, 442, 439, 440, 440)

    4
    ns.exitgames.com resolved to these addresses: 137.116.144.239 (InterNetwork)
    Region Pinging Result: rue[92.38.191.5:5055]: 104ms
    eu: 270 (275, 268, 276, 270, 269)
    us: 242 (249, 243, 242, 242, 800)
    usw: 196 (800, 196, 196, 800, 196)
    cae: 287 (800, 288, 800, 287, 800)
    asia: 4000 (137, 800, 800, 800, 800)
    jp: 149 (156, 149, 149, 149, 149)
    au: 265 (268, 265, 800, 265, 265)
    sa: 382 (390, 382, 382, 382, 382)
    in: 413 (349, 800, 800, 800, 413)
    ru: 332 (800, 800, 332, 800, 800)
    rue: 104 (113, 104, 104, 800, 800)
    kr: 173 (800, 173, 800, 800, 174)
    za: 431 (435, 431, 432, 431, 432)

    5
    ns.exitgames.com resolved to these addresses: 137.116.144.239 (InterNetwork)
    Region Pinging Result: asia[92.38.183.13:5055]: 98ms
    eu: 274 (279, 274, 274, 800, 800)
    us: 250 (800, 250, 250, 252, 250)
    usw: 196 (201, 800, 800, 196, 800)
    cae: 281 (288, 800, 281, 281, 800)
    asia: 98 (107, 98, 800, 99, 98)
    jp: 130 (139, 130, 131, 131, 131)
    au: 265 (275, 267, 265, 266, 265)
    sa: 405 (413, 404, 405, 404, 407)
    in: 387 (800, 338, 800, 800, 436)
    ru: 333 (341, 333, 333, 333, 800)
    rue: 111 (118, 111, 800, 113, 110)
    kr: 174 (181, 175, 800, 173, 174)
    za: 430 (439, 431, 430, 430, 431)

    6
    ns.exitgames.com resolved to these addresses: 137.116.144.239 (InterNetwork)
    Region Pinging Result: us[92.223.82.17:5055]: 242ms
    eu: 343 (277, 800, 800, 800, 343)
    us: 242 (252, 800, 243, 242, 243)
    usw: 190 (800, 191, 192, 190, 190)
    cae: 289 (292, 800, 290, 800, 289)
    asia: 97 (800, 97, 97, 97, 97)
    jp: 146 (153, 146, 146, 146, 800)
    au: 268 (276, 800, 268, 268, 268)
    sa: 561 (537, 530, 530, 800, 623)
    in: 348 (355, 348, 349, 800, 800)
    ru: 376 (381, 377, 376, 800, 376)
    rue: 110 (115, 110, 110, 800, 800)
    kr: 164 (171, 164, 164, 164, 164)
    za: 446 (455, 447, 446, 446, 446)

    7
    Region Pinging Result: asia[92.38.183.13:5055]: 101ms
    eu: 282 (299, 281, 800, 800, 284)
    us: 251 (258, 800, 251, 251, 251)
    usw: 190 (800, 800, 800, 190, 800)
    cae: 287 (292, 287, 287, 287, 288)
    asia: 101 (111, 800, 101, 101, 800)
    jp: 150 (156, 150, 800, 150, 150)
    au: 267 (800, 267, 268, 267, 267)
    sa: 519 (525, 800, 800, 800, 519)
    in: 350 (357, 800, 349, 800, 351)
    ru: 377 (800, 800, 377, 377, 377)
    rue: 111 (115, 800, 111, 800, 111)
    kr: 164 (171, 165, 165, 164, 164)
    za: 433 (440, 433, 433, 433, 433)

    8
    Region Pinging Result: asia[92.38.183.13:5055]: 108ms
    eu: 282 (290, 282, 284, 282, 800)
    us: 231 (237, 231, 231, 232, 800)
    usw: 197 (800, 197, 197, 197, 197)
    cae: 289 (296, 289, 289, 289, 289)
    asia: 108 (105, 102, 103, 102, 127)
    jp: 142 (150, 142, 143, 143, 143)
    au: 284 (273, 800, 267, 267, 318)
    sa: 385 (800, 384, 386, 800, 386)
    in: 377 (800, 336, 800, 800, 418)
    ru: 384 (389, 800, 800, 382, 387)
    rue: 110 (114, 110, 110, 110, 110)
    kr: 186 (191, 186, 187, 800, 800)
    za: 436 (800, 437, 435, 436, 800)

    9
    Region Pinging Result: us[92.223.82.17:5055]: 242ms
    eu: 275 (282, 274, 274, 274, 279)
    us: 242 (241, 256, 249, 231, 233)
    usw: 197 (204, 197, 197, 800, 800)
    cae: 288 (800, 800, 288, 288, 288)
    asia: 238 (249, 240, 240, 800, 236)
    jp: 145 (154, 145, 147, 145, 145)
    au: 266 (800, 266, 268, 266, 265)
    sa: 570 (800, 800, 528, 800, 613)
    in: 342 (800, 800, 800, 800, 342)
    ru: 367 (374, 367, 368, 368, 800)
    rue: 110 (115, 110, 110, 800, 800)
    kr: 278 (800, 280, 280, 277, 277)
    za: 431 (436, 431, 431, 432, 431)

    10
    Region Pinging Result: rue[92.38.191.5:5055]: 110ms
    eu: 276 (280, 275, 275, 277, 278)
    us: 233 (239, 233, 233, 800, 800)
    usw: 189 (197, 800, 189, 189, 189)
    cae: 299 (292, 306, 297, 286, 307)
    asia: 185 (196, 185, 800, 186, 800)
    jp: 129 (800, 129, 800, 129, 130)
    au: 267 (274, 267, 800, 800, 267)
    sa: 384 (392, 385, 384, 384, 385)
    in: 336 (344, 800, 800, 336, 800)
    ru: 369 (372, 800, 371, 369, 368)
    rue: 110 (800, 110, 111, 110, 800)
    kr: 236 (800, 231, 800, 239, 240)
    za: 437 (440, 437, 437, 437, 437)
  • JohnTube
    JohnTube ✭✭✭✭✭
    Hi @Jebel,

    Thanks a lot, this is very useful!
  • Thanks for the effort you put into this.
    Result number 6 and 9 are totally puzzling. No idea how "us" gets selected with comparably worse ping.

    Aside from that, we see a lot of 800(ms) results, which are lost. This is to be expected from your location.

    Are you planning to launch the game in China Mainland? If so, you need to fix the region to CN and use a different Name Server.
    See "Using The Chinese Mainland Region".


  • Tobias
    Tobias admin
    edited February 2020
    Are those results all from 2019.3 Editor? Or a build? If a build: IL2CPP or Mono?
    Which exact version of Unity 2019.3 do you use?
  • @ Tobias,yes. The all logs are from 2019.3 Editor. but the build apk also connectes to one of asia and rue. I got logs from the unity editor for convenience .
  • Which exact version of Unity 2019.3 do you use?
  • Unity 2019.3.1f1
  • Hi,

    I am on the same setup ( mac, 2019.3.1f1) and I can't reproduce.

    Can you try with the little change, it implemented a delay for the connection. Can you tell us if that helps?

    https://dropbox.com/s/s0tx9ousm6c0zhd/Pun2DelayedConnectionToMaster.unitypackage?dl=1

    Thanks,

    Jean

  • Hey, Sorry for late response. I was busy with other things recently and come back now:). I have tested again on three android devices and Unity Editor with no any code change. They are work no any problem and they are stable connect to 'asia' region with low latency. Great Job!!! I will purchase your Photon Unity Networking 2 - PLUS Version for my network games. Thank you for your awesome product and kingly support.
  • Hi,

    Cool, I am glad it works. Make sure you leave a review, we always appreciate your feedback.

    Bye,

    Jean
  • JohnTube
    JohnTube ✭✭✭✭✭
    hey everyone,

    I want to announce that we now have a new feature: "Development Region".
  • This still happens to us in the middle east getting US as the best region. Funnily enough at the beginning the latency shows up as ~100 (which isn't geographically possible), but then updates to true values of ~400+. Are there any news on this issue?
  • @Idodi, we need some help, as we can't repro this case.

    First reset best region selection (either from Unity Editor or using code: ServerSettings.ResetBestRegionCodeInPreferences()).
    Then set logging level to INFO.
    When you connect you will get the result of the 5 ping time per region and not just the average per region.
    We need those pings for all the regions.

    Then we can have a look and maybe understand the case.
  • Idodi
    Idodi
    edited November 2020
    Hi Tobias, thank you for the quick reply.

    I'd like to elaborate on my case a little more since it can be relevant for the investigation.

    We're running our game in WebGL, and I believe that the issue could be related to that. I believe the first ping to every server returns incorrect information or perhaps this could be a Web Browserthrottling issue?

    This seems to happen only in some regions which should explain why we get a "bad" region sometimes! it's when the first ping is high enough to create an average ping that is incorrect.

    I'm adding a screenshot of the network traffic in Chrome

    https://ibb.co/WtTqMYX

  • The first ping result should get discarded. The average should not be affected by it.
    So .. I still would like to check the logs with full results, if anyone can repro this.
  • Idodi
    Idodi
    edited November 2020
    Is there any chance it's not getting discarded? perhaps specifically in WebGL? I'm not sure if this happened to us in the Editor.

    In any case, I'll try to reproduce this with the logs and keep you updated.
  • The logic should be the same in WebGL as on any other export. The platform dependent part is the actual pinging code.
  • Can you direct us to where the ping code is?
  • Was there any progress made on this issue? we're also experiencing an issue where players would sometimes connect to a region that is not even whitelisted (we whitelisted US East and Europe and users will occasionally connect to Russia).
  • @Idodi, which client are you using and which version is it? Update to the latest and report via mail, if the issue persist. Thanks.

    In general: WebGL clients have to use WSS and the pinging via that connection is not really effective. It can be off quite easily and so sometimes a bad choice is made.
  • @Tobias was this issue ever resolved? Our project was working fine until a week ago, now we're almost consistently auto connecting to South Africa while it should be connecting to EU.

    In the server settings it says Best Region Preference: 'az' ping:13ms after connecting, but in-game we're getting around 200 ping.

  • A few days ago, we had to redirect regions and got big DDOS attacks. Maybe the clients picked up AZ then.

    The clients only store the best region code, not the IPs, so it's odd the client should keep going to AZ. Also, the ping time is odd.

    Did you mail us to developer@photonengine.com? Send the AppId you are using and I will try to have a look.

    To resolve this, I would point to the Region Whitelist in the Dashboard. Clients should ping all regions again, if they figure the list changed.

  • MikoriGames
    edited December 2021

    Ping in the Europe server is higher now. Should i do something to fix it? Thanks