Sending large blocks of data with a stream system - is Photon throttling transfer rate ?


Hey there,

We have an app in development which is a collaborative 3D design tool in VR and I have been working on the network side for more than 6 months now using Photon.

We use PhotonRealtime and PhotonVoice and both have a 100 CCU plan.
Right now we are in Alpha and we are very far from reaching the limits, I'm mostly doing tests locally with 2 computers and few tests with the rest of the team all over the world.

So we use Photon to send the following messages :
- Avatar states (very little data, 15 times per second)
- Modelling data (i will not enter in detail here, but we are doing simultaneous multi user editing, so it can get a bit complex, but the data is very small too, messages also sent at 15 times per second)
- 3D and texture resource upload/download, and this can be big data (several MB)

So for our resource system we basically have users than can bring in resources from their local drive and all users will eventually receive them.
Of course we are aware that Photon might not be very well suited for sending large amount of data, and that synchronization from the Cloud might be a better solution, but still, we are trying to push as much as we can using Photon in the light of trying to get our product to the market quickly.

So here is the problem.

We have developped a streaming system, and basically what it is doing is splitting a chunk of data into slices, and sending these slices over, while making sure that not more that X amount of data is in flight (using an reception acknowledgment mechanism), so we dont reach the limit on the server client buffer.

And this seems to be working well, but the transfer rates are extremely slow, and here are the finding :


- Splitting a chunk in slices of 1KB : quite a reliable transfer, upload speed 10KB/second
- Splitting a chunk in slices of 10KB : slices sometimes take more than 10 sec to arrive
I tried all settings from slices of 1KB to slices of 10KB, with a flight windows of 50KB to 400KB, and in all cases i can never reach an upload rate of more than about 10KB/second (on top of what we already send, which is very small), and as soon as the slide goes above 1KB. they can take ages to be transferred over.

Of course there could be a bug in our system, and something i dont see at the moment, but is what i try to describe an expected behaviour ?

I must mention that we use 4 SequenceChannels if that makes any difference.

All the best,
Pierre

Comments

  • Hi @PierreBlanc.

    Yes, this is kind of expected. Photon is not a file server and simply not written for transferring files.

    Please see https://doc.photonengine.com/en-us/realtime/current/troubleshooting/faq#can_i_send_a_huge_message_using_photon_ for further information.

    That said, from my experience sending messages with several MB in size over Photon works far better when using TCP than when using UDP, which you currently seem to be doing.
  • Hi Kaiserludi,

    Thanks for the prompt reply. However the page of the doc you mentioned does mention "get in touch with us if you really have to send a very big message"

    Im not sure if TCP is an option, I have been reading your forums for the past months and it seems that TCP is not really a viable backend with Photon.
  • Hi @PierreBlanc .

    What do you mean with "not a viable backend with Photon"?
    For Realtime data UDP usually is a better choice than TCP, but that is independent of Photon and would be just as true without Photon. TCP however might still be good enough. If that is the case for your app depends a lot on your specific app. You could just give it a try, switch the protocol and see how it works out.
    For file streaming TCP is usually a much better choice than UDP, but again, that is also the case without Photon being used.
    So ideally you would send your first two types of data over a Photon Realtime UDP connection, the Voice data over a Photon Voice UDP connection, and those multiple MB files over a HTTP or FTP Client library through a dedicated file server.
    The best option with Photon only - without a separate HTTP or FTP client - would be to open a third Photon connection, but let that connection use TCP. This connection would ideally be to a self-hosted Photon server, for which you could configure your server side buffer to something big enough for even the biggest size that you multiple MB uploads can ever become. I know of a Photon server customer who did exactly that for storing save games and it worked very well for them with 20MB save games.
    You could of course also have that Photon TCP connection be a second connection to Photon Realtime, but then you would be limited to chunks of 500kb and would need to give the servers enough time to empty their buffers before sending the next chunk from the same client. Also by sending this kind of data over Photon Cloud you might end up with considerable traffic over-usage costs.
  • PierreBlanc
    edited April 2019
    I was referring to some earlier reading of the forums where TCP was mentionned as a fallback that is not guarrented to work as expected with all features of Photon. But i may have misunderstood.

    When I enable TCP instead of UDP (but for my single connection, i didnt try have another client), the upload rate increases by almost 10x, but the sender is spending a large amount of time in SendOutgoingCommands, which leads to very few fps in the app, and no other data is sent (such as my avatar updated). So i guess it doesnt use the Sequencing channels in the same way as over a UDP connection.

    It makes sense since it is a single connection and i tried to queue 400KB of data to be sent.

    With the UDP approach i was just sending outgoing commands as fast as I could, every frame (90FPS) and that didnt seem to be a problem on all platforms (windows/IOS/android). In this case however i count on the fact that lower sequence channel number have priority over large ones, if i get it right but it seems to be the case since everything works as expected, except it is very slow.
  • Hi @PierreBlanc.

    Channels are one of the Photon features that are not supported with TCP.

    Hence when you queue a lot of low-prio data, then your later queued high-prio data will only be processed once all the previously-queued data has been processed.
    However that would no longer be relevant, once you use a separate secondary connection for the upload-data anyway.
    I don't know why the frame rate would drop that much. I am not really too familiar with the .NET client, though, so a colleague might be able to tell more about this.
  • OK, I have just checked back with a colleague and apparently the .NET client is using a blocking call to the socket. My assumption previously has been that it would use an asynchronous call just like the C++ client.

    So you would need to run your separate TCP connection on a different thread to prevent it from blocking the main thread with the UDP connection and the UI.
  • Ok super thats a lot of information, thank you very much
  • A late but important bit of input for you: If you use the Photon Cloud, all plans come with a free traffic amount of 3 GB per plan CCU. The 100 CCU plan includes 300 GB traffic at the moment. Additional fees apply for each month and region the free amount is exceeded.
    If you host Photon yourself, this does not apply, of course.

    Hope it's working well now?!