About protocol, custom types and same data for client/server

devast

Good day.

While developing with photon is rather nice and saves a lot of time, I found one thing to be somehow boring and tedious task: creating new operations. Let's count how many classes one must update/create to introduce a new operation:

NOTE: I will explain whole post in terms of photon mmo example

How it is going at the moment:
1. edit OperationCode, ParameterCode enums
2. add new class for operation on server side
3. on client side one have to convert operation parameters to Hashtable when sending request and convert operation from Hashtable when recieving response. it's very easy to make a mistake here and get an error because server/client side will try to parse different data (for instance, send { (byte)ParameterCode.Par1, (int)number } on server however on client try to parse (short)number instead of int). moreover to keep code syncronized on sever and client you must change several classes on different sides.
4. also often it's inconvenient to keep data on client side as hashtable, consequently new classes are added, which more or less look like their server counterparts (e.g. I have an array of room information on server, and I want to send that to the client, everything is converterted from server array to array of hastables to fit photon protocol, then sent to the client, recieved by client, hashtable converted from protocol format to client format).

While step 1 couldn't be avoided, steps 2, 3 and 4 bothered me for a long, long time. What if I could make some shared classes between client and server and be able to automatically serialize/deserialize operations? That would be nice!

So instead of just filling a request form, I spent a week to make this possible. Jumping ahead I'll show what can be done right now after my addition:
1. same as before
2. add ONE shared class of operation on client and server, as previous mark class properties with Request/Response attributes (although I've rewritten those attributes), put it in the Common library
3. instead of using meaningless hastables on operation level, define custom types (explained later), provide conversion to and from hashtable in ONE class in two different methods.
4. that's all! rest of the work is done automatically!

Let's have a look at test operation I made

	public class TestCustomType : OperationBase
	{
		[RequestParameter((ParameterCode)100)]
		[ResponseParameter((ParameterCode)100)]
		public int IntPar { get; set; }

		[RequestParameter((ParameterCode)101)]
		[ResponseParameter((ParameterCode)101)]
		public float[] ByteArrayPar { get; set; }

		[RequestParameter((ParameterCode)102)]
		[ResponseParameter((ParameterCode)102)]
		public SubType SubTypePar { get; set; }

		[RequestParameter((ParameterCode)103)]
		[ResponseParameter((ParameterCode)103)]
		public Hashtable Hash { get; set; }

		[RequestParameter((ParameterCode)104)]
		[ResponseParameter((ParameterCode)104)]
		public SubType[] SubArr { get; set; }
	}

Here you see some unknown class SubType, this is custom type. To make this type be serializable/deserializable, few steps should be provided.

Firstly, here is the class itself:

	[CustomType(CustomType.Test)]
	public class SubType
	{
		public string Str { get; set; }

		public static Hashtable ToHashtable(object obj)
		{
			SubType instance = (SubType)obj;
			return new Hashtable()
			{
				{ (byte)0, instance.Str}
			};
		}
		public static object FromHashtable(Hashtable hashtable)
		{
			return new SubType
			{
				Str = (string)hashtable[(byte)0]
			};
		}
	}

Obviously custom types must be converted in the form photon protocol will understand. 2 steps are needed to make this happen: change client protocol, accordingly change server protocol.

I'll start with client side, as PhotonUnity3d.dll is not obfuscated and could be decompiled (while all server code is protected for unknown for me reasons, this only makes learning photon alot harder). At first, I added the following class inside client dll to register custom types:

namespace ExitGames.Client
{
	using System;
	using System.Collections;
	using System.Collections.Generic;

	// Register custom types here
	public static class CustomTypeFactory
	{
		static Dictionary<Type, KeyValuePair<byte, Func<object, Hashtable>>> serializers;
		static Dictionary<byte, Func<Hashtable, object>> constructors;

		static CustomTypeFactory()
		{
			serializers = new Dictionary<Type, KeyValuePair<byte, Func<object, Hashtable>>>();
			constructors = new Dictionary<byte, Func<Hashtable, object>>();
		}

		// Using System.Type instead of client interface or attribute because input type 
		// could be defined in other library (e.g. common library between client and server,
		// which doesn't have client dll referenced).
		public static void RegisterType(Type customType, byte code, Func<object, Hashtable> serializeMethod, Func<Hashtable, object> constructor)
		{
			serializers.Add(customType, new KeyValuePair<byte, Func<object, Hashtable>>(code, serializeMethod));
			constructors.Add(code, constructor);
		}

		public static bool TryGetSerializer(Type type, out byte code, out Func<object, Hashtable> serializer)
		{
			KeyValuePair<byte, Func<object, Hashtable>> result;
			if (serializers.TryGetValue(type, out result))
			{
				code = result.Key;
				serializer = result.Value;
				return true;
			}
			serializer = null;
			code = 0;
			return false;
		}

		public static bool TryGetConstructor(byte code, out Func<Hashtable, object> constructor)
		{
			return constructors.TryGetValue(code, out constructor);
		}
	}
}

Its only purpose is to register custom types and get appropriate serialize/deserialize methods. All custom types are converted into hashtables, although those hashtables contain an additional element { byte, byte } to signal that hashtable is actually not just a hashtable. Key byte is flag (I named it ParameterCode.CustomTypeFlag = 255) and value byte is actually custom type id (look at the attribute above SubType)

[CustomType(CustomType.Test)]
public class SubType
...

Then I changed the protocol itself, so it can call those 2 methods for custom types (photon guys can PM me, so I can send VS solution with comments and bookmarks). Overall idea is rather simple: on serialization detect custom types and call serializer, on deserialization call constructor. I couldn't change protocol flags (because server side is locked). Every custom type instance contains 2 additional bytes as overhead when converted into hashtable. And there is a problem with arrays of custom types, i.e. arrays of hashtables from protocol point of view, I can't really tell whether array type is usual hashtable or some custom type when array is empty (I'm using same method as photon uses to determine type of array of arrays that is read first array element which obviously may not exist).

Then I wrote operation serialization via parameter attributes (same as on server side for Operation class). I read attributes and then using reflection feed parameters to the protocol when serializing (revert the process for deserialization).

One more thing to be added. Client should call CustomTypeFactory.Register method for every custom type it uses, otherwise conversion will fail.

And this is it for the client side. Let's move on to the server side.

To keep server and client dlls as much independent as possible, I had to copy same CustomTypeFactory class to the server project, as server needs to know about custom types as well. Also don't forget to register same types with same number. Here (and on client side too) I've used one auxiliary class which makes registering easy and and error free.

using System;
	using System.Collections;
	using System.Reflection;

	public static class CustomTypeUtils
	{
		static Type serializerType;
		static Type constructorType;

		static CustomTypeUtils()
		{
			serializerType = typeof(Func<object, Hashtable>);
			constructorType = typeof(Func<Hashtable, object>);
		}

		//TODO: кешировать запрос!
		public static bool GetId(Type customType, out byte id)
		{
			object[] attributes = customType.GetCustomAttributes(typeof(CustomTypeAttribute), true);
			if (attributes.Length != 0)
			{
				CustomTypeAttribute attribute = (CustomTypeAttribute)attributes[0];
				id = attribute.Code;
				return true;
			}
			id = 0;
			return false;
		}

		public static Func<object, Hashtable> GetSerializer(Type customType)
		{
			MethodInfo mi = customType.GetMethod("ToHashtable");
			if (mi != null)
			{
				return (Func<object, Hashtable>)Delegate.CreateDelegate(serializerType, mi);
			}
			return null;
		}

		public static Func<Hashtable, object> GetConstructor(Type customType)
		{
			MethodInfo mi = customType.GetMethod("FromHashtable");
			if (mi != null)
			{
				return (Func<Hashtable, object>)Delegate.CreateDelegate(constructorType, mi);
			}
			return null;
		}
	}

Then i just called registering of all custom types from Common library using reflection. So one can forget about bustling with custom types completely.

Let's return to the protocol. As said above: all server dlls are obfuscated and thus CANNOT be changed. So I had no other option as to add an additional recursive check to operation parameters. On receiving of a request, server deserialization is called looking for any hashtable inside parameters and checking if that hashtable contains CustomTypeFlag, if so it calls constructor if the type is registered. Obviously on sending response operation parameters must be checked for content of custom types and call an appropriate serialization. (Notice I discarded standard Operation class on server side and read request/create response on low level.) This way adds heavy overhead on server side at the moment, as there are now two steps to process every operation either on request or response. But the only way, as server protocol is protected.

That's all for server side as well. Let's make an overview of everything I've said.

Recall test operation and SubType class in the beginning of the post. On the client side now I can write the following code:

		TestCustomType par = operationParameters.TestCustomType;
		par.IntPar = 15;
		par.ByteArrayPar = new float[] { 1, 5, 9 };
		par.SubTypePar = new SubType() { Str = "iamstring" };
		par.Hash = new System.Collections.Hashtable();
		par.SubArr = new SubType[]
		{
			new SubType() { Str = "s1" },
			new SubType() { Str = "s99" }
		};

And then call the operation.

On the server side:

			TestCustomType operation = mmoPeer.OperationParameters.TestCustomType;
			string error;
			if ((error = operation.ReadRequest(request)) != null)
			{
				return operation.CreateErrorResponse(request, ErrorCode.InvalidOperationParameter, error);
			}

			TestCustomType obj = operation;
			log.Debug(obj.IntPar);
			log.Debug(obj.ByteArrayPar.Length);
			log.Debug(obj.SubTypePar.Str == null);
			log.Debug(obj.SubTypePar.Str);
			log.Debug(obj.Hash);
			log.Debug(obj.SubArr == null);
			if (obj.SubArr != null)
				foreach (SubType sub in obj.SubArr) log.Debug(sub.Str);

			return operation.CreateResponse(request);

Code is compact and clear. No manual conversions are to be made. We can use same classes on a client and a server, which obviously makes operation addition even easier. As the programmer in relatively small team I look for any possibility to decrease developing time and time required for code editing and support.

So, what I really want from photon team is to add serialization of custom types on the protocol level. Making shared classes and automatic parameters serialization on both sides will also be a nice feature, although my code works above standard photon functionality and doesn't require to be included into the standard library.

One more thing I want to add in conclusion. I predict in the future I'll need to change the protocol itself, that is the way how data is serialized/deserialized, so one more request is to extract the protocol and be able substitute custom protocol into the library. (Maybe an interface or base class.)

Thanks for all who read this HUGE post (if there are any ). Hope to see an answer from photon guys.
See you!

P.S. Custom class could also be serialized automatically if Properties are marked with attributes providing byte codes. Still haven't decided which way is better.

P.P.S. I plan to add same functionality for events, so add event automatic conversion on request sheet as well!

Tobias

Thanks for sharing this.
We are looking into serialization for custom (any) object type but we're not there yet.

KEMBL

Tobias wrote:

Thanks for sharing this.
We are looking into serialization for custom (any) object type but we're not there yet.

Take a look on Devast idea closer, this approach will very useful and already looks solid.