problem with Restarting application due to file changes

mindlube

The feature of Photon3 where it can restart automatically if you upload new .dlls is very convenient and I want to continue using it in my development and testing. However I am getting IOExceptions now. I have an embedded database which I open in Setup() and I commit/close in TearDown(). This works fine when I use PhotonControl to Start & stop the Instance service. But it throws IOException when automatically restarted by file changes. Guessing from the logfile, that the restarted process is launched *before* the TearDown() of the previous process

Here is a normal stop/restart by using the PhotonControl menu:

2012-06-16 11:33:18,783 [1] INFO  Photon.SocketServer.ApplicationBase [(null)]
2012-06-16 11:33:47,565 [1] DEBUG PhotonHostRuntime.PhotonDomainManager [(null)] - RequestStop
2012-06-16 11:33:47,565 [1] INFO  Photon.SocketServer.ApplicationBase [(null)] - Application is stopping: AppId=ChickieDominos3-dev
2012-06-16 11:33:47,612 [1] INFO  PhotonHostRuntime.PhotonDomainManager [(null)] - Stop
2012-06-16 11:33:47,612 [1] DEBUG Mindlube.ChickieDominos3.App [(null)] - TearDown() ; closing volante db...
2012-06-16 11:33:47,612 [1] INFO  Photon.SocketServer.ApplicationBase [(null)] - Application stop: AppId=ChickieDominos3-dev
2012-06-16 11:34:48,577 [4] DEBUG PhotonHostRuntime.PhotonDomainManager [(null)] - AssemblyLoadEvent: GeneratedSerializerAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
2012-06-16 11:34:48,608 [1] INFO  Mindlube.ChickieDominos3.App [(null)] - Created application instance: type=Mindlube.ChickieDominos3.App
2012-06-16 11:34:48,639 [1] INFO  Photon.SocketServer.ApplicationBase [(null)] - Application start: AppId=ChickieDominos3-dev; AppPath=C:\Photon\ExitGames-Photon-Server-SDK_v3-0-24-3243-RC9\deploy\ChickieDominos3-dev, Type=Mindlube.ChickieDominos3.App

IOExceptions occur only when the restart is because of file changes detected. It's odd that the Setup() occurs before the TearDown(), in the logfile anyways.

2012-06-16 11:35:23,115 [17] ERROR Photon.SocketServer.ApplicationBase [(null)] - System.IO.IOException: The process cannot access the file 'C:\Photon\ExitGames-Photon-Server-SDK_v3-0-24-3243-RC9\deploy\ChickieDominos3-dev\assets\volante.dbs' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
   at Volante.OsFile..ctor(String filePath, Boolean readOnly)
   at Volante.Impl.DatabaseImpl.Open(String filePath, Int32 cacheSizeInBytes)
   at Mindlube.ChickieDominos3.App.InitVolanteDb()
   at Mindlube.ChickieDominos3.App.Setup()
   at Photon.SocketServer.ApplicationBase.PhotonHostRuntimeInterfaces.IPhotonControl.OnPhotonRunning()
2012-06-16 11:35:23,162 [17] INFO  PhotonHostRuntime.PhotonDomainManager [(null)] - Stop
2012-06-16 11:35:23,162 [17] DEBUG Mindlube.ChickieDominos3.App [(null)] - TearDown() ; closing volante db...
2012-06-16 11:35:23,162 [17] ERROR Photon.SocketServer.ApplicationBase [(null)] - System.ArgumentNullException: Value cannot be null.
   at System.Threading.Monitor.Enter(Object obj)
   at Volante.Impl.DatabaseImpl.Commit()
   at Mindlube.ChickieDominos3.App.TearDown()
   at Photon.SocketServer.ApplicationBase.PhotonHostRuntimeInterfaces.IPhotonControl.OnStop()

Could this be a bug in Photon? Any suggestions? I suppose it's possible that Volante is doing something weird with Threading in it's Commit(). But Photon is maybe not letting it finish.
I just don't want to use the PhotonControl toolbar menu item to stop/start every time I build a new .dll! Thanks

dreamora

The problem points towards a problem with the file in use ie an open file handle / db access that was not properly released when th teardown function got called inside of photon as it should have been.

mindlube

But it works fine when I turn off the service manually or with Photon Control! I surmise that TearDown() is called *differently* when the server restarts for file changes vs. when the service is stopped manually. What is the difference in how Photon is calling TearDown()? Different context or different rules about threading? The API docs simply say
"This method is called when the current application is being stopped. The inheritor can execute cleanup routines here."
Which is exactly what I'm trying to do...

dreamora

When you shut down the service then any running code, including the one holding the lock to the file, are terminated.
Upon restart, the socket server itself remains alive and so does the .NET 'script layer', its only the application domains that get refreshed (Tobias might have more indepth technical informations here on what happens in detail on the auto refresh) so if there is code in volate that somehow survives that refresh it will keep the lock.

mindlube

It is indeed a mystery so if anyone from exitgames has any advice I would like to hear it. Thanks

Tobias

My first impression is that TearDown should not be called differently per situation, so this is something we will check.

There are several possible settings for the shutdown: EnableShadowCopy, EnableAutoRestart and ForceAutoRestart.
Our configs use EnableAutoRestart and it's describes as: "existing connections to the old copy are allowed to continue until all connections disconnect at which point the app domain is unloaded."
Maybe this is causing the issue. You could try with ForceAutoRestart instead.

Our server guy is currently on vacation and these things usually take a few days.
Sorry for the inconvenience but we're checking things out soon.

mindlube

Tobias, for the suggestion, but it still fails with the same error. Here is my .config entry now:

<Application
	  Name="ChickieDominos3-prod"
	  BaseDirectory="ChickieDominos3-prod"
	  Assembly="CD3PhotonServer"
	  Type="Mindlube.ChickieDominos3.App"
	  ForceAutoRestart="true"
	  WatchFiles="dll;config"
	  ExcludeFiles="log4net.config">
      </Application>

Tobias

We will take a look at the order of stopping and starting new application domains.
It's unfortunate that the lock is exclusive in this case.

Allow us some days to evaluate the options.

mindlube

I'll be curious to see what you come up with! It's not stopping me from working, it's just a convenience & productivity thing. If you need my source code for my server just let me know. Thanks

BenStahl

On a restart the running instance will not be closed before the new instance has been successfully started.
So Setup in the new instance will be called before TearDown in the currently running instance, causing the file access problem.

You have to write some synchronization mechanism to detect if the file is not used any longer.
For example this can be done using a mutex or named pipes for inter process communication.

Here an example using named pipes.

private FileStream dbFile;
private System.IO.Pipes.NamedPipeServerStream serverStream;
		
protected override void Setup()
{
	// start a new thread, otherwise photon application start will be blocked
	var thread = new Thread(this.InitializeDatabase);
	thread.Start();
}

private void InitializeDatabase()
{
	try
	{
		var client = new System.IO.Pipes.NamedPipeClientStream("MyPipe");
		
		try
		{
			// try to connect to named pipe
			client.Connect(100);
		}
		catch (Exception ex)
		{
			// client cannot connect
			// no server instance is running
		}

		// if client is connected there is allready an instance running
		if (client.IsConnected)
		{
			// read a byte from the named pipe will block until the 
			// the running instance is disposing the server stream
			client.ReadByte();
		}

		client.Dispose();

		this.serverStream = new System.IO.Pipes.NamedPipeServerStream("MyPipe");
		this.dbFile = File.Open("C:\\Temp\\DataBase.txt", FileMode.OpenOrCreate, FileAccess.ReadWrite);
	}
	catch (Exception ex)
	{
		log.Error(ex);
	}
}

protected override void TearDown()
{
	// release the database file
	this.dbFile.Close();
	this.dbFile.Dispose();

	if (this.serverStream != null)
	{
		this.serverStream.Dispose();
	}
}