Deterministic Prototyping in Unity DOTs

I wanted to do some prototyping with time-manipulating mechanics in a casual RTS project. I always found the gameplay and technical implementation details of games that have experimented in this area (such as Braid and Achron) very interesting. To begin with I’m curious to see what it would be like to allow players to see the future and also have the ability to rewind back to the past. I thought it would be fun to figure out the technical implementation and was curious if it would make for interesting gameplay too.

Past and Future

Rewinding to past states can be accomplished with recording game state. This could be similar to the method described in Braid, though I’m not sure yet if I’d want all the frames up to the rewind time or just a single snapshot of the world at the target time in the past.

To know the ‘current future’ (the future assuming no new player inputs) I plan to run a second version of the game simulation ahead of the present. For the future to be true the simulation must be deterministic. I’ve made a deterministic RTS in the past using fixed-point maths but Unity’s new DOTs (data-orientated-technology-stack) comes with a Burst compiler that is planning to support cross-platform deterministic floating point maths eventually. In addition it has multi-threading job and entity component systems that have got a lot better over the last two years. It seemed like a good fit to start using these to write my simulation code in. I did also dabble in writing a native plugin in C++ but in the end it became apparent I was going to spend a great deal of time recreating things that Unity can adequately do. The burst compiler doesn’t give cross-platform determinism yet but Unity ECS, and DOTS Physics are deterministic on the same CPU architectures. This is enough to start prototyping with for now.

Simulation & Presentation

I setup a basic casual ClashRoyale style RTS with debug rendering. The game battle code is split between the simulation, which is the battle logic, and the presentation, which is the rendering of the simulation state. The simulation can be stepped multiple times independently of a presentation update. This is useful when running multiple worlds (such as a predicted future world in this case) where you may not want to render after every update. It’s also a great debug feature. To display the future I plan to run a second simulation world 7 seconds ahead of the present time. The first task is to ensure that the simulation is deterministic so that the future is true.

A reasonably busy test battle scenario was setup and run multiple times to start testing determinism. It was quickly obvious that the sim wasn’t deterministic at all, with units in different positions & health states across multiple runs. To address this, checksum hash calculations were added for key data such as unit positions, health and locomotive forces in order to diagnose where determinism was lost.

The checksum calculations are logged to a file at the end of the sim update. Each checksum entry has the value and a description which is usually the file and line number. After a few checksum calculations were added the determinism break was found to come from some multi-threaded jobs in the locomotion & health system. Realising that Unity’s NativeMultiHashmap.Concurrent is not deterministic; I removed the use of ScheduleParallel calls involving this container (it wasn’t especially necessary anyway). This lead to identical checksum values on every run. Having the ability to step the sim up to and over 1000 updates in approximately one second (this skips rendering and updates the sim in a loop from a single FixedUpdate call), really helped increase iteration time.

Implementation Details

To have this level of control of the simulation and presentation systems, ICustomBootstrap was used to override Unitys ECS world creation. All the standard Unity systems are created but not added to the automatic player update loop. Game systems are not added to any of the Unity system groups or the player loop; I manually update the game systems inline as shown below.

int logicFramesToProcess = m_paused ? m_singleStepCount : 1;

if (logicFramesToProcess > 0)
{
   for (int step = 0; step < logicFramesToProcess; ++step)
   {
    StepSimLogic ();

    int simStepsProcessed = m_simStepCount - 1;
    Checksum.FlushToFile (simStepsProcessed);
  }

  StepViewLogic ();
}

void StepSimLogic ()
{
  // Unity initialisation systems
  m_world.GetExistingSystem<InitializationSystemGroup> ().Update ();

  // Update battle simulation systems here
  m_world.GetExistingSystem<Sim.ExportPhysicsEventsSystem> ().Update ();
  m_world.GetExistingSystem<Sim.AttackTargetSeekSystem> ().Update ();
  m_world.GetExistingSystem<Sim.LocomotionSystem> ().Update ();
  ...

  // Unity simulation systems
  // + Sim.ExportPhysicsEventsSystem
  m_world.GetExistingSystem<SimulationSystemGroup> ().Update ();

  SimRunningTime += m_simDeltaTime;
  m_simStepCount += 1;
}

void StepViewLogic ()
{
  // Update battle presentation systems here
  m_world.GetExistingSystem<View.UIInputProcessor> ().Update ();
  m_world.GetExistingSystem<View.WorldToScreenSystem> ().Update ();
  m_world.GetExistingSystem<View.ProjectileRenderCreator> ().Update ();
  m_world.GetExistingSystem<View.LinearMovementSystem> ().Update ();
  ...

  // Unity presentation systems
  m_world.GetExistingSystem<PresentationSystemGroup> ().Update ();
}

For now I’ve preferred to update most of my own systems manually inline rather than using Unitys [UpdateBefore(systemA),UpdateAfter (systemB)] attributes. However, it is important to note that there is a CollisionEvents system that must be updated after StepPhysicsWorld and before EndFramePhysicsSystem. CollisionEvents is therefore added to the standard SimulationSystemGroup so its update is coordinated correctly with the Unity.Physics systems.

ICustomBootstrap is overridden to store sim, view or unity systems in lists and create all worlds required. Note there is no call for ScriptBehaviourUpdateOrder.UpdatePlayerLoop(world); because I want to manually control system update order and timing.

public bool Initialize (string defaultWorldName)
{
  Debug.Assert (s_instance == null); 
  s_instance = this;

  Debug.Log ("BattleManager.ICustomBootstrap.Initialize");

  IReadOnlyList<Type> systems = DefaultWorldInitialization.GetAllSystems (WorldSystemFilterFlags.Default);
  List<Type> simGameSystems = new List<Type> ();
  List<Type> viewGameSystems = new List<Type> ();
  List<Type> unitySystems = new List<Type> ();

  for (int i = 0; i < systems.Count; i ++)
  {
    if (systems [i].Namespace == "Sim")
    {
      simGameSystems.Add (systems [i]);
      continue;
    }
    else if (systems [i].Namespace == "View")
    {
      viewGameSystems.Add (systems [i]);
      continue;
    }

    unitySystems.Add (systems [i]);
  }

  //
  // Make an exception for ExportPhysicsEventsSystem as it must be updated
  // between various Unity.Physics.Systems

  simGameSystems.Remove (typeof (Sim.ExportPhysicsEventsSystem));
  unitySystems.Add (typeof (Sim.ExportPhysicsEventsSystem));

  //
  // Initialise worlds

  m_presentWorld = new WorldController ();
  m_presentWorld.Initialise ("sim", simGameSystems, viewGameSystems, unitySystems);

  World.DefaultGameObjectInjectionWorld = m_presentWorld.m_simWorld;

  m_alternateWorld = new WorldController ();
  m_alternateWorld.Initialise ("past", simGameSystems, viewGameSystems, unitySystems);

  const bool defaultWorldInitialisationDone = true;
  return defaultWorldInitialisationDone;
}

Later in the initialisation flow a single update of m_presentWorld is done before its data is copied. m_presentWorld was set to the DefaultGameObjectInjectionWorld so it contains all the converted GameObject data. Copying the data over gives both worlds the exact same starting state.

//
// Update present world once for game object conversion systems and then
// copy all entities into the alternate world.

m_presentWorld.m_simWorld.Update ();
MemoryBinaryWriter memoryWriter = new MemoryBinaryWriter ();
SerializeUtility.SerializeWorld (m_presentWorld.m_simEntityManager, memoryWriter, out object [] unityObjects);

unsafe
{
  MemoryBinaryReader memoryReader = new MemoryBinaryReader (memoryWriter.Data);
  ExclusiveEntityTransaction transaction = m_alternateWorld.m_simEntityManager.BeginExclusiveEntityTransaction ();
  SerializeUtility.DeserializeWorld (transaction, memoryReader, unityObjects);
  m_alternateWorld.m_simEntityManager.EndExclusiveEntityTransaction ();
}