Good news! I’ve been invited to speak at the Montreal International Game Summit 2009
I haven’t decided on a name for the talk yet, but it will focus on networked physics and combine material from my GDC 2009 tutorial, lecture and new material I’m going to develop over the next few months.
The basic overview of the talk is something like this:
- How the internet works and why we should use UDP
- Bandwidth considerations, packet structure, 64kbit/sec
- Describe basic algorithm for synchronization: push/pull/serialize with visual smoothing
- Synchronize one server controlled cube, client is just a viewer of simulation
- Synchronize two player controlled cubes, client and server cube.
- Effects of latency, client side prediction
- High simulation cost for rewind and replay, client side prediction breaks down in a dynamic world
- Dynamic world synchronization: options – client side prediction, no prediction, styrofoam, authority scheme
- Fiedler’s Cubes demo showing authority scheme to avoid high cost
- Corrections, reverse corrections
- Compression techniques
- Cool demo to go out
- Extended Q&A
The idea is to pretty much combine all the talks I’ve done so far on networked physics and present the information in a clear, simple manner – anybody who comes to this talk will leave with all the knowledge they need to network a physics simulation.
And of course, I get to spend a weekend in Montreal with my sweetheart about 2 months after we get married
So you see, it’s win-win folks 
{ 17 comments… read them below or add one }
Aww… I wish Montreal wasn’t so far from here
Sounds like a great talk!
I’m currently implementing a networking system based on the authority scheme for a hobby project. Would be fun to compare your stable implementation with what I’ve cooked up (I haven’t been doing networking programming for long, so I don’t even have my own reliable UDP code and use ENet instead). It’s mostly doing fine, although PhysX is quite problematic at times. For one, the function to access collision islands crashes with a null pointer read sometimes (which I side-step by catching and ignoring access violations). I’ve tried contact reports earlier, but they weren’t reliable when hard-snapping object positions. But then, collision islands aren’t perfect either – the reported islands have a lag of one step. As a result I use both islands and contact reports, yet none of these tell me about collisions between kinematic character controllers
I guess Havok is a bit more sane in these areas?
If I could come to your talk, I’d probably be most interested in the ‘Corrections, reverse corrections’ part. It’s the one that’s causing me the most troubles… When the server assumes authority over an object, the client doesn’t use the server’s states directly. Instead, I emulate input replay by applying deltas between the ahead-of-time client-side states (that were rendered invalid by an authoritative state snapshot) to the authoritative state. This is quite easy with player controllers, since they are kinematic actors and precise sweeps with immediate feedback are possible with them, though it gets rather tricky with dynamic actors. When doing it with tank models (2 actors for the hull and turret, 10 actors and joints for wheels), I’m getting ‘rippling’ effects in the actor motions and so far the only way I’ve found to prevent them was to dampen the state deltas slightly… yet it causes the local inputs to feel a bit lagged.
Sorry for the rant, but perhaps you’ll use it to slightly bias the presentation, given a sample of what issues newbies might encounter. In any case, have fun!
BTW, thanks a lot for the awesome articles, slides and replies, they’ve helped me a lot, probably more than the Unreal or Source networking articles (heck, I’ve fixed my timestep thanks to you!). I’d tried doing networked physics with time rollback a while ago, but it turned out too costly. This led me to think of other ideas and the main resolution was to give some authority to the client. It wasn’t until I read your posts about the authority scheme that I managed to clean it up and put up a working implementation
thanks, glad to help you out
could you explain this part in more detail, i don’t know what you mean here:
“When the server assumes authority over an object, the client doesn’t use the server’s states directly. Instead, I emulate input replay by applying deltas between the ahead-of-time client-side states (that were rendered invalid by an authoritative state snapshot) to the authoritative state. This is quite easy with player controllers, since they are kinematic actors and precise sweeps with immediate feedback are possible with them, though it gets rather tricky with dynamic actors. When doing it with tank models (2 actors for the hull and turret, 10 actors and joints for wheels), I’m getting ‘rippling’ effects in the actor motions and so far the only way I’ve found to prevent them was to dampen the state deltas slightly… yet it causes the local inputs to feel a bit lagged.”
me, i just snap the physics state when i receive an authoritative update from another player, and apply a visual smoothing for position and orientation on top – what exactly do you do in this case?
cheers
The client simulates ahead of the server, so when the authoritative state arrives, it’s in the past. Snapping to it means that you ignore any changes that happened to the object on the client side and ‘travel back in time’. The previous ‘time rollback’ technique compensated for it by catching up with more simulation steps. Say, the client receives an authoritative state T.auth for tick T, but it’s already simulated ticks T_1 and T_2.
* Kinematic actors:
The authoritative state T.auth is applied by hard-snapping. Then for each tick ahead of the server, the difference from the previous state is applied as well. So in this case (T_1 – T) and (T_2 – T_1). This is done by sweeping the controller using the continuous collision detection facilities of PhysX. The queue of past states for the object is set to the ones computed by the sweep, so states T_1 and T_2 will already be adjusted to the server’s state when another authoritative state comes in. The end result is that if the player is moving forward, and gets pushed to the side by another player, he won’t be snapped to the past state, but will rather see its current state changed by the past event.
* Dynamic actors:
The situation is similar, but sweeping is not possible, so the T_1 and T_2 states are guessed to be T_1 = (T.auth + (T_1 – T) * (1 – epsilon)) and T_2 = (T_1 + (T_2 – T_1) * (1 – epsilon)). The state of the actor is snapped to T_2.
I apply visual smoothing on top of that. The ‘rippling’ effect I mentioned was when not using the (1-epsilon) scaling of state deltas. It might be a bug in my code, but I think this is due to how forces manifest themselves with snapping in PhysX…
ah ok, so you are predicting ahead in some way to bring the latent time stream up to present time?
i don’t do this anymore, mostly because for my objects they are usually tumbling or stacked cubes, and i can’t really do it without running the whole simulation up for a second – it’s too expensive
i guess you are using some sort of linear sweeping to approximate the rewind and replay step yeah?
cheers
Hey,
Mentioned your great blog at my site =) Hope you dont mind.
thanks!
Hehe, I see how this might be problematic for lots of cubes, but for these I’d hard-snap as well. I only do the trick for objects actually owned by the player – i.e. the controller or a vehicle it’s in. The objects assigned to the client due to auth/touch tracking are considered temporarily owned and get none of the fancy.
). Still, when the correction is ‘large’ within some epsilon (and possibly some sweep-based heuristics in the future), it reverts to a hard-snap.
For the player controllers, I guess you could say it’s a linear sweep, though it’s actually done the same way normal simulation commences for them. It’s the move() function of the controller as provided by PhysX – along with response and sliding. It actually works pretty well. With snapping only, when one controller walked onto a stationary one owned by another player, the server would take over and you’d get a snap. In this case there’s no snapping at all. I guess the server should not take objects over based on them only touching but on actual state mismatches instead, but you’d get these with some auxiliary object between two controllers anyway.
For dynamic objects (only a tank model atm), it’s simply attempting to be a slightly better guess. Snapping and then optimistically applying deltas seems to yield better results than just snapping it to the past state. It might produce some erroneous results in some cases. In an extreme one, the object could phase through a wall (but I’d blame quantum tunneling
got it, sounds like a good half-way point for objects with mostly linear motion – how do you handle the jitter over the network, do you have any jitter buffering or time synchronization or do you timestamp and sweep ahead to offset time differences, hold in a jitter buffer before presenting to the sim, or just discard everything but most recent state?
cheers
I’m afraid I haven’t given much thought or work to this part… I currently do time synchronization and some queuing. Clients try to run ahead of the server by the uni-directional ping (well, client-server tick offset) plus 3 standard deviations of it (in a moving window of the last 2 seconds, then smoothed a bit). When the server receives data that’s supposed to be used in a future tick, it waits in a queue. Clients just apply any data that arrives from the server, although they catch-up (sim a lot) when it’s for a future tick – I still have the olde roll-back catch-up functionality there
got it, that works well for client/server – i’m working more peer-to-peer these days so there is no concept of relative time, each peer has his own time stream, i use the same std. deviation type buffering (adaptive jitter buffer) but i don’t call it time synchronization anymore, i’m just trying to get smooth playback, i let each peer maintain their own concept of time, within limits
yes, interaction between different time streams are undefined. but, they are already undefined given that we predict on each peer, can anything be more undefined than undefined?
anyway, it seems you have the client/server approach down pretty well, so congrats!
Thanks! Unfortunately I may be stuck with the client/server model since I’m targeting PCs with player-vs-player gameplay. It’s going to be tricky making the authority scheme relatively secure even without peer-to-peer
If it doesn’t scale well enough, perhaps something special could be done for ‘team deathmatch’ since well, it’s a two sided co-op mode. I’m basically trying to make a Battlefield-like game in my spare time, although there’s going to be a second project using the same net code for Quake-like gameplay.
I think I see what you mean with the adaptive jitter buffer. Sounds like a cool trick
And I guess interacting time streams aren’t really much different from what happens in the client-server model when two players collide – you’re bound to have some undefined behavior anyway.
The peer-to-peer model seems more or less clear now (except there are always lots of tiny and not so tiny details from concept to implementation). I’ve been wondering how you’d resolve authority clashes, but I see you had already given the answer to Maciej Sawitus in another post. I had thought it would be something more fancy than “he who connects first shalt have the upper hand”, but I guess it boils down to KISS again
I’d have guessed at something distance-based, e.g. giving auth to the player whose object was nearer to their controller/camera.
Is this confirmed? If yes, I’m going to register for the conference just for your talk!!
Too bad EA doesn’t sponsor me on this since it’s not GDC…
awesome, well the title of my talk is “Solving the Networked Physics Puzzle”, I’m told it is confirmed, but I guess be 100% sure you should wait until they announce the sessions!
They also have me signed up to host a networking round-table, so you should come to that too
See you there!
ok. confirmed!
Awesome! Thanks very much. I’ll see you there
You mentioned a few options for dynamic world synchronization. I think your articles cover most of them but I can’t seem to find anything on “styrofoam”. Could you elaborate on that one a bit?
Styrofoam physics is a trick to avoid synchronizing cosmetic physics objects. Treat the objects as if they have zero mass. The player can push these objects around, but they do not push back. Obviously this is not good for all situations but if you have a lot of debris flying about or if you just want secondary motion driven by the player and the actual sync of the the objects on each machine is not so important, it is a useful trick to consider. cheers