GDC Slides Released
OK I just got approval from EA and Pandemic so here are the slides for my GDC 2008 lecture:
Mercenaries 2: Networked Physics in a Large Streaming World
There slides are slightly different to what I presented at GDC, based on feedback, and what I wished I had presented! A little bit better structure for the middle section here, mentioning time synchronization and a bit more what we do for the pure client/server model, before launching into the authority management section.
I also explicitly mention that we simulate on both sides, because some folks were confused by this.
Another point to clear up, authority management is determined predictively on both machines, its not like the client requests authority from the server and waits for round trip, the client yanks white objects into blue authority as soon as it thinks it should, and the server is free to tell the client (at latent time), no no no mate, those are RED objects, and yank them back
Finally, these are just the slides, there are no demo videos yet, but I’m hoping to record some next week, then maybe release a version of the talk with me presenting the lecture and demos, recorded in Pandemic’s sound booth – I’ve been doing a fair bit of aussie voice work for them lately, so I’m gonna see if I can call in a favor!
Cool! Thanx for presentation – you are inventor – this something new for the industry.
Thanks for the slides! Is it possible to upload some of the demos somewhere?:P
Will there be a proper article on the technique? I would also like to know more about applications in non-coop environments.
First, I will release audio for the talk + demo videos – after this, I’ll consider writing an article, but I’ll not likely get time until *after* we ship Mercs2 – we’re crunching right now!
Great presentation gaffer, i love your stuff! I’ve used TinyPTC myself long time ago and i’ve read your zen of networked physics. One question though: If two players would grab the same box at the same local time and influence it, who would own the box locally? Would both clients have a BLUE copy and then the deltas are used to compute the combined average (on server) of the influences?
Thanks!
OK, for the general case, each client would have their own color – lets say BLUE, GREEN, YELLOW etc.
so client 1 grabs the cube and makes it BLUE, client 2 grabs the same cube locally and makes it GREEN
Since we are server topology, once each client sends the object update to the server (with each client telling server, hey i have authority) the server gets a chance to resolve who gets the authority
Here is how I would do it:
1. The first one to grab authority gets it (according to server)
2. If they are both on the same frame, the server could random(2) or always give the authority to the lower player id
This will generalize up to n players, but remember this technique does rely on a world of sparse stacks where most of the time players don’t interact with the same stack
cheers
I should also mention that authority is absolute, there is no good way to combine or mix authorities — this is because objects are in stacks, so you have to pick an exact known physics state (according to *one* of the machines) for that entire stack – and run with that
Interpolation, blending or smoothing towards corrected points is BAD in this model because your stack will become unstable
solution: snap physics state hard and apply visual smoothing on top!
cheers
Nice talk, cheers. One of the better ones I saw this year. Thanks for posting the slides!
Thanks for posting the slides, it was a great read!
I’m a big fan of your previous articles on networking and physics, and they served as a great inspiration for my current project.
It seems that few of the key concepts you’ve talked about can not be readily applied in non-Coop games (but vs.), so it’s a bit Coop specific. Nevertheless, it sounds great what you’re able to do: huge streaming world, largely dynamic, with coop, and all over the network! That’s 4 large beasts, all tackled at once. Amazing work. Good luck with finishing your game.
Looking forward to those videos/demos.
Great writeup – I really appreciate seeing the simulation thought behind a great looking game of the near future.
I just have a question about the impact of running the client-sims ahead of the server…
Doesn’t having client-authority + clients running sim ahead in time = double the amount of time-pop effect for the player(s) seeing the object but not controlling it? I understand the havok (physics) layer is the 60hz sim and that client-view is a smoothed view over the top, but if 2 clients have pings of 300ms (say around top of acceptable range for a FPS like this) and A controls a block (now blue) in sight range of both A and B, then the sim runs on A; sent to Server (150ms); server updates its local scene with block to be moved when it catches up (still 100ms in future on server sim) and sends info on what frame the blue block starts to move out to clients in range (B). B now receives that data (300-ish ms from when A took control) and has to incorporate that movement over that last 300 ms into its sim, checking for interaction with any B-controlled (Green) blocks.
I guess that leaves ping x 1 as the latency factor, not ping x 2 as I was worrying about. Actually, would a traditional server-controlled sim be ping / 2?
Is this pretty much how it would go, or is there a better/worse scenario for clients both running ahead of the server simulation?
Don
have a great day
Yes it does double the ping, but this technique is not for FPS where characters are running around a static world, its for physically correct networking of stacks of objects.
We need to be physically correct or the stacks will fall.
The way to be physically correct is to run the client ahead of the server, so it delivers input and state ahead of the server simulating that frame
Other techniques which are used for FPS are not physically correct, state from the network is “eased towards” as opposed to being snapped exactly, because after network transmission its already physically incorrect (state applied at different time than on client)
Using FPS techniques like these are fine for twitch competitive games, but as soon as you need to stack objects, they fall down (literally)
The pops only occur when the client and server interact with the same object stack, at 50-100ms latency (common when this technique is widely used), this pop will be no problem at all – just smoothing it out
cheers
Just found your GDC 2008 slides. Very interesting stuff. I’d love to see a demo… is one available?
Also, a question… say you have a see-saw. One guy steps on it, taking authority. He moves towards the middle, decreasing his leverage. Another guy steps on the see-saw, at the end. He has greater leverage, but the first guy still has authority over the see-saw, right? Is this handled properly in your scheme? I’m still wrapping my head around it
hey there, unfortunately there is no demo available – it does work with the system, but the client never gets to make the see-saw blue, it remains red (because red wins), so the client interaction with the see-saw is latent
this will cause some artifacts, the size of which will be proportional to the amount of latency
cheers
Thanks for the quick response! I see how it works in this case.
I also wonder about scalability for an MMORPG. I see that your solution uses distributed simulation as a means of authority distribution (since each client and the server are all deterministic). However all clients and the server are still simulating all objects, it’s just that some of that simulation work will be “thrown away” when an authoritative update comes down the pipe.
I wonder if it’s possible to truly distribute the physics processing so there is no “thrown away” work. I.e. I simulate the objects over which I have authority, and nobody else simulates them; other clients (and the server) contact me to get information about the objects I am simulating. Similarly, I contact not just the server, but also all other clients in the area to get updates. Think P2P physics. Clearly, stacks wouldn’t work in this case, since in the “washing machine” model the objects I am simulating would be propagated to other machines after unpredictable delays. But for simpler interactions, I wonder if it’s workable.
Think of a Second Life style environment where servers get bogged down when too many avatars are in one region. Distributing the workload is I think important here, and one of the more difficult parts to distribute is the physics.
Do you think your ideas can be used for the idea of a distributed physics workload as well?
yes, it absolutely does work
i’ve submitted a GDC 2009 talk proposal called “Drop-In COOP for Open World Games” that covers this exact topic.
stay tuned!
here is a quick summary for the talk:
on the important aspect of distributing simulation costs:
(this is all about running the sim authoritatively on each client,w hen they are far apart in the world)
and finally, some interesting options, if you cannot 100% trust the client…
There is *lots* more to the talk, coving many cool tricks and architectural changes to be made to support the no-server, peer simulation sharing approach, but the basic idea is the same as my authority management simulation talk, just taken to its logical conclusion as you just did sir!
cheers
Wow, sounds really cool. I’ll have to ask my boss if they have funding for me to attend!!!
I noticed your abstract mentioned “low player count, cooperative multiplayer games”. My particular interest is massive player count, cooperative virtual world environments (like Second Life X 1000). Do you think such things are possible using P2P? Imagine a virtual rave or a virtual olympics opening ceremony, with massive user count in a small area.
I do not think it is possible, because each peer would have O(n^2) bandwidth cost – this is why you have a server
Now if you combine a client/server network topology with distributed simulation – maybe
So yes, there is a good reason why you are not seeing lots of physics simulation in MMOs right now
“Finally, these are just the slides, there are no demo videos yet, but I’m hoping to record some next week, then maybe release a version of the talk with me presenting the lecture and demos, recorded in Pandemic’s sound booth”
Seeing as you have resigned from Pandemic Studios (
Congratulations on your engagement by the way ) I went and bought the audio of the talk from CMP store. Excellent talk, I was just wondering if there is any chance you can release the demos, which sound like they are newer modification of the old cube demos from some of your previous articles?
After a little bit of thinking about your talk I have a few questions I hope you would not mind answering Glenn.
In a post elsewhere on this blog you say
“Check out what one of our designers said after trying the game out under worst case conditions (64kbit/sec, 10% packet loss, 250ms latency…)”
The game was released on the Xbox360 which I believe has a test which it must pass of worst network condition having an 8kbs upstream[1]. Given that in your latest articles you use a buffer of 256 bytes which, if all of it was used, would give a maximum of thirty two network ticks per second to send to one other player.
What was the average network upstream in use in COOP mode?
How likely do you think the implementation your GDC talk is about would be used for other games (not on a PC yet console ) where there maybe say a sixteen player online version?
If an upstream could not handle sending to another 15 players what sort of tricks do you think could be used, maybe splitting the world into sections (sort of like a multi server MMO) where each player has authority of a piece?
Given that the world is pretty much destructible what sort of impact does this have on the network upstream?
How does the game handle when a player joins another in COOP mode in relation to the status of the world, ie is what the two people see the same or just a close approximation of the destruction of the game world?
So could one person hide behind a half destroyed wall for instance and be a shored that he is actually hiding and not in view?
Thanks.
1 Shawn Hargreaves. Networking, Traffic Jams, and Schrödinger’s Cat. Gamefest 2008
with mercs2 being 2 player we were able to use the full up/down stream for synchronization between players. the idea was to trade off the number of players to synchronize more objects in the world
the real problem to scaling this up is both the bandwidth cost, and the per-player simulation cost. i like to solve the simulation cost by only simulating objects around each player and using an extension of the authority management scheme in the lecture notes. alternatively you could have a seriously powerful dedicated server to do all the physics sync, but i think this is not good.
of course, given that you split the work between each machine, it is now peer to peer, so the bandwith per each node scales up O(n) and the total bandwidth O(n^2). i think this actually scales pretty good to 4 players, maybe 8 at the most, but that would be pushing it. beyond this you should consider a dedicated server.
regarding damage, in mercs2 we take the lowest health value across all machines, so it is not possible for an object to be destroyed on one machine but not on the other – it will eventually become destroyed as synchronization gets across. the worst case here is that damage gets applied differently on each machines, and you end up with the union of damaged objects across both machines
this ends up biasing towards destruction, which as you can imagine, actually works pretty good for a game like mercs2
cheers
i should also say that most of the techniques used in mercs2 assume
a) closed platform (cant hack EXE)
b) coop play (no incentive to cheat)
c) secure comms (cant inject packets…)
its possible to have b) and c) on a PC, but i dont really think you can get a). so… i would not recommend my techniques for a PC game
cheers
Error 404 – Not Found
Is the link to the slides still up?
down right now, i will get them back up this weekend – sorry about that
still 404…
but using google i found:
http://physics.hardwire.cz/mirror/NetworkedPhysics-GDC2008.pdf