Gaffer on Games

Traditional networking techniques work well for simple linear motion, but start to break down when networking complex rigid-body physics simulations where objects can tumble, stack and interact with each other. This light-hearted and humorous tutorial takes a look at various options, techniques and pitfalls to watch out for when networking this sort of simulation, providing you with a tool-bag of new techniques and ideas you can use to network your physics based game.

I’ll be performing this talk at the Montreal International Game Summit on November 16, 2009 – 10:15AM in the Verdun Room. I’m honored to be invited to perform one of the first sessions at the conference and hope to see some of you guys there! Please come up after the talk if you would like to discuss anything at all I will be available for questions.

Plus, this will be my first time visiting Montreal so I’m very excited!

I have the demo for the talk almost ready and have been working on the slides over the last few weekends. Right now I’m trying to lock down the overall theme of the talk and the content. My goal is to provide lots of interesting information without going to deep into the specifics on any one thing. Ideally folks should leave the talk with a great outline of the whole field and some great ideas and inspiration to take home to apply to their own projects!

Here is a more detailed version of my talk plan:

I intend to start by asking the question: why is it that very few games provide meaningful interaction with networked physics objects? In other words, folks it’s 2009 – why exactly are we still running around mostly static worlds shooting other people in the face?

There are many reasons. It is difficult to light a dynamic world. Character movement and animation is difficult. But also, it seems very difficult to network a dynamic world. Why is this? Traditional networking techniques don’t work. Objects can stack and tumble and interact with each other.

In truth: It’s like a puzzle. It’s actually really simple – you just need to think in a different way.

Quick 5-10 minute recap of a game networking engine. UDP vs. TCP, how to construct packets, serialization, reliability, flow control. All of the stuff in my online book and more at a brisk pace for folks already in the field. My goal is simply to get everybody on the same page and provide some very useful, concrete information for folks in the field. Here is my network engine and how I do everything.

The plumbing is out of the way. Now, lets explore options for getting a physics simulation on one computer to be simply *viewed* on another. Two main options: 1) run the simulation on one side and present an approximation on the client side (pure client/server model), 2) run the simulation on both sides and synchronize via state replication. Compare and contrast these two approaches. Touch on time synchronization and real-time nature of protocol and why it is now this way vs. traditional networking technique.

Latency compensation. Consider the same situation: A simulation synchronized between two computers, but now client controls their own object. Recap traditional client side prediction from FPS games. Attempt to apply it to network this physics simulation. Problems: Rewind and replay cost. Simulation islands. Complexity. Authority scheme as an alternative. Dining philosophers problem. Distributed programming. Dykstra. Consistency vs. throughput trade-off. Case studies.

Live demo. P2P authority scheme. Late joins. Corrections. Reverse corrections. Challenge: Think outside the box. Maybe client/server should not be the automatic choice? How important is anti-cheat for your game? How important is meaningful interaction with a dynamic world? Which one wins? Is it possible to have both? How?

Future looking. What can you do moving forward? What can physics engine developers do to make their engines network better? What do we need before this becomes practical? What will game networking look like in 10 years? Bandwidth considerations. Real-time protocol requirements. IPv6. RSVP, Flow Routing. QoS guarantees. Network neutrality.

Summary. Bring it back to the original point about the puzzle. Conclusion. Best approaches for each situation. Call to action. Triumphant dance. Q&A.

That’s about it for now. Please be sure let me know what you think and if you are planning on attending the conference. Sound off in the comments below – thanks guys!

Come with me now folks on a magical journey into the mind of abaraba – the timestep crusader.

Regular readers of this blog may remember the last time abaraba shared his brilliant ideas with us.

Now he’s back in a brand new episode!

You’ll discover incredible, fantastical things: how P2P actually uses less bandwidth than client/server and can scale to a larger number of players, how s=ut+1/2at^2 always provides exact results for numerical integration – and ultimately, how every single numerical integrator is completely and totally wrong because acceleration is always constant across the timestep.

So please, sit back – relax, and enjoy the following tasty links *:

devmaster.net
gamedev.net (1)
gamedev.net (2)
gamedev.net (3)
gafferongames.com
natscience.com
sci.physics.relativity

If you missed the original episode, you’re in luck – you can still catch it here:

devmaster.net
intel.com
nvidia.com
bautforum.com
gamedev.net

I hope you enjoyed your daily dose of stupid, if so then please vote this up on reddit. Remember, only *YOU* can expose the truth about time-stepping!

* ps. Hope you aren’t planning on doing anything else for the next 45 minutes…

UPDATE: Ruh-roh! Looks like abaraba has discovered philosophy!

Here are the slides for my tutorial tomorrow

The goal of the tutorial is to show people the basic ideas of how to network a physics simulation

The key points are:

• Use UDP, not TCP
• Aim for 64kbit/sec – That’s 256 byte packets, 30 times per-second **
• Handle out of order packets with a sequence number
• Send input and physics state in your packet
• Snap the physics state and extrapolate
• Run the simulation on both sides, and use the input to improve extrapolation
• Use visual smoothing to hide errors

Of course, there is a lot more to it but I hope this serves as a good starting point if you want to get into networked physics

Cheers!

** actually, that’s a little bit over 64kbit/sec, especially when you include IPsec header for consoles and UDP packet header, it’s just such a nice number for folks to remember, I couldn’t resist a little white lie

Two words: demo talk.

My entire talk is based around an interactive demo. It’s a demo talk. There are no slides. I have 45 minutes of material inside this interactive demo to show you guys and it’s all good stuff.

I just finished the demo today. Here’s a screenshot:

Now a bit of info on what info I’m actually presenting…

Open world games are very difficult to network. I learned this lesson the hard way on Mercs2. The key problem is that open world games have really high CPU, memory and streaming costs, so if you try networking these games using client/server you end up with two, three or four times the CPU, memory and streaming cost on the server player’s machine.

It gets worse. If players are free to go where they like then streaming performance degrades at a rate worse than linear, because you have to do lots of additional seeks when players are in different areas of the map.

All of this adds up to a gigantic pain in the ass. The sort of problem that makes grown programmers cry. And I did. You see there is really no way out other than reducing quality and tethering the players together *OR* offloading all the work to a dedicated server.

And a dedicated server gets expensive. So for Mercs2 we basically threw our hands up, reduced the hibernation distance by 50% when the second player joined the game (notice the extra pop-in during Mercs2 COOP? this is why…), and tethered the players together within 500m.

Yes we may have shipped the game with COOP, but only just.

After I shipped Mercs2 I had a hunch. What if there was some way to distribute CPU, memory and streaming cost across each player in the game? I wasn’t 100% sure, but I suspected it may be possible if I used peer-to-peer instead of client/server.

This way I could avoid paying for a dedicated server and I wouldn’t have to reduce quality or tether the players together. Overall the quality of the game would be significantly better in COOP.

So after Mercs2 shipped, I quit my job at Pandemic and started working on a demo…

If you come to my talk you get to see this demo.

In fact, everything I tell you in my talk, I’ve already coded and can show you working right in front of your eyes. I’ll even compile the fucking code live during the talk as I flip #defines to go from one part of the demo to the next.

I’ll show you how to keep a physically simulated world of one million objects in sync using peer-to-peer. How to avoid feedback, how to use an authority scheme to eliminate latency. And finally, how to support late joins so players can join and leave at any time. In summary, how to keep one million physically simulated cubes in sync using peer-to-peer. Even cubes that were pushed around before the player joined the game.

If you want to learn how do all of this then you really should come to my GDC talk!