Pandemic Shut Down
Very sad news earlier this week: Pandemic Studios was shut down by EA.
No point moping about although. What’s done is done. I choose to remember the good times not the bad: the friends I made and the colleagues I worked with at Pandemic and the shared experiences we went through. No matter how difficult and stressful they were, these experiences bind us together.
Here is my own personal Pandemic story: having worked for Irrational Games, and then Team Bondi – Pandemic flew me out from Sydney, Australia to work on Mercenaries 2 in Los Angeles. Obviously, this was a very significant event in my life, resulting in me settling in LA, meeting my wife, and eventually being hired by Sony. Along the way there were a lot of ups and downs: Getting acquired by EA, one week having an incredible demo and winning support from management, the next week struggling to get resources for the designers to test in COOP and fighting over stupid things like syncing debris and pausing in multiplayer. It was one hell of a rough ride!
In the last 6 months on Mercs2 – Andrew Goldman, CEO of Pandemic, donated his office to the multiplayer team so that we could all work closely together and make sure Mercs2 shipped with COOP. It’s a story of 8 friends working very closely together (literally!) under tight deadlines. Of persistence and determination in the face of tremendous odds, when many of us – myself included – were not even sure it was even *possible* to network an large open world game like Mercs2. We kept on and in the end, succeeded. Mercs2 became the first open world game to ship with drop-in / drop-out COOP throughout the entire singleplayer campaign.
So here, never before told – is the story of the Mercs2 Multiplayer Room and the men who made it what it was.
It’s Cold in Montreal!
Spent most of my flight thinking through my slides and material and making adjustments, then watched through Harry Potter the Half Blood Prince, but we arrived before the movie finished! Now I must find out how it ends… 
Arrived Friday night around 9pm. Discovered this while in the customs line that my passport was no longer in my pocket. Holy fucking shit! Frantic 5 minutes – back to the plane and we discovered it had slipped out of my pocket and down the side of my seat. Phew!
Through customs I arrived at the Hilton Bonaventure and checked into a very nice room. At this point, I dressed up warmly and headed out to Old Montreal to check out some restaurants and bars. A very nice area but a bit quieter than I expected. I found a really nice bar with good food and had great short ribs, a nice syrah and Creme Brulee for dessert. Oh yes.
This morning I woke up feeling a bit tired… a slight time difference between here and LA. Went around to explore downtown Montreal. I found St Catherine Street and the main area and walked up along all the shops. Walked past the strip Club area. Pushy bouncers. No thanks I don’t want to go in. No really. I mean it. Hit the Apple store to buy another USB ethernet adaptor for my Macbook Air (I left it home =p), then enjoyed some good Coffee at Cafe Myriade.
A few hours on my talk then met up with Aaron Contreras an old Pandemic buddy. Checked out the plateau area around Ubi and the italian neighbourhood. Good times. Aaron has a sweet studio style apartment. Jealous!
Back at the hotel now and some more work on the slides. I want to finalize the first half of the talk tonight, then hopefully I’ll have a bit of time tomorrow to spend walking around Montreal at lunch instead of spending the whole day preparing for the talk!
cheers all
Update: the demo from the talk is now available at google code, slides and resources from the talk are available here.
Solving the Networked Physics Puzzle
Traditional networking techniques work well for simple linear motion, but start to break down when networking complex rigid-body physics simulations where objects can tumble, stack and interact with each other. This light-hearted and humorous tutorial takes a look at various options, techniques and pitfalls to watch out for when networking this sort of simulation, providing you with a tool-bag of new techniques and ideas you can use to network your physics based game.
I’ll be performing this talk at the Montreal International Game Summit on November 16, 2009 – 10:15AM in the Verdun Room. I’m honored to be invited to perform one of the first sessions at the conference and hope to see some of you guys there! Please come up after the talk if you would like to discuss anything at all I will be available for questions.
Plus, this will be my first time visiting Montreal so I’m very excited!
I have the demo for the talk almost ready and have been working on the slides over the last few weekends. Right now I’m trying to lock down the overall theme of the talk and the content. My goal is to provide lots of interesting information without going to deep into the specifics on any one thing. Ideally folks should leave the talk with a great outline of the whole field and some great ideas and inspiration to take home to apply to their own projects!
Here is a more detailed version of my talk plan:
I intend to start by asking the question: why is it that very few games provide meaningful interaction with networked physics objects? In other words, folks it’s 2009 – why exactly are we still running around mostly static worlds shooting other people in the face?
There are many reasons. It is difficult to light a dynamic world. Character movement and animation is difficult. But also, it seems very difficult to network a dynamic world. Why is this? Traditional networking techniques don’t work. Objects can stack and tumble and interact with each other.
In truth: It’s like a puzzle. It’s actually really simple – you just need to think in a different way.
Quick 5-10 minute recap of a game networking engine. UDP vs. TCP, how to construct packets, serialization, reliability, flow control. All of the stuff in my online book and more at a brisk pace for folks already in the field. My goal is simply to get everybody on the same page and provide some very useful, concrete information for folks in the field. Here is my network engine and how I do everything.
The plumbing is out of the way. Now, lets explore options for getting a physics simulation on one computer to be simply *viewed* on another. Two main options: 1) run the simulation on one side and present an approximation on the client side (pure client/server model), 2) run the simulation on both sides and synchronize via state replication. Compare and contrast these two approaches. Touch on time synchronization and real-time nature of protocol and why it is now this way vs. traditional networking technique.
Latency compensation. Consider the same situation: A simulation synchronized between two computers, but now client controls their own object. Recap traditional client side prediction from FPS games. Attempt to apply it to network this physics simulation. Problems: Rewind and replay cost. Simulation islands. Complexity. Authority scheme as an alternative. Dining philosophers problem. Distributed programming. Dykstra. Consistency vs. throughput trade-off. Case studies.
Live demo. P2P authority scheme. Late joins. Corrections. Reverse corrections. Challenge: Think outside the box. Maybe client/server should not be the automatic choice? How important is anti-cheat for your game? How important is meaningful interaction with a dynamic world? Which one wins? Is it possible to have both? How?
Future looking. What can you do moving forward? What can physics engine developers do to make their engines network better? What do we need before this becomes practical? What will game networking look like in 10 years? Bandwidth considerations. Real-time protocol requirements. IPv6. RSVP, Flow Routing. QoS guarantees. Network neutrality.
Summary. Bring it back to the original point about the puzzle. Conclusion. Best approaches for each situation. Call to action. Triumphant dance. Q&A.
That’s about it for now. Please be sure let me know what you think and if you are planning on attending the conference. Sound off in the comments below – thanks guys!
Debugging Multiplayer Games
It seems that every few months I read a post-mortem or I hear about a case where a team had to cut multiplayer at the last minute, because it didn’t fit into the schedule, or they couldn’t get it working, or because they want to “focus on the singleplayer experience”. This is especially true of teams that try to convert their existing singleplayer codebase to to support multiplayer.
Hell, I’ve even been in this situation myself with Mercs2, and I can tell you we only just made it by the skin of our teeth – despite having a good team, the full support of the studio and plenty of resources to help us out.
So why is it that so many teams fail to integrate multiplayer at the last minute? Why does it at first seem to be going so well, but then bog down as teams struggle to get it multiplayer to shipping quality?
In my experience it boils down to this: debugging multiplayer games is hard. Not just your ordinary hard. More like pulling your hair out, grown man reduced to tears, holy shit it’s 6AM I’ve been up all night and the E3 demo is in 4 hours and it still doesn’t work hard.
So in this article, I’m going to share my general process, techniques and tools that I use for debugging multiplayer games.
I hope these techniques will help you out debugging your multiplayer game!
Read the article in Networking for Game Programmers
The Return of the Timestep Crusader
Come with me now folks on a magical journey into the mind of abaraba – the timestep crusader.
Regular readers of this blog may remember the last time abaraba shared his brilliant ideas with us.
Now he’s back in a brand new episode!
You’ll discover incredible, fantastical things: how P2P actually uses less bandwidth than client/server and can scale to a larger number of players, how s=ut+1/2at^2 always provides exact results for numerical integration – and ultimately, how every single numerical integrator is completely and totally wrong because acceleration is always constant across the timestep.
So please, sit back – relax, and enjoy the following tasty links *:
devmaster.net
gamedev.net (1)
gamedev.net (2)
gamedev.net (3)
gafferongames.com
natscience.com
sci.physics.relativity
If you missed the original episode, you’re in luck – you can still catch it here:
devmaster.net
intel.com
nvidia.com
bautforum.com
gamedev.net
I hope you enjoyed your daily dose of stupid, if so then please vote this up on reddit. Remember, only *YOU* can expose the truth about time-stepping!
* ps. Hope you aren’t planning on doing anything else for the next 45 minutes…
UPDATE: Ruh-roh! Looks like abaraba has discovered philosophy!
Montreal International Game Summit 2009
Good news! I’ve been invited to speak at the Montreal International Game Summit 2009
I haven’t decided on a name for the talk yet, but it will focus on networked physics and combine material from my GDC 2009 tutorial, lecture and new material I’m going to develop over the next few months.
The basic overview of the talk is something like this:
- How the internet works and why we should use UDP
- Bandwidth considerations, packet structure, 64kbit/sec
- Describe basic algorithm for synchronization: push/pull/serialize with visual smoothing
- Synchronize one server controlled cube, client is just a viewer of simulation
- Synchronize two player controlled cubes, client and server cube.
- Effects of latency, client side prediction
- High simulation cost for rewind and replay, client side prediction breaks down in a dynamic world
- Dynamic world synchronization: options – client side prediction, no prediction, styrofoam, authority scheme
- Fiedler’s Cubes demo showing authority scheme to avoid high cost
- Corrections, reverse corrections
- Compression techniques
- Cool demo to go out
- Extended Q&A
The idea is to pretty much combine all the talks I’ve done so far on networked physics and present the information in a clear, simple manner – anybody who comes to this talk will leave with all the knowledge they need to network a physics simulation.
And of course, I get to spend a weekend in Montreal with my sweetheart about 2 months after we get married
So you see, it’s win-win folks 
Aussie Aussie Aussie – OI OI OI!
Incredible. Just Incredible! Heads are screwed on right down under. After years of bungling they’ve made the right call. Just perfect!
Kevin Rudd has announced that the NBN tender process has been terminated, and that the government will go it alone on a new $43 billion broadband network.
The new wholesale-only network will connect 90% of homes with fibre to the home and will offer 100Mbit/s, with “next-generation” wireless and “third-generation” satellites to cover the remaining population. The network will be “open access” so retail ISPs can build their own products to sell to businesses and consumers.
If they keep this up, I might just start getting a bit homesick!
source: whirlpool.net.au
Almost There
If it before Friday 9AM, please please come to my talk Drop-In COOP for Open World Games in West Hall, Room 2014.
If it is after, then thankyou, good afternoon, good evening and goodnight.
If you would like to play the demo of Fiedler’s Cubes – please call me on 310-***-****, or email gaffer@gaffer.org
I’ll be in San Francisco all day today and all weekend after the conference
Wish me luck!
Update: the demo from the talk “Fiedler’s Cubes” is now available at google code.
One Down, One To Go!
The tutorial went pretty well, I barely made the demo in time and had a few minor logistical issues, but the audience seemed to enjoy my little jokes and self-effacing australian humor so all seems to be forgiven.
I’ll say just one thing about this talk and then that’s it.
This is the first time I’ve ever spoken in public and not been deeply ashamed of my performance afterwards.
Not saying it was great. I didn’t hit it out of the park. But I did what I needed to do, I was entirely myself up on stage and I formed just a tiny small spark of connection with the audience. And by that metric, for a mere programmer who really doesn’t know what he’s doing, I think I did OK.
If you are looking for the slides they are here.
If you are looking for more information about game networking and networked physics go here.
Networking for Physics Programmers
Here are the slides for my tutorial tomorrow
The goal of the tutorial is to show people the basic ideas of how to network a physics simulation
The key points are:
- Use UDP, not TCP
- Aim for 64kbit/sec – That’s 256 byte packets, 30 times per-second **
- Handle out of order packets with a sequence number
- Send input and physics state in your packet
- Snap the physics state and extrapolate
- Run the simulation on both sides, and use the input to improve extrapolation
- Use visual smoothing to hide errors
Of course, there is a lot more to it but I hope this serves as a good starting point if you want to get into networked physics
Cheers!
** actually, that’s a little bit over 64kbit/sec, especially when you include IPsec header for consoles and UDP packet header, it’s just such a nice number for folks to remember, I couldn’t resist a little white lie
