One Down, One To Go!

The tutorial went pretty well, I barely made the demo in time and had a few minor logistical issues, but the audience seemed to enjoy my little jokes and self-effacing australian humor so all seems to be forgiven.
I’ll say just one thing about this talk and then that’s it.
This is the first time I’ve ever spoken in public and not been deeply ashamed of my performance afterwards.
Not saying it was great. I didn’t hit it out of the park. But I did what I needed to do, I was entirely myself up on stage and I formed just a tiny small spark of connection with the audience. And by that metric, for a mere programmer who really doesn’t know what he’s doing, I think I did OK.
If you are looking for the slides they are here.
If you are looking for more information about game networking and networked physics go here.
Networking for Physics Programmers

Here are the slides for my tutorial tomorrow
The goal of the tutorial is to show people the basic ideas of how to network a physics simulation
The key points are:
- Use UDP, not TCP
- Aim for 64kbit/sec – That’s 256 byte packets, 30 times per-second **
- Handle out of order packets with a sequence number
- Send input and physics state in your packet
- Snap the physics state and extrapolate
- Run the simulation on both sides, and use the input to improve extrapolation
- Use visual smoothing to hide errors
Of course, there is a lot more to it but I hope this serves as a good starting point if you want to get into networked physics
Cheers!
** actually, that’s a little bit over 64kbit/sec, especially when you include IPsec header for consoles and UDP packet header, it’s just such a nice number for folks to remember, I couldn’t resist a little white lie
Why you really should come to my GDC talk
Two words: demo talk.
My entire talk is based around an interactive demo. It’s a demo talk. There are no slides. I have 45 minutes of material inside this interactive demo to show you guys and it’s all good stuff.
I just finished the demo today. Here’s a screenshot:

Now a bit of info on what info I’m actually presenting…
Open world games are very difficult to network. I learned this lesson the hard way on Mercs2. The key problem is that open world games have really high CPU, memory and streaming costs, so if you try networking these games using client/server you end up with two, three or four times the CPU, memory and streaming cost on the server player’s machine.
It gets worse. If players are free to go where they like then streaming performance degrades at a rate worse than linear, because you have to do lots of additional seeks when players are in different areas of the map.
All of this adds up to a gigantic pain in the ass. The sort of problem that makes grown programmers cry. And I did. You see there is really no way out other than reducing quality and tethering the players together *OR* offloading all the work to a dedicated server.
And a dedicated server gets expensive. So for Mercs2 we basically threw our hands up, reduced the hibernation distance by 50% when the second player joined the game (notice the extra pop-in during Mercs2 COOP? this is why…), and tethered the players together within 500m.
Yes we may have shipped the game with COOP, but only just.
After I shipped Mercs2 I had a hunch. What if there was some way to distribute CPU, memory and streaming cost across each player in the game? I wasn’t 100% sure, but I suspected it may be possible if I used peer-to-peer instead of client/server.
This way I could avoid paying for a dedicated server and I wouldn’t have to reduce quality or tether the players together. Overall the quality of the game would be significantly better in COOP.
So after Mercs2 shipped, I quit my job at Pandemic and started working on a demo…
If you come to my talk you get to see this demo.
In fact, everything I tell you in my talk, I’ve already coded and can show you working right in front of your eyes. I’ll even compile the fucking code live during the talk as I flip #defines to go from one part of the demo to the next.
I’ll show you how to keep a physically simulated world of one million objects in sync using peer-to-peer. How to avoid feedback, how to use an authority scheme to eliminate latency. And finally, how to support late joins so players can join and leave at any time. In summary, how to keep one million physically simulated cubes in sync using peer-to-peer. Even cubes that were pushed around before the player joined the game.
If you want to learn how do all of this then you really should come to my GDC talk!
Update: the demo from the talk “Fiedler’s Cubes” is now available at google code.
GDC2009: Officially Shitting Bricks RIGHT NOW

Checked the calendar today, I fly out to GDC this sunday, for some reason I thought I had a week more than I did. Oh man.
Well I have my final talk layout done, and the demo is nearly finished – but I’m still scared shitless. Why?
I guess it’s always difficult to talk in front of an audience of 500 people, no matter how hard you prepare… there is always more you feel you can do. Oh and even better, my talk is 9AM friday, which means I’m get to stay in this awesome scared shitless state ALL WEEK LONG, meaning I’ll spend most of my time in sanfran practicing and preparing my talk in my hotel room. Argh.
Oh, and that previous post I gave about how to prepare a talk for GDC?
Forget everything I said and buy this book.
Last Minute GDC Crunch!
Still hard at work on my GDC talks!

Practice session at USC went great yesterday, and I just finished the slides for my second talk Networking for Physics Programmers.
It’s turned into something pretty cool, a short 30 minute lecture on how to network a physics simulation, better than boring everybody over an hour, and I can keep the content really nice and dense! Lots of “Aha!” moments is my goal
GDC Practice at USC GamePipe
I’m practicing my GDC talk “Drop-IN COOP for Open World Games” as part of the USC GamePipe seminar series.
I’m talking next week on Thursday, March 12th at 10AM.
So if you’re a student at USC and are interested in game networking, drop on by and give me some feedback!
Game Developer Magazine Recommends my GDC Talk
“DROP-IN CO-OP FOR OPEN WORLD GAMES [LECTURE]
Glenn Fiedler (Sony)
Fiedler was the multiplayer lead programmer on Pandemic/EA’s MERCENARIES 2, which featured a streaming open world and drop-in, drop-out co-op throughout its entire campaign – one of the things critics and fans liked best about the game. Making that work requires a huge amount of technical knowledge and skill. When I consider the great times I had with another co-op open worlder, Realtime Worlds/Microsoft’s CRACKDOWN, which offered a similarly free-form co-op experience, I’m convinced more developers need to emphasize this in open-world games. Here’s where to start.”
Woot!
How to Prepare a Talk for GDC
GDC 2009 is about one month away, and so far I’ve made pretty good progress on my GDC talks this year
I’m doing two talks at GDC this year. My first talk “Networking for Physics Programmers” was written and practiced in rough draft form for the first time around November last year in front of the Sony Santa Monica programming team. I actually gave the talk at the start of my third week working for them. Ow that was a lot of work while starting a new job!
It took me about a week to plan out the talk, including three or four informal practice sessions in front of co-workers the week before. The key thing I learned from those practice sessions was that improvising a talk against bullet points is not the way to go if you want to really nail a talk. You may do OK, you might not completely fuck it up. But you won’t give a highly rated talk because your thoughts will all be in a jumble.
What I discovered is that you need to distill the information in your talk down. Dump it all out in the speaker notes then talk through it out loud. Actually write out what you want to say. Adjust it so it sounds more natural. Practice each slide over and over, adjusting the speaker notes as you improvise and riff off the contents. Say something you thought was cool? Add it to the speaker notes. Speak it out loud as you add it.
Practice the slides in sequence, do they flow naturally? Practice in front of your girlfriend or your cat (or both) and adjust your slides whenever you feel compelled to explain something in more detail. Maybe you glossed over something critical for people to understand your point? You don’t want to lose people just because they don’t understand a technical term you used.
Then finally practice your talk in front of your team at work. Don’t practice it before you’ve already gotten it to this point. You can’t get a first impression from the team again, and nobody really wants to sit down for an hour to listen to a talk they’ve already heard.
After doing all of this preparation my practice talk at work went really well. Much better than my GDC talk last year where I improvised off the slides and rushed through. I thought reading off speaker notes would be more stilted, but it ended up actually being much more natural because I was relaxed, and I knew the talk well enough to maintain eye contact with my workmates most of the time. Having it all written out also really helped me pace out the talk, so I didn’t rush.
So about a month ago, I started work on my second talk “Drop-In COOP for Open World Games” thinking hey I’ve got this all nailed – no worries! I went through the process described above, worked for several weeks on the talk, wrote drafts, refined, talked through the slides, practiced out loud. I did everything I described above. But I just couldn’t shake the feeling that the talk was mundane, boring.
So I left the talk on the shelf and came back to it a few weeks later. And holy shit, it was boring.
What happened? Well the difference is that my first talk is a tutorial and second talk is a lecture. A tutorial is really just about transferring as much info as I can into the minds of people in the audience. Cram that information in. It’s OK if it’s like a shopping list: this is what you do, then that then this. In the end, you’ll be judged on the cool info you transfer to them. Anything else is bonus points really.
But my second talk is a lecture about a new technique for networking COOP games. It’s controversial. I have to sell it: I have to explain what problem it solves, why it’s a good technique, why it is useful and why it’s better than what people are currently doing. Basically, I have an hour to convince people that I’m not smoking game networking crack.
So the last possible thing you should do in this situation is just jump right in and read out a shopping list of how you implement your technique. First you have to convince people you are right, you have to persuade them.
How did I find this out? I’m not an experienced speaker, so I spent a night researching for tips on how to prepare a talk. It turns out that googling for “how to give a good talk” is a goldmine.
This advice was so useful to me, I kept notes and tidbits I thought were particularly good.
So here is all the good advice I could find in one big list. Hopefully it will help you out with your talk as well
cheers
“You have two minutes to engage your audience before they start to doze. Why should I tune into this talk? What is the problem? Why is it an interesting problem?”
“Use a top down approach. 1) Introduction. Define problem. Put in a carrot. Put in context and give an outline.”
“Motivate the audience. Why is this problem important? How does it fit into the larger picture? What are the applications?”
“Body. Abstract the key results. Focus on a key exciting result. Explain significance of your work. Sketch methodology of key ideas. Keep it high level. Use pictures/diagrams. Provide intuition. Gloss over technical details”
“(1) Communicate your arguments and evidence. (2) Persuade the audience that they are true. (3) Be interesting and entertaining”
“Don’t start with a joke. The audience is not accustomed to you or your speaking style yet. Humor will be difficult at this point.”
“Provide an empowerment promise. Explain why your audience will come away from your talk better than they entered”
“Decide how many concepts or points you can adequately get across in the allotted time (one concept per-5 minutes is a reasonable rule of thumb), and prioritize to the most important ones”
“‘then i did this, then i did that…’, Is your chronology important to them? Start with messages the audience will receive. Plan backwards from them, not forwards from you.”
“You must have messages. Work out what they are.”
“Topic order. State messages early. Not a mystery. Call it out, then explain why it is true. Your talk is not a suspense novel”
“Don’t overwhelm your audience. It is better to clearly explain one or two main ideas than to cursorily cover four or five ideas. In a one hour talk, your audience can only absorb one or two central ideas – if you try to cover several years of work, they are going to be overwhelmed and will stop paying attention”
“The conference talks I remember are the ones where the speaker spoke slowly and confidently and made one point well. Talks that try to make too many points are almost always forgettable (and forgotten.) A rushed talk can be good, but it won’t be great.”
“Talks are different from papers. While you must explain your details in your paper, this is not necessary in a talk. The primary goal of a talk is not to explain every detail of your work. It is more important to explain the general concepts, build intuition, and then allow those who are interested to read your paper to get all of the details.”
“Know your goal. A conference talk’s goal is to make the audience want to read your paper. The talk does not replace the paper.”
“Before you make a point, the audience needs to know the point you are about to make and appreciate why you are making it. After you make a point the audience needs to be reminded of the point you just made and understand why it matters.”
“Most good speakers average two minutes per-slide.”
“When time is short omit the general case, not the example”
Moving to Santa Monica!
So for the last three years we’ve been living in a tiny studio apartment in Westwood for $1100 a month.
Now this worked great while I was single and working at Pandemic, but now I’m biking to Santa Monica each morning, and I’m engaged to be married this year so lets just say the studio apartment is getting a bit tight. Throw in two cats and a litterbox, and it’s even more interesting.
So… last weekend we looked around for a new place. Working at Sony Santa Monica, we were thinking Santa Monica or Venice – anywhere close to the beach. The good news is, after checking out about 5 places we found a great 1BR place with hardwood floors 9 blocks from Santa Monica beach for just $1500. A great deal. We’re moving in this weekend.
So, I’ll still be biking to work, but this time I’ll not be weaving through the 405 traffic on my way home in the evenings! I might even be alive to see my 40’s at this rate…
Good times!
Game Networking Resources
There are some pretty good resources on game networking out there, but they are pretty scattered and sometimes hard to find. To remedy this, in this post I’m going to collect all the good game networking links I know. Please share any links you would like to add in the comments section. Thanks!
First up is The Unreal Networking Model by Tim Sweeney. This article covers the concepts of writing a general purpose networking engine on top of a distributed object model. Take careful note of Tim’s ideas about bandwidth vs. CPU growth (still true today), his replication statement design, and the concept of authority enabling selective simulation on the client. Pay special attention to his explanation of client side prediction!
Of course, Tim’s document is very high-level. It doesn’t really explain how you would actually send that stuff over the network. That’s fine if you are going to use Unreal, but for the rest of us who want to write our own engine, there is The Tribes Networking Model. Key ideas include their notify protocol for reliability over UDP and their concept of independent event channels with different reliability guarantees. You can even take a look at the source code for this as part of their open source library OpenTNL. Lots of good stuff here.
Another way to synchronize your game data is via delta encoding as described in the Quake3 Networking Model. Here Brian Hook explains the technique developed by John Carmack for Quake3. The basic idea is to delta encode the entire game state, ensuring that everything gets across without needing to worry about the complexity of using events to synchronize state. Be careful though, because this technique is only applicable if you have a small set of data. Why? Well your packet needs to contain the *complete* difference between a previous base state and your new state, so don’t try using it for open world streaming games!
No networking resource list is complete without Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization by Yahn Bernier. This document describes the key techniques differentiating Counterstrike and Team Fortress 2 from the original Quake networking model, namely their latency compensation techniques for player aiming so you don’t have to lead the player by the round trip time (RTT). Take note because this technique is why these games are so goddamn popular, and simultaneously why they are so prone to cheating
Eventually you are going to have to get familiar with low-level networking issues, especially NAT (network address translation) routers and how to establish connections with players behind them. The standard technique for this is called “NAT Punch-through” and it basically involves “fooling” the router into exchanging packets it would otherwise would not allow. You can find a good overview of this technique here.
As you can imagine getting this working across every router ever made is very tricky. Even the big guys have trouble with it (EA, microsoft etc…). So don’t get arrogant and expect that you are going to get it right first time! Take my advice here and use an existing library that has already worked out all the kinks. If you are writing a PS3 game then PSN NP has matchmaking services for you that handles the NAT punch-through, as does XBox-Live on the 360. But, if you are on the PC you are SOL so take a look at one of the following libraries: OpenTNL, Raknet, Quazal. Do not write this yourself if you can possibly avoid it!
Ok that’s about it for now, I have a lot more docs to add, but that covers the big ones that should get you started.
Let me know in the comments if you have any other good docs to add, cheers!.