Evan
Maestro
- Joined
- Feb 1, 2010
- Messages
- 2,221
- Reaction score
- 8
- Points
- 90
Halo Reach - Lessons from the Beta
![url]](/proxy.php?image=http%3A%2F%2F%5Burl%5Dhttp%3A%2F%2Fcdn1.gamepro.com%2Farticle_img%2Fgamepro%2F215665-1.jpg%5B%2Furl%5D&hash=ab92d86ac1d0824acf9b749253727033)
Glad to see they're doing something about the bugs. I ran into a bbug once during invasion where I was a Spartan with Elite capabilities and Elite weapons. It was pretty cool.
Halo: Reach's multiplayer beta, which ran from May 3 through May 19, had more than 2.7 million participants. In this exclusive interview, Bungie's community director Brian Jarrard discusses the lessons, numbers, and stories that came out of the beta testing, as well as what we can expect to see in Halo: Reach when the full game releases September 14 on Xbox 360.
GamePro: Can you talk about what it was like keeping such a massive beta test under control? How do you manage the tremendous amounts of data and feedback a beta of this scale generates?
Brian Jarrard: We tallied out over 2.7 million unique participants. As far as we know it's the largest console beta of all time. It's definitely exceeded our expectations, so we were really happy with the huge turnout.
The good news is a lot of our testing was automated, in terms of data uploads including connection speeds, different enabled settings, social preferences, and how long it takes players to find games. All of that was automatic.
Where it became way more challenging, though, is in the more subjective anecdotal feedback, from people sending us emails, posting in forums, and sending us notes on Twitter. At this scale, that just became completely overwhelming and it was a real challenge to keep up with the volume and just the sheer amount and varying degrees of feedback. We had to distill all the noise any consistent overlying themes that seemed to rise to the top.
I can tell you just for context that we had over 360,000 forum posts created just in our Bungie.net official forum alone. And that doesn't even include all the other outlets people used to talk about the beta. That became very challenging for us. We realized that a centralized forum was not working at this scale and this volume. Somebody would create a thread or a post that would say something like, "this head shot didn't register." And by the time they'd hit 'submit,' it would already be three pages deep in the forum. And in a minute somebody else would post the exact same thing. It was just very unwieldy.
In response to this, we tried a new approach. We created official pin threads for each major bucket of feedback we'd been seeing. Things from like technical glitches to weapon balance. We tried to give people a little bit more of a direct avenue to give that feedback and to make our lives easier. It was definitely a lot to assess and digest.
GP: How else did this beta differ from past beta tests you've gone through?
BJ: Comparing it to the Halo 3 beta, at the time that was a really big deal for Bungie. We ended up having over 800,000 people playing that beta and we learned a lot from that. That one was rockier for us in some ways, because Halo 3 was trying out a whole bunch of new stuff like custom-created content, file-sharing, and screenshots. A lot of those features are carrying over to Reach, and they've been improved and tweaked. But a lot of the initial pain and learning, we had already gone through with the testing and release of Halo 3. So that was a less risky component for us with Reach.
But the team put a lot of effort into overhauling all the networking systems in the game to make it more efficient with picking the right host and getting people in the best possible game as quickly as possible. So we definitely had a new infrastructure for networking. This will be the largest scale that we've ever put that through.
I should say that prior to the public beta, we had three different rounds of internal beta testing that was just Microsoft employees. We started off generally by doing a lot of smaller Bungie-only rounds where we'd have 40 people playing. Then that graduated up to a larger audience with Microsoft people, about 1,000. Then it moved into an even larger audience with worldwide Microsoft employees to several thousand people.
Then literally over night, we went from over a couple thousand people to doing over a million the first day. We definitely had some problems the first day that were mainly the result of scale. We had some back-end servers and back-end processes that were struggling to keep up. This allowed the engineering team to find and address some bottlenecks. It really stressed our infrastructure and our systems, and it enabled us to find inefficiencies that we were able to improve.
GP: What was involved in beefing up the infrastructure and getting the servers ready for a beta of this scale?
BJ: Our team did a really good job of predicting the worst-case scenario. Halo 3 has had tremendous online usage: huge numbers of people with tons of game uploads and data passing back and forth. We weren't really starting from zero. We had a great foundation to build on, and we've learned a lot over the years. Going into Reach, we had put together a really solid plan and the team was prepared.
In terms of being able to handle the scale, they definitely over-predicted. But what they didn't count on was a couple of real genuine bugs. An issue arose that wasn't so much about there not being enough servers. It was a server cluster that got hung up because of a processing error. It was a bug that wouldn't have come to light if we hadn't had the scale of the beta.
GP: What exactly happened with this server cluster bug?
BJ: Just imagine that there's this huge cluster of servers that sit between your living room and Bungie Xbox Live. And everything that you're doing in the game -- uploading to your fileshare, all the stats at the end of every game -- is uploading to servers that Bungie's running. If any one of those servers or those processes falters, everything starts to get backed up. That's actually what happened on that first Monday.
We had a variety of small issues that all just happened at the same time. But the result was that processing of game uploads slowed to a crawl and with a million people playing games, because we had processors that were offline trying to catch up, they fell further and further behind. The result was that it started taking a very long time to find games in matchmaking. Then eventually you weren't even able to download the data to populate a playlist for matchmaking. And that's because the servers were stuck in this perpetual state of trying to catch up. Once some of those issues were addressed and ironed out, the system rebounded quickly.
GP: How else have you learned from the Halo: Reach beta?
BJ: The biggest benefit was actually just scale. We have a pretty large internal full-time test team. We have a lot of great contingent staff testers. We have the shared testing resources from Microsoft as a publisher. But even if you add every single person up that's formally testing the game, it might be like 100 people. And there's only so many hours that they can devote to the game. So when you have a beta test like this and you have millions of people, we basically got 16 million hours of game play out of this beta test period. There's no way we could have recreated that with traditional test resources.
When you have a beta, you want as big a scale as possible, because ultimately you are trying to emulate what it's going to be like in the first day, week, and month of the game retail launch. As much as we can do to get to that point and to see what that's like now versus in September, it's invaluable to helping everyone on the team be able to react and make sure that we're really ready to have the best experience possible. The more people play, the more people are playing, and the more diverse and unique insights and perspectives you get.
So it's not just about the super hardcore Halo fans that are giving you really nuanced direct feedback about a very specific part of the game. You're getting a much bigger cross section of players who care about different elements that allowed us to get a broader range of feedback and allowed us to talk about things we wouldn't have been able to otherwise. There's definitely things on networking and matchmaking and file-sharing in particular, the ability to upload and tag files in real time, where we can only really test that at a really high scale and feel confident that what we have is going to be working great in September.
GP: There's always the risk that a negative beta experience will lead to negative word of mouth for the game. And with the number of people who took part in the Reach beta, that could have been a lot of bad publicity. Was that ever a concern going into the beta?
BJ: Absolutely. It's terrifying for the whole team. The beta is super-critical for us and we can't do the types of things that we want to do online and all the systems and features we have on Reach without putting it through a large-scale beta, so it was never a question of not doing one.
But you're right. Beta or not, we knew that people were going to be very judgmental and very critical. A lot of people were going to form opinions about the final game based on what the experience was during the beta. It's a slippery slope for us, I think the team did a great job of putting as much polish as possible into the beta. An additional challenge was that, by the time the beta went live, the main code base for the game was already six weeks ahead of what people were playing at home. So in some cases we had bugs that had already been fixed, but they weren't able to be fixed in time for the beta. Unfortunately, that resulted in people having a play experience that was a little bit tarnished by some bugs that we actually had already addressed. But we didn't have the ability to update the beta over the course of its life. We just had to do our best to communicate that we've acknowledged and already addressed this bug.
GP: I'm sure some interesting feedback and player stories came out of the beta. Looking back, anything in particular stand out?
BJ: No one can say that the Halo fan base isn't extremely vocal and extremely passionate. We've come to know that over our years working with our community. It's a testament to the fact that people logged 16 million hours playing this game. That also means that people will go to great lengths to express their opinions to us. That came in the form of a lot of voicemails left throughout the studio. People would dial into our switchboard and just randomly try and reach somebody at their desk and wanted to share their thoughts on what they did or didn't like about this particular weapon or map or some of the game mechanics. There was a lot of vocal feedback, even more so than we anticipated. People went through creative means of trying to get into contact with us to make sure that we heard their feedback.
We would get very opinionated mail, and it's only because people really care. There's people out there that have logged thousands of games of Halo 3 in the past few years. To them Reach was a very different experience, and a lot of that is by design. It was meant to be a new take on the Halo formula: staying true to what really defines Halo, but taking advantages of the liberties we have with this new story and the prequel and trying to find ways to further evolve the gameplay and build on it.
Not everybody received that very well and not everybody was open to that. I would say that was the majority of the vocal noise that we heard. Most of it stemmed from people not responding well to some of the core changes between Halo 3 and Reach. Specifically I'm referring to run speed, jump speed, and this notion of weapons being less efficient the faster you fire them.
I will say that for almost every person that threw a mini tantrum as to why this was the worst thing that ever happened to video games, there was a person who would write an entire essay about why this is the best thing that has ever happened to Halo.
Hearing the designers talk, it makes them feel pretty good when you're about 50-50 split on whether this weapon is too weak or too strong or whether this mechanic is awesome or terrible. That's a pretty good place to be at when you're about middle. And in some of those cases, they were genuine bugs that we've addressed that should alleviate all of the concerns that we've heard from core players.
GP: How will all the feedback generated from the Halo: Reach beta impact the final game?
BJ: A lot of the direct takeaways from the beta -- I would say about half of them were addressed internally in the main build of the game before the beta even launched. The other half were things that have already been internally digested and discussed and tweaked as appropriate and have already been represented in the final game.
Now our focus is really -- in addition to finishing the game, obviously there's a lot more to it than what we showed in the beta -- on trying to communicate out the types of things that we did hear from players and here's how we're going to react or respond to these.
In some cases there were bugs like the melee bug or the issue with the head shots not registering correctly that really did leave some hardcore players with a less than optimal experience. For all they knew, that was the way we wanted the game to be, and that's clearly not the case. Now it's important that we move forward and show the rest of Reach that people haven't seen but also make good on the promises that we've heard in people's feedback. In fact they'll be happy to know that I would say every major "issue" that we heard over the course of the beta has been addressed one way or another.
GP: What were some of the crazier bugs that were found in the course of the beta test?
BJ: The first or second day of the beta, somebody had uploaded a save film where they went to assassinate another player and right when the animation triggered, the player mysteriously ended up in this black room and all the Noble team was standing there next to them. So it seemed like they teleported out of the game space into this weird nether region. That was a weird bug, which it turns out had network implications.
In fact, most of the weird stuff that we saw came about as a result of a host migration in a multiplayer game. It didn't happen very often in the real world , and it's hard to test in smaller population beta tests because its occurrence is so rare. But at this scale, it occurred quite frequently, and we would see some really odd things. Like in invasion games, when the host would drop out and the game would assign to a new host and resume, when they came back in, all sorts of weird things would happen. Now suddenly the Elites would be on defense and the Spartans would be attacking, or suddenly the Elites would all have spartan armor abilities or Spartans would have Elite armor abilities. We saw a couple of different really weird behaviors that would occur that were all results of a game resuming after a host migration. A lot of those we hadn't experienced internally prior to the public beta test.
GP: Now that the beta is over, do you have any last words for the fans before they see the finished product come out in the fall?
BJ: Overall you'll see a much more polished experience. Everything about the game is really coming together in the final stretch of development. Every major issue that has been consistently raised has been addressed in one way or another. Not to say that it's going to be to a degree that everyone out there was wanting.
In the case of something like base run speed and jump height, which for Reach was intentionally meant to be a little different from Halo 3 because you're not playing as Master Chief. These are Spartan-3's. They're a little different in their capabilities and we wanted the game to feel a little different. That said, coming out of the beta, there was so much consistent feedback about how people felt -- the game just felt a little bit too slow to them. It didn't feel quite like Halo, the design team got back together and had some discussion and as a result you're going to see that we have made some tweaks in those areas. I can honestly tell you that before the Beta started, if you had asked anyone on the design team, "Hey, are you guys thinking of changing run speed or jump height," it would have been a very unanimous "no." People were very comfortable with the way the game was implemented and what the design intent was. But strictly based on the feedback of the beta, it was something that warranted additional discussion internally because it came up so much in feedback.
I would also just add that there's lots of stuff that people haven't seen yet. The armory we had and the player awards system in the beta were just the tip of the iceberg. We now have full system implemented commendations that are working across the board as intended. There's a lot more to dive into there. Even the maps themselves have some minor layout tweaks. Just a whole host of various tightenings and tweakings. We just want players to know that we've definitely heard them and their feedback and taken it to heart, and the final game will definitely be better because of that.
Glad to see they're doing something about the bugs. I ran into a bbug once during invasion where I was a Spartan with Elite capabilities and Elite weapons. It was pretty cool.
