Designing and playtesting games - Adventure Forums
You are viewing an archived version of the site which is no longer maintained.
Go to the current live site or the Adventure Gamers forums
Adventure Gamers



 
 
LinkBack Thread Tools
Old 08-21-2007, 07:11 AM   #1
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default Designing and playtesting games

Wired has an excellent article on the playtesting of Halo 3. By analyzing thousands of hours of play by "regular" players, they're able to pinpoint where the game is fun and where it isn't.

My background is in usability, user-centered design and human-computer interaction, so this stuff is right up my alley. I've long thought that game companies were underusing playtesting as a way to tweak their designs, and Bungie and Microsoft finally seem to have set up a proper lab and solid tools for this.

In my opinion, adventure games would benefit from this kind of playtesting. And it doesn't necessarily take a lab with one-way mirrors and dozens of cameras. One thing that has always stuck with me is reading about how Ron Gilbert designed Maniac Mansion. The first version was a paper board game, which he could test and tweak without writing a line of computer code in order to ensure that the whole game was entertaining. Because traditional adventure games are pretty simple in their mechanics, it's quite possible to test them on paper, using sketches or printouts of backgrounds, paper dolls for the characters, post-its or tokens for inventory items, and the designer playing the part of the computer and reading the lines from the script.

Even Underground games could take advantage of this kind of testing, which is cheap and easy. And once the game is in development, you can send out test builds to playtesters online (since the game is non-commercial, leaks are not a major concern), with code that logs their actions, and maybe prompts them to rate their enjoyment or the difficulty at various points, and then returns the data to you (with their knowledge, of course).

Does anyone here actually test the games they make on other players? How do you go about it?
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 08-21-2007, 10:15 AM   #2
Unreliable Narrator
 
Squinky's Avatar
 
Join Date: Apr 2004
Location: Le Canada
Posts: 9,873
Send a message via AIM to Squinky Send a message via MSN to Squinky
Default

Squee! I'm actually planning on making an AA article in the future about the testing process. I worked in QA a couple of years ago, so I find this subject to be of great importance and personal interest.

So far, I've only allowed one person to play Chivalry in its unfinished state. I did this more to give myself personal motivation than anything else; however, the comments I got back were very helpful in terms of what was enjoyable and what was frustrating, and even inspired me to change/add a couple of things. Back when I was testing TGTTPOACS, I installed a message board to enable my testers to report suggestions as well as bugs, and I was able to decide democratically which suggestions to implement and which ones to leave as-is. Most notably, I added a handful of extra dialogue lines to acknowledge actions that certain people attempted but only received a default response for, and added functionality to allow characters to move more quickly via double-clicking.

I've never thought of using raw data before; I don't think it should be too difficult to create a plugin in one of the major adventure game engines that logs a transcript of sorts detailing a player's actions. (Maybe I'll create one if I'm not too busy/lazy, but somehow I find that doubtful; finishing the game itself is my top priority.) This is, I believe, how IF is tested. Another option would be to ask testers to record themselves playing through Fraps or something, though this would likely be a tad more time-consuming depending on the number of testers one uses.

Still, I have to nitpick about the fact that only measuring raw data isn't really enough in my view. When people play my games, what matters most to me is what they thought and felt about the narrative and the themes derived therefrom, and whether the gameplay helped or hindered the expression of said themes. I think that this can only be measured qualitatively rather than quantitatively, by actually asking playtesters about their impressions. Hence, if I only have time to employ one method, this is definitely the one I'd choose.
__________________
Squinky is always right, but only for certain values of "always" and "right".
Squinky is offline  
Old 08-22-2007, 11:03 PM   #3
I turn novels into games
 
Enter the Story's Avatar
 
Join Date: Jun 2004
Location: Scotland
Posts: 307
Default

Quote:
Originally Posted by After a brisk nap View Post
Does anyone here actually test the games they make on other players? How do you go about it?
I agree with a poster in indiegamer who advocated iterative development. That is, you are constantly sending out builds for testing and developing based on this. Obviously you have your own ideas and interpret feedback in this light, but ideas are easy: it's desirability that's hard to achieve. If the ideas are strong enough they can survive the process.

I am amazed by the number of people who say "I sent it for testing and even changed one or two thngs." Just one or two? These designers must be geniuses, and mind-reading geniuses, and very wealthy high selling mind reading geniuses, if they only need to change one or two things!

Last edited by Enter the Story; 08-22-2007 at 11:08 PM.
Enter the Story is offline  
Old 08-23-2007, 01:28 AM   #4
Member
 
TheTwelve's Avatar
 
Join Date: Sep 2004
Posts: 89
Default

I also have a background in usability and HCI and have applied it liberally during the design and testing phases of my games.

Since my current project is on a much larger scale than my previous games, I'm making sure to have more eyes on the game as I release occasional alpha versions for play testing. I currently have two people outside of the main team who have been looking at the versions and giving feedback. I ask some specific questions to make sure that the interface is intuitive and easy and make sure that the puzzles make sense, and then also ask more broad questions about characterization, dialog, graphics, etc. I've gotten some good feedback and have made some tweaks here and there that will hopefully improve the overall experience.

As far as prototyping puzzles to make sure they're fun/playable, I did a bit of that with Linus. You can play a flash version of the slider-puzzle-ish thing in this post mortem article on my blog. Though really, I was prototyping to make sure it wasn't too difficult (which actually led to the broken panel on the right to easy it up a bit).

I also, of course, have a long period of beta testing when production is complete to iron out any bugs that may have somehow slipped through.
__________________
TheTwelve is offline  
Old 08-30-2007, 06:24 AM   #5
SSH
Super Scottish Hero
 
SSH's Avatar
 
Join Date: Feb 2006
Location: Edinburgh, UK
Posts: 2,872
Default

Here I shall put my ISEB certified testing skills into action:

Strictly speaking, alpha versions of products are releases that are tested "on site" rather than given out to the wider world and betas are tested "off site".

Planning for testing has to be an integral part of any software design. Ideally, each subcomponent of any software should be thoroughly tested on its own before it is integrated into a larger system, but this can be hard to do with games.
SSH is offline  
Old 08-30-2007, 07:40 AM   #6
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default

Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.

Bug testing for games is pretty much the same as bug testing for any other software. Playtesting is more unique, more like a cross between usability tests and the test screenings of films.
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 08-30-2007, 10:01 AM   #7
Unreliable Narrator
 
Squinky's Avatar
 
Join Date: Apr 2004
Location: Le Canada
Posts: 9,873
Send a message via AIM to Squinky Send a message via MSN to Squinky
Default

Quote:
Originally Posted by After a brisk nap View Post
Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.
I tend to combine the two when getting people to test my games; I ask my testers what they think of the experience as a whole, and also ask them to report any bugs they find along the way. Do you find that there are problems with such an approach?
__________________
Squinky is always right, but only for certain values of "always" and "right".
Squinky is offline  
Old 08-31-2007, 10:24 AM   #8
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default

If it works for you, then that's great. It won't get you published at CHI, but user testing should be a pragmatic activity.

I think that playtesters should absolutely also report bugs, and that you can get useful information about the gameplay from bugtesters. In principle, I can think of a few potential problems with combining the two activities:

- User testing (in various forms) should ideally go on throughout the design process. OK, so bug testing should also start as soon as you have any code, but beta testing usually take place near the end of development. Make sure you don't leave play testing until it's too late to fix fundamental problems in the design.

- The people who are good at bug testing are not always/usually the same people who are best for play testing. Play testers should resemble your audience as much as possible, while good bug testers are people who exhaustively try everything, no matter how strange. In particular, professional testers (who work on commercial games, playing every build again and again) do not make good play testers, because they are far more expert than regular players.

- Similarly, when bug testing a game, the experience is not necessarily the same as just playing it.

- Asking your play testers about bugs, especially when the game is in an early stage, runs a high risk of distracting them from the essential elements that you're trying to gather feedback on. For instance, they may complain about missing animations and placeholder graphics when you're just trying to find out what they think about the puzzles.

- You get far more useful play testing data in person, by direct observation. This is not as important for bug testing, so there's an argument for using different methods to collect each type of information.

In general, you have to consider your return on investment. For a major game, especially a commercial effort, you should probably spend some time on proper play testing. For a one-person, Underground title developed in your spare time, informal feedback is likely to get you some of the same benefits while taking a lot less effort. (A big part of the reason for this is that the audience for Underground titles are pretty savvy: mostly experienced adventure gamers, and many involved in the Underground scene themselves. That means they can provide good feedback by inspection. Also, they're easy to get in touch with online, but relatively hard to reach in real life, unless you attend the various AGS meetups. It's not like you can just stop people on the street and expect to find Trilby fans.)

However, there is really no substitute for seeing people actually play your game. If you have friends or family who play adventure games, you should exploit them mercilessly for the sake of your art. Failing that, developers who are still at university (or have university contacts) have access to a large population of cheaply bribed subjects. Just make sure to screen for the kind of players you want (probably people familiar with adventures).
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 09-22-2007, 01:52 PM   #9
Senior Member
 
Join Date: Jun 2004
Posts: 113
Default

Quote:
Originally Posted by After a brisk nap View Post
Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.

Bug testing for games is pretty much the same as bug testing for any other software. Playtesting is more unique, more like a cross between usability tests and the test screenings of films.

I don't see any differences between those testing phases as the purpose is for tester to find bugs.
splat44 is offline  
Old 09-22-2007, 03:13 PM   #10
Unreliable Narrator
 
Squinky's Avatar
 
Join Date: Apr 2004
Location: Le Canada
Posts: 9,873
Send a message via AIM to Squinky Send a message via MSN to Squinky
Default

LOL. RTFM, n00b.
__________________
Squinky is always right, but only for certain values of "always" and "right".
Squinky is offline  
Old 09-22-2007, 03:17 PM   #11
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default

Well, the difference is that playtesters aren't looking for bugs. They're reporting their experience so that the developer can improve the design. The article linked to in the first post gives some examples of how playtesting is used to tweak the level design in an action game.
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 09-22-2007, 08:01 PM   #12
Junior Member
 
Join Date: Dec 2006
Posts: 15
Default

I'm in the process of creating a toolkit and game: www.CircumReality.com . It's still a ways from being done, but a working version is available from the site.

I have implimented the following usability testing hooks:

1) It's multiplayer and everything is logged.

2) Not only is everything logged, but it's easy to keep metrics, like the average time to complete a task.

3) An administrator can "spy" on a player and see what they're seeing on the screen. Unfortunately, you can't see a video of the player's facial expressions or read their minds.
MikeRozak is offline  
Old 09-23-2007, 06:58 AM   #13
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default

That sounds very useful. When people do this kind of testing, we usually consider it very important that the participants in the test know that they're under observation. It's called "informed consent," and in most cases it's considered necessary for ethical research. I would urge you to follow this principle.
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 09-23-2007, 09:26 AM   #14
Psychonaut
 
Lucien21's Avatar
 
Join Date: Jan 2004
Location: Edinburgh
Posts: 5,114
Default

I think playtesting can be a powerful tool for good or evil and it probably depends on the strength of the design and the vision behind the game in the first place.

I watched a decent interview with the designers of Bioshock who talked about their initial playtesting. They stated that when asked to playtest a lot of people suddenly become critics about the design of the game. I.e I think this character looks wrong or this colour should be blue etc.

I think they are more helpful for useability rather than design. i.e Is the interface easy to use, are they getting enough information at the right times to proceed etc. Obviously in bigger companies it's better to be watching the people play the game as then the developer gets a sense of what is working and what isn't by watching the players. However this probably is going to happen in small or independant companies.

Other information will come out of it, but I think it should mostly be ignored if it is design related. (Unless it is universal)

Too many cooks spoil the broth.
__________________
I'm not insane, my mother had me tested!
Lucien21 is offline  
Old 09-23-2007, 09:40 AM   #15
Puts the 'e' in Mark
 
Marek's Avatar
 
Join Date: Sep 2003
Posts: 3,138
Default

Valve is another company that spends a lot of time on playtesting, and they know what to do and not do with the feedback they get from it, which really shows in their amazing games.

I think a few terms are getting confused here though...

QA = Quality Assurance, i.e. making sure the game doesn't break, doesn't have bugs, has sensible menu's, conforms to platform holder and publisher specifications, has correct subtitles, etc. etc.
Playtesting = watching people play as a means of improving the gameplay itself
Prototyping = creating simplified versions of a game, for instance as a board game, so the designer can cheaply and effectively work on the mechanics without being distracted by production issues

Paper prototypes are huge fun to make, by the way. I've done GDC workshops that involved a lot of paper prototyping, and have applied these methods in my game design work on a few occasions, and it's really fun to see how you can abstract down a complex computer game to a very pure and whole board game. At least, if you are able to do that, you know the core of the computer game will be pretty solid.

I agree the game industry could use more playtesting. I also think it could use less focus testing (or none at all). Ask people what they want and you're only going to find out what they want that they already know, not what they want but don't yet know they want...
Marek is offline  
Old 09-24-2007, 03:29 PM   #16
Elegantly copy+pasted
 
After a brisk nap's Avatar
 
Join Date: Jan 2005
Posts: 1,773
Default

Quote:
Originally Posted by Lucien21 View Post
I watched a decent interview with the designers of Bioshock who talked about their initial playtesting. They stated that when asked to playtest a lot of people suddenly become critics about the design of the game. I.e I think this character looks wrong or this colour should be blue etc.
Well, that's why you can't just ask players what your game should be like. If they knew how to design games, they'd be making their own.

User testing is helpful to find current problems or areas with room for improvement. It doesn't tell you what the solutions are. Even when participants tell you what they think the solution should be, that doesn't necessarily mean they're right.

Also, asking the right questions to get at the kind of information you're interested in takes skill and training. If the Bioshock devs mainly got comments about art direction, maybe they weren't doing it right. If you ask people about the "design" of the game, for instance, I'd guess a lot of them won't realize that you're talking about how it plays, not primarily how it looks.

That said, user feedback is always going to include irrelevant information. People can get incredibly hung up on tiny details that have nothing to do with what you're investigating, like color choice. You learn to smile, nod, and gently change the subject.
__________________
Please excuse me. I've got to see a man about a dog.
After a brisk nap is offline  
Old 09-24-2007, 03:31 PM   #17
Psychonaut
 
Lucien21's Avatar
 
Join Date: Jan 2004
Location: Edinburgh
Posts: 5,114
Default

I don't think they asked about the design, but as you say these things are usually volunteered.
__________________
I'm not insane, my mother had me tested!
Lucien21 is offline  
Old 09-25-2007, 12:27 AM   #18
Game Creator Hobbyist
 
Trumgottist's Avatar
 
Join Date: Nov 2003
Location: Stockholm (or Gotland)
Posts: 2,609
Default

Quote:
Originally Posted by After a brisk nap View Post
User testing is helpful to find current problems or areas with room for improvement. It doesn't tell you what the solutions are. Even when participants tell you what they think the solution should be, that doesn't necessarily mean they're right.
It's still a good idea to listen to the suggestions. In making Frasse, I had a lot of help from my testers. Bug hunting, while they found some of those too, where their least valuable contribution.
__________________
Play my game: Frasse and the Peas of Kejick. The Special Edition is now available! (Mac OS X or Windows.)
Trumgottist is offline  
Old 09-25-2007, 04:50 AM   #19
Freeware Co-ordinator
 
stepurhan's Avatar
 
Join Date: Oct 2003
Location: South East England.
Posts: 7,309
Default

Quote:
Originally Posted by Trumgottist View Post
It's still a good idea to listen to the suggestions. In making Frasse, I had a lot of help from my testers. Bug hunting, while they found some of those too, where their least valuable contribution.
We live to serve master.

Looking from the other side I think testers have a certain responsibility to point out things that don't work or look odd. However, they equally have to appreciate that, unless something really is a problem, it may not be worth changing. By the later stages of testing it is quite possible that any change to resolve a minor niggle will actually require major rewriting of the code. Time is money (or equally precious for itself to a freeware developer) Whilst you might not like the shade of blue is it a deal-breaker?
__________________
No Nonsense Nonsonnets #43

Cold Topic

A thread most controversial, that’s what I want to start
Full of impassioned arguments, of posting from the heart
And for this stimulation all will be thankful to me
On come on everybody it won’t work if you agree
stepurhan is offline  
Old 09-26-2007, 02:34 PM   #20
Senior Member
 
Ninja Dodo's Avatar
 
Join Date: Sep 2003
Posts: 2,459
Default

One interesting approach that is used is to playtest the entire game from start to finish regularly (with new people each time), because if you play parts in isolation you never know whether the player would have acted the same way had they had the prior experience of the rest of the game. A puzzle that seems extremely hard on its own might be a piece of cake given preceding 'training'...
Ninja Dodo is offline  
 



Thread Tools

 


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.