Adventure Forums

Adventure Forums (https://adventuregamers.com/archive/forums/)
-   AG Underground - Freeware Adventures (https://adventuregamers.com/archive/forums/ag-underground-freeware-adventures/)
-   -   Designing and playtesting games (https://adventuregamers.com/archive/forums/ag-underground-freeware-adventures/20738-designing-playtesting-games.html)

After a brisk nap 08-21-2007 07:11 AM

Designing and playtesting games
 
Wired has an excellent article on the playtesting of Halo 3. By analyzing thousands of hours of play by "regular" players, they're able to pinpoint where the game is fun and where it isn't.

My background is in usability, user-centered design and human-computer interaction, so this stuff is right up my alley. I've long thought that game companies were underusing playtesting as a way to tweak their designs, and Bungie and Microsoft finally seem to have set up a proper lab and solid tools for this.

In my opinion, adventure games would benefit from this kind of playtesting. And it doesn't necessarily take a lab with one-way mirrors and dozens of cameras. One thing that has always stuck with me is reading about how Ron Gilbert designed Maniac Mansion. The first version was a paper board game, which he could test and tweak without writing a line of computer code in order to ensure that the whole game was entertaining. Because traditional adventure games are pretty simple in their mechanics, it's quite possible to test them on paper, using sketches or printouts of backgrounds, paper dolls for the characters, post-its or tokens for inventory items, and the designer playing the part of the computer and reading the lines from the script.

Even Underground games could take advantage of this kind of testing, which is cheap and easy. And once the game is in development, you can send out test builds to playtesters online (since the game is non-commercial, leaks are not a major concern), with code that logs their actions, and maybe prompts them to rate their enjoyment or the difficulty at various points, and then returns the data to you (with their knowledge, of course).

Does anyone here actually test the games they make on other players? How do you go about it?

Squinky 08-21-2007 10:15 AM

Squee! I'm actually planning on making an AA article in the future about the testing process. I worked in QA a couple of years ago, so I find this subject to be of great importance and personal interest.

So far, I've only allowed one person to play Chivalry in its unfinished state. I did this more to give myself personal motivation than anything else; however, the comments I got back were very helpful in terms of what was enjoyable and what was frustrating, and even inspired me to change/add a couple of things. Back when I was testing TGTTPOACS, I installed a message board to enable my testers to report suggestions as well as bugs, and I was able to decide democratically which suggestions to implement and which ones to leave as-is. Most notably, I added a handful of extra dialogue lines to acknowledge actions that certain people attempted but only received a default response for, and added functionality to allow characters to move more quickly via double-clicking.

I've never thought of using raw data before; I don't think it should be too difficult to create a plugin in one of the major adventure game engines that logs a transcript of sorts detailing a player's actions. (Maybe I'll create one if I'm not too busy/lazy, but somehow I find that doubtful; finishing the game itself is my top priority.) This is, I believe, how IF is tested. Another option would be to ask testers to record themselves playing through Fraps or something, though this would likely be a tad more time-consuming depending on the number of testers one uses.

Still, I have to nitpick about the fact that only measuring raw data isn't really enough in my view. When people play my games, what matters most to me is what they thought and felt about the narrative and the themes derived therefrom, and whether the gameplay helped or hindered the expression of said themes. I think that this can only be measured qualitatively rather than quantitatively, by actually asking playtesters about their impressions. Hence, if I only have time to employ one method, this is definitely the one I'd choose.

Enter the Story 08-22-2007 11:03 PM

Quote:

Originally Posted by After a brisk nap (Post 435545)
Does anyone here actually test the games they make on other players? How do you go about it?

I agree with a poster in indiegamer who advocated iterative development. That is, you are constantly sending out builds for testing and developing based on this. Obviously you have your own ideas and interpret feedback in this light, but ideas are easy: it's desirability that's hard to achieve. If the ideas are strong enough they can survive the process.

I am amazed by the number of people who say "I sent it for testing and even changed one or two thngs." Just one or two? These designers must be geniuses, and mind-reading geniuses, and very wealthy high selling mind reading geniuses, if they only need to change one or two things!

TheTwelve 08-23-2007 01:28 AM

I also have a background in usability and HCI and have applied it liberally during the design and testing phases of my games.

Since my current project is on a much larger scale than my previous games, I'm making sure to have more eyes on the game as I release occasional alpha versions for play testing. I currently have two people outside of the main team who have been looking at the versions and giving feedback. I ask some specific questions to make sure that the interface is intuitive and easy and make sure that the puzzles make sense, and then also ask more broad questions about characterization, dialog, graphics, etc. I've gotten some good feedback and have made some tweaks here and there that will hopefully improve the overall experience.

As far as prototyping puzzles to make sure they're fun/playable, I did a bit of that with Linus. You can play a flash version of the slider-puzzle-ish thing in this post mortem article on my blog. Though really, I was prototyping to make sure it wasn't too difficult (which actually led to the broken panel on the right to easy it up a bit).

I also, of course, have a long period of beta testing when production is complete to iron out any bugs that may have somehow slipped through.

SSH 08-30-2007 06:24 AM

Here I shall put my ISEB certified testing skills into action: ;)

Strictly speaking, alpha versions of products are releases that are tested "on site" rather than given out to the wider world and betas are tested "off site".

Planning for testing has to be an integral part of any software design. Ideally, each subcomponent of any software should be thoroughly tested on its own before it is integrated into a larger system, but this can be hard to do with games.

After a brisk nap 08-30-2007 07:40 AM

Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.

Bug testing for games is pretty much the same as bug testing for any other software. Playtesting is more unique, more like a cross between usability tests and the test screenings of films.

Squinky 08-30-2007 10:01 AM

Quote:

Originally Posted by After a brisk nap (Post 437163)
Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.

I tend to combine the two when getting people to test my games; I ask my testers what they think of the experience as a whole, and also ask them to report any bugs they find along the way. Do you find that there are problems with such an approach?

After a brisk nap 08-31-2007 10:24 AM

If it works for you, then that's great. It won't get you published at CHI, but user testing should be a pragmatic activity.

I think that playtesters should absolutely also report bugs, and that you can get useful information about the gameplay from bugtesters. In principle, I can think of a few potential problems with combining the two activities:

- User testing (in various forms) should ideally go on throughout the design process. OK, so bug testing should also start as soon as you have any code, but beta testing usually take place near the end of development. Make sure you don't leave play testing until it's too late to fix fundamental problems in the design.

- The people who are good at bug testing are not always/usually the same people who are best for play testing. Play testers should resemble your audience as much as possible, while good bug testers are people who exhaustively try everything, no matter how strange. In particular, professional testers (who work on commercial games, playing every build again and again) do not make good play testers, because they are far more expert than regular players.

- Similarly, when bug testing a game, the experience is not necessarily the same as just playing it.

- Asking your play testers about bugs, especially when the game is in an early stage, runs a high risk of distracting them from the essential elements that you're trying to gather feedback on. For instance, they may complain about missing animations and placeholder graphics when you're just trying to find out what they think about the puzzles.

- You get far more useful play testing data in person, by direct observation. This is not as important for bug testing, so there's an argument for using different methods to collect each type of information.

In general, you have to consider your return on investment. For a major game, especially a commercial effort, you should probably spend some time on proper play testing. For a one-person, Underground title developed in your spare time, informal feedback is likely to get you some of the same benefits while taking a lot less effort. (A big part of the reason for this is that the audience for Underground titles are pretty savvy: mostly experienced adventure gamers, and many involved in the Underground scene themselves. That means they can provide good feedback by inspection. Also, they're easy to get in touch with online, but relatively hard to reach in real life, unless you attend the various AGS meetups. It's not like you can just stop people on the street and expect to find Trilby fans.)

However, there is really no substitute for seeing people actually play your game. If you have friends or family who play adventure games, you should exploit them mercilessly for the sake of your art. Failing that, developers who are still at university (or have university contacts) have access to a large population of cheaply bribed subjects. Just make sure to screen for the kind of players you want (probably people familiar with adventures).

splat44 09-22-2007 01:52 PM

Quote:

Originally Posted by After a brisk nap (Post 437163)
Yeah, but playtesting is really completely different from alpha/beta testing. The purpose of the latter is to find bugs. The purpose of the former is to figure out if there are parts of the design that should be changed to make the game more fun.

Bug testing for games is pretty much the same as bug testing for any other software. Playtesting is more unique, more like a cross between usability tests and the test screenings of films.


I don't see any differences between those testing phases as the purpose is for tester to find bugs.

Squinky 09-22-2007 03:13 PM

LOL. RTFM, n00b.

After a brisk nap 09-22-2007 03:17 PM

Well, the difference is that playtesters aren't looking for bugs. They're reporting their experience so that the developer can improve the design. The article linked to in the first post gives some examples of how playtesting is used to tweak the level design in an action game.

MikeRozak 09-22-2007 08:01 PM

I'm in the process of creating a toolkit and game: www.CircumReality.com . It's still a ways from being done, but a working version is available from the site.

I have implimented the following usability testing hooks:

1) It's multiplayer and everything is logged.

2) Not only is everything logged, but it's easy to keep metrics, like the average time to complete a task.

3) An administrator can "spy" on a player and see what they're seeing on the screen. Unfortunately, you can't see a video of the player's facial expressions or read their minds.

After a brisk nap 09-23-2007 06:58 AM

That sounds very useful. When people do this kind of testing, we usually consider it very important that the participants in the test know that they're under observation. It's called "informed consent," and in most cases it's considered necessary for ethical research. I would urge you to follow this principle.

Lucien21 09-23-2007 09:26 AM

I think playtesting can be a powerful tool for good or evil and it probably depends on the strength of the design and the vision behind the game in the first place.

I watched a decent interview with the designers of Bioshock who talked about their initial playtesting. They stated that when asked to playtest a lot of people suddenly become critics about the design of the game. I.e I think this character looks wrong or this colour should be blue etc.

I think they are more helpful for useability rather than design. i.e Is the interface easy to use, are they getting enough information at the right times to proceed etc. Obviously in bigger companies it's better to be watching the people play the game as then the developer gets a sense of what is working and what isn't by watching the players. However this probably is going to happen in small or independant companies.

Other information will come out of it, but I think it should mostly be ignored if it is design related. (Unless it is universal)

Too many cooks spoil the broth.

Marek 09-23-2007 09:40 AM

Valve is another company that spends a lot of time on playtesting, and they know what to do and not do with the feedback they get from it, which really shows in their amazing games.

I think a few terms are getting confused here though...

QA = Quality Assurance, i.e. making sure the game doesn't break, doesn't have bugs, has sensible menu's, conforms to platform holder and publisher specifications, has correct subtitles, etc. etc.
Playtesting = watching people play as a means of improving the gameplay itself
Prototyping = creating simplified versions of a game, for instance as a board game, so the designer can cheaply and effectively work on the mechanics without being distracted by production issues

Paper prototypes are huge fun to make, by the way. I've done GDC workshops that involved a lot of paper prototyping, and have applied these methods in my game design work on a few occasions, and it's really fun to see how you can abstract down a complex computer game to a very pure and whole board game. At least, if you are able to do that, you know the core of the computer game will be pretty solid.

I agree the game industry could use more playtesting. I also think it could use less focus testing (or none at all). Ask people what they want and you're only going to find out what they want that they already know, not what they want but don't yet know they want...

After a brisk nap 09-24-2007 03:29 PM

Quote:

Originally Posted by Lucien21 (Post 441015)
I watched a decent interview with the designers of Bioshock who talked about their initial playtesting. They stated that when asked to playtest a lot of people suddenly become critics about the design of the game. I.e I think this character looks wrong or this colour should be blue etc.

Well, that's why you can't just ask players what your game should be like. If they knew how to design games, they'd be making their own.

User testing is helpful to find current problems or areas with room for improvement. It doesn't tell you what the solutions are. Even when participants tell you what they think the solution should be, that doesn't necessarily mean they're right.

Also, asking the right questions to get at the kind of information you're interested in takes skill and training. If the Bioshock devs mainly got comments about art direction, maybe they weren't doing it right. If you ask people about the "design" of the game, for instance, I'd guess a lot of them won't realize that you're talking about how it plays, not primarily how it looks.

That said, user feedback is always going to include irrelevant information. People can get incredibly hung up on tiny details that have nothing to do with what you're investigating, like color choice. You learn to smile, nod, and gently change the subject.

Lucien21 09-24-2007 03:31 PM

I don't think they asked about the design, but as you say these things are usually volunteered.

Trumgottist 09-25-2007 12:27 AM

Quote:

Originally Posted by After a brisk nap (Post 441204)
User testing is helpful to find current problems or areas with room for improvement. It doesn't tell you what the solutions are. Even when participants tell you what they think the solution should be, that doesn't necessarily mean they're right.

It's still a good idea to listen to the suggestions. In making Frasse, I had a lot of help from my testers. Bug hunting, while they found some of those too, where their least valuable contribution.

stepurhan 09-25-2007 04:50 AM

Quote:

Originally Posted by Trumgottist (Post 441257)
It's still a good idea to listen to the suggestions. In making Frasse, I had a lot of help from my testers. Bug hunting, while they found some of those too, where their least valuable contribution.

We live to serve master. :D

Looking from the other side I think testers have a certain responsibility to point out things that don't work or look odd. However, they equally have to appreciate that, unless something really is a problem, it may not be worth changing. By the later stages of testing it is quite possible that any change to resolve a minor niggle will actually require major rewriting of the code. Time is money (or equally precious for itself to a freeware developer) Whilst you might not like the shade of blue is it a deal-breaker?

Ninja Dodo 09-26-2007 02:34 PM

One interesting approach that is used is to playtest the entire game from start to finish regularly (with new people each time), because if you play parts in isolation you never know whether the player would have acted the same way had they had the prior experience of the rest of the game. A puzzle that seems extremely hard on its own might be a piece of cake given preceding 'training'...


All times are GMT -8. The time now is 01:49 AM.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Design & Logo Copyright ©1998 - 2017, Adventure Gamers®.
All posts by users and Adventure Gamers staff members are property of their original author and don't necessarily represent the opinion or editorial stance of Adventure Gamers.