Game testing

Part of a series on:
Video game industry
Activities/jobs Game art design Game design Game development / Game developer Game modification Game producer Game programming / Game programmer Game publisher Game studies Game testing Game journalism Level design
Types Adult game Casual game Indie game Non-game Interactive art Interactive movie Nonviolent video game Serious game Advergame Art game Edugame Exergame Traditional game
Topics Digital rights management (DRM) Outsourcing Software license Software license agreement (EULA)
Lists List of indie game developers List of video game developers List of video game industry people List of video game publishers List of video game websites
v · d · e

Game testing, a subset of game development, is a software testing process for quality control of video games.^[1][2][3] The primary function of game testing is the discovery and documentation of software defects (aka bugs). Interactive entertainment software testing is a highly technical field requiring computing expertise, analytic competence, critical evaluation skills, and endurance.^[4][5]

History

In the early days of computer and video games, the developer was in charge of all the testing. No more than one or two testers were required due to the limited scope of the games. In some cases, the programmers could handle all the testing.^{[citation needed]}

As games become more complex, a larger pool of QA resources, called "Quality Assessment" or "Quality Assurance" is necessary. Most publishers employ a large QA staff for testing various games from different developers. Despite the large QA infrastructure most publishers have, many developers retain a small group of testers to provide on-the-spot QA.

So now most game developers rely on their highly technical and game savvy testers to find gliches and 'bugs' in either the programming code or graphic layers. Game testers usually have a background playing a variety of different games on a multitude of platforms. They must be able to notate and reference any problems they find in detailed reports, meet deadlines with assignments and have the skill level to complete the game titles on their most difficult settings. Most of the time the position of game tester is a highly stressful and competitive position with little pay yet is highly sought after for it serves as a doorway into a rapidly growing industry.^{[citation needed]}

A common misconception is that all game testers enjoy alpha or beta version of the game and report occasionally found bugs.^[5] In contrast, game testing is highly focused on finding bugs using established and often tedious methodologies before alpha version.

Overview

Quality assurance is a critical component in game development, though the video game industry does not have a standard methodology. Instead developers and publishers have their own methods. Small developers do not have QA staff, however large companies may employ QA teams full-time. High-profile commercial games are professionally and efficiently tested by publisher QA department.^[6]

Testing starts as soon as first code is written and increases as the game progresses towards completion.^[7][8] The main QA team will monitor the game from its first submission to the QA until as late as post-production.^[8] Early in the game development process the testing team is small and focuses on daily feedback for new code. As the game approaches alpha stage, more team members are employed and test plan is written. Sometimes features that are not bugs are reported as bugs and sometimes programming team fails to fix issues first time around.^[9] A good bug-reporting system may help the programmers work efficiently. As the projects enters beta stage, the testing team will have clear assignments for each day. Tester feedback may determine final decisions of exclusion or inclusion of final features. Introducing previously uninvolved testers with fresh perspective may help identify new bugs.^[8][10] At this point the lead tester communicates with the producer and department heads daily.^[11] If the developer has external publisher, then coordination with publisher's QA team starts. For console games, a build for console company QA team is sent. Beta testing may involve volunteers, for example, if the game is multiplayer.^[10]

Testers receive scheduled uniquely identifiable game builds^[10] from the developers.^{[citation needed]} The game is play-tested and testers note any uncovered errors. These may range from bugs to art glitches to logic errors and level bugs. Testing requires creative gameplay to discover often subtle bugs. Some bugs are easy to document, but many require detailed description so a developer can replicate or find the bug. Testers implement concurrency control to avoid logging bugs multiple times.^{[citation needed]} Many video game companies separate technical requirement testing from functionality testing altogether since a different testing skillset is required.^[5]

If a video game development enters crunch time before a deadline, the game-test team is required to test late-added features and content without delay. During this period staff from other departments may contribute to the testing—especially in multiplayer games.^{[citation needed]}

Most companies rank bugs according to an estimate of their severity:^[12]

• A bugs are critical bugs that prevent the game from being shipped, for example, they may crash the game.^[10]
• B bugs are essential problems that require attention, however the game may still be playable. Multiple B bugs are equally severe to an A bug.^[10]
• C bugs are small and obscure problems, often in form of recommendation rather than bugs.^[11]

Game tester

A game tester is a member of a development team who performs game testing.

Roles

The organization of staff differs between organizations; a typical company may employ the following roles associated with testing disciplines:

• Game producers are responsible for setting testing deadlines in coordination with marketing and quality assurance.^[13] They also manage many items outside of game testing, relating to the overall production of a title. Their approval is typically required for final submission or "gold" status.^{[citation needed]}

• Lead tester, test lead^[9] or QA lead^[6] is the person responsible for the game working correctly^[9] and managing bug lists.^[10] A lead tester manages the QA staff.^[6] Lead tester works closely with designers and programmers, especially towards the end of the project. Lead tester is responsible for tracking bug reports and managing that they are fixed.^[9] They are also responsible that QA teams produce formal and complete reports.^[10] This includes discarding duplicate and erroneous bug reports, as well as requesting clarifications.^[6] As the game nears alpha and beta stages, lead tester brings more testers into the team, coordinates with external testing teams and works with management and producers.^[12] Some companies may prevent the game going gold until lead tester approves it.^[11] Lead testers are also typically responsible for compiling representative samples of game footage for submission to regulatory bodies such as the ESRB and PEGI.^{[citation needed]} A lead tester is often an aspiring designer or producer.^[6]

• Testers are responsible for checking that the game works, is easy to use, has actions that make sense, and contains fun gameplay.^[11] Testers need to write accurate and specific bug reports, and if possible providing descriptions of how the bug can be reproduced.^[14] Testers may be assigned to a single game during its entire production, or brought onto other projects as demanded by the department's schedule and specific needs.

• SDET (Software Development Engineer in Test) or Technical Testers are responisible for building automated test cases and frameworks as well as managing complex test problems such as overall game performance and security. These individuals usually have strong software development skills but with a focus on writing software which exposes defects in other applications. Specifc roles and duties will vary between studios. Many games are developed with out any Technical Testers.

Employment

Game QA is less technical than general software QA. Many game testers require little experience and sometimes only a high school diploma and no technical expertise will suffice.^[6][15] Game testing is normally a full-time job for experienced testers,^[16] however many employees are hired as temporary staff,^[2][17] such as beta testers. In some cases, testers employed by a publisher may be sent to work at the developer's site. The most aggressive recruiting season is late summer/early autumn^{[citation needed]}, as this is the start of the crunch period for games to be finished and shipped in time for the holiday season.

Some games studios are starting to take a more technical approach to game QA that is more inline with traditional software testing. Technical Test positions are still fairly rare throughout the industry but these jobs are often full time positions with long term career paths and require a 4 year computer science degree and signifcant expereince with test automation.

Some testers use the job as a stepping stone in the game industry.^[15][3] QA résumés, which display non-technical skill sets, tend towards management, then to marketing or production.^{[citation needed]} Applicants for programming, art, or design positions need to demonstrate technical skills in these areas.^[18]

Compensation

Game testing personnel are usually paid hourly (around US$10–12 an hour). Testing management is usually more lucrative, and requires experience and often a college education. An annual survey found that testers earn an average of $39k annually. Testers with less than three years experience earn an average of US$25k while testers with over three years experience earn US$43k. Testing leads, with over six years experience, earn on an average of US$71k a year.^[19] Typically, they will make $35-45k with less experience.^{[citation needed]} Some employers offer bonuses for the number of bugs found.

Process

A typical bug report progression of testing process is seen below:

• Identification. Incorrect program behaviour is analyzed and identified as a bug.

• Reporting. The bug is reported to the developers using a defect tracking system. The circumstances of the bug and steps to reproduce are included in the report. Developers may request additional documentation such as a real-time video of the bug's manifestation.

• Analysis. The developer responsible for the bug, such as an artist, programmer or game designer checks the malfunction. This is outside the scope of game tester duties, although inconsistencies in the report may require more information or evidence from the tester.

• Verification. After the developer fixes the issue, the tester verifies that the bug no longer occurs. Not all bugs are addressed by the developer, for example, some bugs may be claimed as features (expressed as "NAB" or "not a bug"), and may also be "waived" (given permission to be ignored) by producers, game designers, or even lead testers, according to company policy.

Methodology

Main article: Software testing

There is no industry standard method for game testing, and most methodologies are developed by individual video game developers and publishers. Methodologies are continuously refined and may differ for different types of games (for example, the methodology for testing a MMORPG will be different from testing a casual game). Many methods, such as unit testing, are borrowed directly from general software testing techniques. Outlined below are the most important methodologies, specific to video games.

• Functionality testing is most commonly associated with the phrase "game testing", as it entails playing the game in some form. Functionality testing does not require extensive technical knowledge. Functionality testers look for general problems within the game itself or its user interface, such as stability issues, game mechanic issues, and game asset integrity.

• Compliance testing is the reason for the existence of game testing labs.^{[clarification needed]} First-party licensors for console platforms have strict technical requirements titles licensed for their platforms. For example, Sony publishes a Technical Requirements Checklist (TRC), Microsoft publishes Technical Certification Requirements (TCR), and Nintendo publishes a set of "guidelines" (Lotcheck). Some of these requirements are highly technical and fall outside the scope of game testing. Other parts, most notably the formatting of standard error messages, handling of memory card data, and handling of legally trademarked and copyrighted material, are the responsibility of the game testers. Even a single violation in submission for license approval may have the game rejected, possibly incurring additional costs in further testing and resubmission. In addition, the delay may cause the title to miss an important launch window, potentially costing the publisher even larger sums of money.

The requirements are proprietary documents released to developers and publishers under confidentiality agreements. They are not available for the general public to review, although familiarity with these standards is considered a valuable skill to have as a tester.^{[citation needed]}

Compliance may also refer to regulatory bodies such as the ESRB and PEGI, if the game targets a particular content rating. Testers must report objectionable content that may be inappropriate for the desired rating. Similar to licensing, games that do not receive the desired rating must be re-edited, retested, and resubmitted at additional cost.

• Compatibility testing is normally required for PC titles, nearing the end of development as much of the compatibility depends on the final build of the game.^{[citation needed]} Often two rounds of compatibility tests are done - early in beta to allow time for issue resolution, and late in beta or during release candidate.^{[citation needed]} Compatibility testing team test major functionality of the game on various configurations of hardware. Usually a list of commercially important hardware is supplied by the publisher.^[8]

Compatibility testing ensures that the game runs on different configurations of hardware and software. The hardware encompasses brands of different manufacturers and assorted input peripherals such as gamepads and joysticks.^{[citation needed]}

The testers also evaluate performance and results are used for game's advertised minimum system requirements. Compatibility or performance issues may be either fixed by the developer or, in case of legacy hardware and software, support may be dropped.

• Localization testing act as in-game text editors.^[2] Although general text issues are a part of functionality testing, QA departments may employ dedicated localization testers. In particular, early Japanese game translations were rife with Engrish, and in recent years localization testers are employed to make technical corrections and review translation work of game scripts^[20] - catalogued collections of all the in-game text. Testers native to the region where a game is marketed may be employed to ensure the accuracy and quality of a game's localization.^[8]

• Soak testing, in the context of video games, involves leaving the game running for prolonged periods time in various modes of operation, such as idling, paused, or at the title screen. This testing requires no user interaction beyond initial setup, and is usually managed by lead testers. Automated tools may be used for simulating repetitive actions, such mouse clicks. Soaking can detect memory leaks or rounding errors that manifest only over time. Soak tests are one of the compliance requirements.^{[citation needed]}

• Beta testing is done during beta stage of development. Often this refers to the first publicly available version of a game. Public betas are effective because thousands of fans may find bugs that the developer's testers did not.

• Regression testing is performed once a bug has been fixed by the programmers. QA checks to see whether the bug is still there (regression) and then runs similar tests to see whether the fix broke something else. That second stage is often called "halo testing"^{[citation needed]}; it involves testing all around a bug, looking for other bugs.

• Load testing tests the limits of a system, such as the number of players on an MMO server, the number of sprites active on the screen, or the number of threads running in a particular program. Load testing requires either a large group of testers or software that emulates heavy activity.^[2]

• Multiplayer testing may involve separate multiplayer QA team if the game has significant multiplayer portions. This testing is more common with PC games. The testers ensure that all connectivity methods (modem, LAN, Internet) are working. This allows single player and multiplayer testing to occur in parallel.^[8]

Console hardware

For consoles, the majority of testing is not performed on a normal system or consumer unit. Special test equipment is provided to developers and publishers. The most significant tools are the test or debug kits, and the dev kits. The main difference between from consumer units is the ability to load games from a burned disc or from a hard drive, as well as being able to set the console for any publishing region. This allows game developers to produce copies for testing. This functionality is not present in consumer units to combat software piracy and grey-market imports.^{[citation needed]}

• Test kits have the same hardware specifications and overall appearance as a consumer unit, though often with additional ports and connectors for other testing equipment. Test kits contain additional options, such as running automated compliance checks, especially with regard to save data. The system software also allows the user to capture memory dumps for aid in debugging.^{[citation needed]}

• Dev kits are not normally used by game testers, but are used by programmers for lower-level testing. In addition to the features of a test kit, dev kits usually have higher hardware specifications, most notably increased system memory. This allows developers to estimate early game performance without worrying about optimizations. Dev kits are usually larger and look different from a test kit or consumer unit.^{[citation needed]}

See also

• Software testing
• Test plan
• Game development
• Software release life cycle

Notes

1. ^ Bates 2004, pp. 176-180
2. ^ ^{a b c d} Moore, Novak 2010, p. 95
3. ^ ^{a b} Oxland 2004, p. 301-302
4. ^ Bates 2004, pp. 178, 180
5. ^ ^{a b c} Oxland 2004, p. 301
6. ^ ^{a b c d e f} Bethke 2003, p. 52
7. ^ Bates 2004, p. 176
8. ^ ^{a b c d e f} Bethke 2003, p. 53
9. ^ ^{a b c d} Bates 2004, p. 177
10. ^ ^{a b c d e f g} Bates 2004, p. 178
11. ^ ^{a b c d} Bates 2004, p. 179
12. ^ ^{a b} Bates 2004, pp. 178-179
13. ^ Moore, Novak 2010, p. 72
14. ^ Bates 2004, p. 180
15. ^ ^{a b} Bates 2004, p. 261
16. ^ Moore, Novak 2010, p. 25
17. ^ Moore, Novak 2010, p. 2
18. ^ Moore, Novak 2010, pp. 84, 237-238
19. ^ Fleming, Jeffrey (2008). "7th Annual Salary Survey". Game Developer (United Business Media) 15 (4): 8. 
20. ^ Adams, Rollings 2003, p. 17

References

• Adams, Ernest; Rollings, Andrew (2003). Andrew Rollings and Ernest Adams on game design. New Riders Publishing. ISBN 1592730019. 
• Bates, Bob (2004). Game Design (2nd ed.). Thomson Course Technology. ISBN 1592004938. 
• Moore, Michael E.; Novak, Jeannie (2010). Game Industry Career Guide. Delmar: Cengage Learning. ISBN 978-1-4283-7647-2. 
• Oxland, Kevin (2004). Gameplay and design. Addison Wesley. ISBN 0321204670. 

External links

• Sloperama: "Working as a Tester" from game industry veteran, Tom Sloper
• Game tester from the International Game Developers Association (IGDA)
  • IGDA's Quality Assurance Special Interest Group
• Book Extract: Game QA & Testing - Ready, Set, Go!
• "Testing Video Games Can't Possibly Be Harder Than an Afternoon With Xbox, Right?" from Seattle Weekly