Automated testing

From HEWIKI
Jump to: navigation, search

Contents

Overview

Manual testing can be useful in finding defects in a product but is time consuming, laborious and for products with the size and scope of MMOs can be extremely expensive. Automated testing involves the creation of programs to test your product in a repeatable and more efficient manner helping to reduce defects and potentially revealing classes of defects that would not otherwise be encountered until too late (such as what happens under the load of thousands of users). HeroEngine provides multiple frameworks allowing for the automated testing of HeroEngine itself, and also for your specific game implementation. These will allow you to stress test your game code and or hardware to determine what your capacity is, or if you have important bottlenecks that need to be fixed.

A commonly asked question is "How many users will I be able to service per physical server?", unfortunately the answer to the question is not simple as X users per machine. The number of users a physical server can support varies on a large number of factors almost all of which depend on your implementation and/or game design as the vast majority of all server load is directly attributable to game logic.

Some of the Factors affecting users per physical server:

Fortunately, automated testing allows you to determine the answer for your game by creating tests that put your game systems under load with the additional benefit of providing testing for your game.

There are two different frameworks which available to perform automated testing, based on the type of testing that you are doing:

Usage

What Must I Do to "Fully" Load Test A Game?

If you need to test a full game implementation to determine production load, you need:

The constraints under which you are probably operating likely do not allow for a fully realistic test to be done (or at least not until late beta).

We recommend you determine the areas that are of concern and devise smaller tests that exercise just that functionality and then measure the resulting performance. For example, when we needed to test Repository Server infrastructure we created a test using the XLC that requested files randomly from amongst the files listed in a test file. Doing so revealed that a single Repository Server process was more than capable of keeping up with several thousand simulated users saturating the gigabit Ethernet card occupying less than a single core of an eight core physical server.

Client Authentication

By default, authentication for HeroEngine is handled through a stub implementation (HEUAccess) of the UACCESS Protocol. A stub, or your real authentication system, must be implemented to allow the clients to connect to the service directory and retrieve a list of subscriptions. For example, you might implement a stub that allowed any client specifying an account name "TEST-" to connect using any password to facilitate testing of large numbers of pseudo users using the lightweight or headless clients.

An example of such a stub can be found on the HEUAccess page.

Extremely Lightweight Client

Overview

ExtremelyLightweightClientArchitecture.gif

This client is used to test underlying functionality of HeroEngine such as logging into a Service Directory or downloading files from the Repository. It is not used to run AI or Game Logic. Technically, the extremely lightweight client requires building four separate executables that are used together to perform the tests.

The client runs on Windows OSes and assuming modern hardware you can expect to run 200-250 per machine without any issues as you move into the 300-400 range you tend to run out of windows handles before having issues with the actual application itself.


Building the Solutions

Creating the Extremely Lightweight Client requires building four different executables:

Building: Lightweight Client Director 

 Directory Listening Port: tcp:209.16.216.141:62341
 Service Directory Port: tcp:hedev.heroengine.com:10373
 World: Dev
 Account Prefix: TEST-A
 Character Name Prefix: TestA

Building: Stress Server 

 GROUP=SERVER
 LWD_LISTENER_PORT=50404
 USER_LISTENER_PORT=50403
 LWD_LISTENER_HOST=209.16.216.141
 USER_LISTENER_HOST=209.16.216.141
 //LWD_LISTENER_HOST=timmy.heroengine.com
 //USER_LISTENER_HOST=timmy.heroengine.com
 END_GROUP

Building: Stress Client 

 GROUP=CLIENT
 PORT=50403
 HOST=209.16.216.141
 //HOST=timmy.heroengine.com
 END_GROUP

Deployment

Deployment of the Extremely Lightweight Client framework entails copying the folder you created in the Build step to the various physical servers you intend to use for testing and modifying some configuration values.

Physical (Control) Server

One machine should be designated as the control server and it is this machine that the Stress Server and Stress Client applications run. The control server may also simultaneously function as a test server by running a Light Client Director in addition to the Stress Server/Client applications.

Requirements:



Physical Test Server Setup

For each of the servers you intend to use to run test clients, you must perform a few simple setup steps.

Requirements:



The important configuration change that must be made is that each physical test server should be given a different Account and Character Name Prefix.

For example, if you are using three test servers the first one should use TEST-A, the second one TEST-B, and the third one TEST-C. The reason is that the Director uses the prefix to create unique account names by incrementing a local (to the server) counter spinning up TEST-A1, TEST-A2, and so on. If two machines are configured to use the same prefix, the Lightweight Clients will end up disconnecting each other due to duplicate logins.

The prefix you choose is not important persay, but it may be important depending on how you have implemented HEUAccess to allow authentication of testing clients, for example the provided sample code allows any account named TEST* access. Consequently, you should check your HEUAccess implementation for any particular requirements it may have on account names.

Likewise, character name prefix should be unique per physical server (i.e. per Light Client Director).

Stress Server

ExtremelyLightweightClientStressServer.gif

The stress server communicates with the light client directors to coordinate the tests started by the stress client application. The only thing you can really do with the Stress Server application is start or stop it. You can think of the stress server as the equivalent of the Master Control server application (not to be confused with the Master Control Console GUI you run on your local machine).

Users ; Number of stress clients connected to the stress server
LWD ; Number of lightweight client directors connected to the stress server
CTR ; Clients left yet to run, this will most often display 0 due to the speed at which clients can be spun up
CCLC ; Current client launch count
CRC ; Currently running clients


Stress Client

ExtremelyLightweightClientStressClient.gif

The stress client is used to configure and start/stop test runs by communicating with the stress server, you can think of this as the equivalent of the Master Control Console application.

Users ; Number of stress clients connected to the stress server
LWD ; Number of lightweight client directors connected to the stress server
CTR ; Clients left yet to run, this will most often display 0 due to the speed at which clients can be spun up
CCLC ; Current client launch count
CRC ; Currently running clients


Exit Codes

0 - success
134 - Terminated normally by the Lightweight Client Director (via the button), this generally qualifies as success.
137 - Internal error, check solution
Others - check solution


Test Modes

Currently, the stress client supports three different test modes: 

 // Possible parameters are as follows:
 //
 // --director-transport     - string which contains the transport to the director process           (auto generated)
 // --client-sequence        - Number set by the director                                            (auto generated)
 // --mode                   - possible modes are servicedirectory, repository, game
 // --service-directory      - transport of the service directory to connect to
 // --username                                                                                       (auto generated)
 // --password                                                                                       (auto generated)
 // --world                                                                                          (auto generated)
 // --character                                                                                      (auto generated)
 // --extended-logging       - set to true additional information displayed in stress client
 //
 // Repository parameters
 // 
 // --file-list
 // --request-count
 // --request-duration
 // --request-limits
 // --request-rate
 // --request-order
 // --cache-directory
 // --bandwidth-target


Example

--mode=servicedirectory --service-directory=hedev.heroengine.com


Additional information can be found in the code located at \\<PerforceRoot>\hetesting\extremely_light_client\LightClient.cpp.

Light Client Director

ExtremelyLightweightClientDirector.gif

The Light Client Director is in charge of spinning up Extremely Lightweight Clients on a particular physical server (i.e. you have one LCD per physical machine you are using in your test). The LCD can be thought of as the equivalent of a FireupDaemon on a HeroEngine Server and you may wish to write a .bat file to loop running the LCD process so that they auto restart after the Stress Server issues a Reset command (i.e. shutdown LCDs).


Please note, the Launch Clients button is deprecated functionality as launching clients is now initiated via the Stress Client UI


Headless client

Overview

CautionSign.gif The Headless Client has roughly the same requirements as the Player Client, consequently machines used to host Headless Clients must be capable of running the Player Client.


HeadlessClientArchitecture.gif

Headless client is more similar to a real client than the Extremely Lightweight Client, and runs AI scripts allowing it to do anything HSL can do (which is pretty close to everything). It is a version of the player client that runs as a server process. It is controllable through script, and basically connects to your game and plays your game under script control, to simulate what a player character might do under your game implementation.

The headless client takes up significantly more RAM than the Extremely Lightweight Client as it must load all of the models and the physics representation for the area, the DOM, the GOM, etc. Additionally, it is much heavier weight from CPU perspective as it is running all game logic normally. Due to the number of imponderables related to your game design, area design, and systems you can run anywhere from a few to an upper limit of low tens on a modern desktop (more on actual server hardware, but not an order of magnitude more).

Usage

In order to utilize the Headless client you must:

Building the Solutions

The test client requires that you build the following solutions:

Building: FireStorm

FireStorm is the communication layer of HeroEngine and must be compiled in release mode to prepare the firestorm.lib.

Building: HeroBlade
Building: HJTesting

Create a server group

Server groups are arbitrary groupings of physical machines created using Master Control Console upon which HeroEngine's director processes (those configured to utilize a particular group) may spin up their child processes.

Hardware Requirements:

The software installation and configuration changes described below in #Deployment must be made on each of the machines in the server group.

Deployment

Deployment is performed by hand on each of the servers upon which you intend to spin up test clients.

HeroengineServerDirectoryStructureTesting.png


 [TEST_DIRECTOR]
 Executable_name=testdirector.exe
 Executable_path=C:\Program Files\HeroEngine\TESTINGHEDEV\

 [TEST_CLIENT]
 Executable_name=testclient.exe
 Executable_path=C:\Program Files\HeroEngine\TESTINGHEDEV\

FirestormDaemonsInstallDirectory.gif


 C:\Documents and Settings\All Users\Application Data\HeroEngine\configs\FIRESTORMDAEMONSHEDEV.cfg
 C:\Documents and Settings\All Users\Application Data\HeroEngine\configs\HJSERVERSHEDEV.cfg
 C:\Documents and Settings\All Users\Application Data\HeroEngine\configs\TESTINGHEDEV.cfg

RestartingFireupDaemon.gif


Set Up Daemon/Process Definitions

Using Master Control Console create TEST_DIRECTOR Daemon and TEST_CLIENT Process Definitions. Once the definitions are created, restart Master Control (i.e. type /reset in the chat panel of Master Control Console.

Director Daemon Definition
MasterControlConsoleEditConfigsAddDaemon.gif

Using Master Control Console, edit the configurations to add a new daemon definition for the TEST_DIRECTOR. Which will require you enter the following parameters: 

[TEST_DIRECTOR]
Executable_name=testdirector.exe
Executable_path=C:\Program Files\HeroEngine\TESTINGHEDEV\


MasterControlConsoleEditConfigsNewDaemon.gif

Director Process Definition
MasterControlConsoleEditConfigsAddProcess.gif

Using Master Control Console, edit the configurations to add a new process definition for the TEST_DIRECTOR. Which will require you enter the following parameters: 

[TEST_DIRECTOR]
Executable_name=testdirector.exe
Executable_path=C:\Program Files\HeroEngine\TESTINGHEDEV\


MasterControlConsoleEditConfigsAddTestDirectorProcess.gif

Modify the following Process Parameters:

MasterControlConsoleEditConfigsModifyProcessParameters.gif

 tcp:servicedirectory.play.net:10373


 Connect as FS_SERVICE to the database for the Master Control Schema (generally this is the same as your world, but some configurations may differ)
 select daemon from daemon where daemon_name = 'TEST_CLIENT';



Client Daemon Definition
MasterControlConsoleEditConfigsAddDaemon.gif

Using Master Control Console, edit the configurations to add a new daemon definition for the TEST_CLIENT. Which will require you enter the following parameters: 


[TEST_CLIENT]
Executable_name=testclient.exe
Executable_path=C:\Program Files\HeroEngine\TESTINGHEDEV\

MasterControlConsoleEditConfigsAddTestClientDaemon.gif


Add Test Director Service to the World
MasterControlConsoleEditConfigsAddServiceToWorld.gif

Once defined, you must now add a service to the world to be tested so you can actually fire up the service and start using it.

MasterControlConsoleEditConfigsAddProcessToAService.gif


Reset Master Control

Reset Master Control so that the new definitions are loaded.

Fireup Director Service

Optionally, shutting down the world and bringing it back up will (in effect) do the same thing

Implement Script Behavior

There are essentially two scripted components:

Implementing Server Test Framework

Basically, you need to write a system or simple command that utilizes the external function LaunchTestClient() passing appropriate parameters to the system. A more elaborate system would support the ability to collect data from the test clients and correlate them for later analysis and/or comparison. Ultimately, how complicated you make your test framework is a function of the features you want it to support. 

// Sends a message to the test director to launch a headless test client with the given arguments
// method _TestDirectorResponse(requestID as ID, launchSuccess as Boolean) will be called after the launch attempt
external function LaunchTestClient(callback as NodeRef, requestID as ID, username as String, password as String, testNumber as ID, testArguments as String)
Implementing Client Test Framework

The client test framework needs to understand what kind of test it is supposed to be performing, and then support the logic required to perform the test. You can think of this as AI for the client, basically telling it what to do based on the conditions known to it. For example, you might write a test that understands how to create a new character and exercise the features of your character creation system talking to the server scripts just as a real client would and perhaps even going so far as to fake mouse clicks on UI elements on the client. 

// Obtain the headless client test parameters.  Note that these values are not set for non headless clients
external function GetTestParameters(testNumber references ID, testArguments references String) as Boolean


Retrieved from "http://wiki.heroengine.com/wiki/Automated_testing"
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox