The first full integration automation framework available in the Unity game engine.
Traditional QA Test Automation Framework
The Trilleon test automation framework is a comprehensive library of code built for C# in the Unity engine. Modeled after Selenium, it is the first comprehensive Unity toolset that allows you to build a traditional automation (integration) test framework.
The Short and Sweet List – What do I get with Trilleon?
On Devices, Test Execution Both Locally and In the Cloud (PC, Mac, Android, IOS, WebGL)
Plays Your Game As Any User Would
Near Plug-and-Play Setup Experience
Actively Sends Test Results and Performance Metrics So You Can Tune Your Game
Record and Playback
Out-Of-The-Box, Jenkins-Ready Server Framework (Appium/Python/Shell Scripts Supplied)
Currently Supports Single Player Or Turn-Based Multiplayer Testing
Framework Impact On Memory Usage Negligible (Great Even For Mobile Games!)
Several Editor Windows For Easy Development and Launching Of Tests
Reactive, Visually-Appealing, Informative Test Reports (Html Reports and Simple JUnit Xml)
Game-Agnostic Client Code Library (C#)
Wide Array Of Options That Allow Complete Control Over Test Interactions And Dependencies
Easy To Strip In Production Build Process, But Lightweight If You Choose Not To
Abstraction For Easy Integration Of Developer-Specific Code (Ex: TextMeshProUGUI?)
Auto-Generates Test Function Stubs For Interacting With Selected Hierarchy Objects
IL2CPP and Mono2x Ready
Does not work with Web Player platform (No longer supported by Unity)
Unity 5.3+ Tested; Likely Works With No Or Minimal Modifications 4.x+
Works In Both .Net 2.0 and .Net 2.0 (Subset)
Considerate Of Earlier Unity Version Foibles (ex. No ForEach Usage)
What Are My Options?
Unity Test Tools is the only other full QA test integration framework available to Unity developers. However, it does not provide a substantial footing for development of traditional test automation suites. Unity Test Tools primarily allows you to build scenarios, and to test those scenarios around situations that approximate the game as a user would experience it. The ultimate result becomes an amalgamation of unit testing and automation testing. Additionally, the ability to run such automation outside of the Unity Editor (on devices, and playable builds), as part of an automatic build process, is limited or non-existent.
For years, people have tried developing traditional automation frameworks for their video games that can interact with any game using the same engine. Some have resorted to broadcasting xy coordinates to an external application through a debug log. The external test driver would then parse, interpret, and click the specified coordinates for an intended target. In this way, your automation only has insight into information you explicitly tell the game code to broadcast; setting such a framework up demands that the game be built around this requirement every step of the way. This isn’t a timely, developer-friendly, or scalable approach. Not to mention that heavy use of logging can have a satistically-significant effect on performance.
Other people have built frameworks around image recognition; programmatically comparing baseline images to what the game is rendering to find coordinates of buttons, or confirm text on screen. This process is incredibly unreliable. Your baseline images must be replaced whenever the interface elements change. These baselines must account for aspect ratio, device resolution, and asset quality variants. Anything dynamic or unexpected will throw off the process. Tests developed in this way cannot be relied on as part of a build process.
Enter Trilleon: With the new Disruptor Beam® framework, you will find the tried and true automation approaches of browser applications and standard native applications have been ported into the world of video game development.
Why Should I Use Trilleon?
Trilleon is modeled syntactically after Selenium, offering a multitude of tools built to interact with every aspect of your game, while remaining independent of the actual game code (with the exception of Page/Test Objects that wrap game code interactions in a single, organized place).
Selenium relies heavily on open source support to develop and maintain driver executables. This means that an additional application needs to be run to broker interaction between a test suite and most browsers or applications.
In web and native application automation, this external driver has access to a DOM consisting of XML formatted in child-parent hierarchical relationships. This simple, but highly-ordered structure is easy for an external process to read through and find the object that it needs to interact with. Both the application and driver develop a mutual understanding of what object is being referenced, and the driver in turn is delivered the information needed to simulate an interaction. Video games are a completely different animal. The volatile and mutative nature of a video game is not conducive to a DOM. Therefore is no easy way to retrieve information on game objects from outside the game. Indeed, if there were, it could be a security hazard.
Because of this “middle man” driver in web and native app automation, it becomes difficult to completely rely on the results of your automation, such that you can integrate your automation suite into a Jenkins build step that marks the build as unstable if any test failures occur. What this ultimately means is that, in very large automation suites, it is almost guaranteed that one or more tests will fail simply because the driver encountered a transient error that is out of your control. Because of this, gated builds relying on automation results are never as reliable as unit tests are.
This is a liability that Trilleon eliminates; there is no true driver for interaction between tests and the game (Although appium is used within Jenkins CI to load and launch the game, and dismiss device level alerts. But it does not execute any test code). With Trilleon, your game tests itself. The only thing that prevents you from investing full confidence in the results of your tests is your confidence in the quality of the test code (and how it interacts with your game).
Trilleon provides a diverse selection of test-manipulation tools; some of which even Selenium does not offer. An example of this supra-Selenium functionality is the Dependency Architecture system. With this option, you can map relationships between tests throughout your framework, or exert complete control over the order tests run within the context of the entire suite, linked test classes, or individual test classes. The framework has several layers of validation that prevents and eliminates circular dependencies; supporting you as you scale your framework from a few dozen to a few hundred tests. You can even view an interactive, graphical representation of these dependencies in the form of a custom editor window. Another functionality example is the BuddySystem Architecture which allows you to test game code that relies on more than one client (multiplayer [Turn-based currently supported, and real time support is a WIP]). More information on these topics can be found below.
Trilleon offers easy setup with cloud device farm integration for your automation suite. The code you will need is all there; simply plug it in and customize as needed. In no time, you will be ready to run your tests in TestDroid, AWS Device Farm, Xamarin, or any other provider that offers appium-python integration options. Just as important, full instructions/code are provided to set up launching of test runs on local devices through a Jenkins build step.
There are several Unity Editor Window tools included in Trilleon. For example, there’s the “Assistant” tool which allows inspection of elements in the hierarchy window of the editor. This displays all automation-relevant details, including links to the objects and scripts that reference it, and allows you to choose an action or assertion before generating code stubs that can be pasted into your tests. This code accounts for references to the selected game object, and provides a direct link to that reference. In cases where no references are found, a generic search script will use the finder functionality to locate it based on the requested attribute.
Additionally, a Record and Playback tool has just been developed that catches automation-relevant actions taken in your editor, generating a full test script that you can paste into a C# file and immediately playback. In this manner, no manual test-writing will be required to make automation tests. Therefore people of any technical skill level can generate complete, flexible, and reliable automation test scripts.
Try it out, and decide for yourself if you are ready to take advantage of the contribution to quality that a scalable automation test suite provides.
Trilleon is very nearly plug-and-play. It can be as simple as dropping the automation folder into your Assets folder and inserting two lines of code to generate the Automation Custodian object on scene load, and to enable initialization of the framework. That's it; it takes two steps and 20 seconds to satisfy the minimum requirements needed to begin developing your new Trilleon automation framework!
This architecture allows you to map relationships between tests throughout your framework, or exert complete control over the order tests run within the context of the entire suite (or smaller subset runs), linked test classes, or individual test classes.
In a traditional Selenium automation suite, built around a web application for example, it is undesirable and rarely necessary to write tests that rely on one another for completion. Doing so should be avoided at all costs. However, a video game is another animal altogether. Dependencies between testable chunks of code are a common occurrence, and can generally only be mitigated by various cheat commands. Because of this, the Dependency Architecture in Trilleon is essential, and makes this an easy process to visualize, maintain, and rely on.
Trilleon has several layers of validation that prevent and eliminate invalid dependencies; supporting you as you scale your framework from a few dozen to a few hundred tests.
This architecture allows you to test code that requires more than one client. This tool allows synchronous (action/reaction-based) and, eventually (WIP), asynchronous (real time) multiplayer tests. The BuddySystem maps dependencies across two or more devices so that multiple concurrent test runs can communicate with one another to perform action and reaction tests to provide maximum test coverage. Real-time client multiplayer test architecture will be built off of this in the future.
Attribute Versatility And Options
Attributes allow for versatile and robust test execution that satisfies almost all scenarios. These attributes range from those that declare a method as an automation test, to those that declare dependencies this test has on others. There are SetUp, SetUpClass, TearDown, and TearDownClass methods, as expected in an automation framework (in addition to Global SetUp and TearDowns). Ignore attributes prevent a test from running, and report the reason the test is being ignored in generated reports. Disregard attributes allow you to have any SetUp or TearDown flag ignored in preparation and cleanup phases of execution.
Additionally the Debug attribute marks a class as a special test class that is not included in “Run All” commands. Such a class can only be run when tests or their categories are explicitly called. This is particularly useful for generating tests that perform a specific behavior undesirable in normal integration tests (such as intentional failures). This is also useful for non-integration tests of any variety.
Write Your First Tests
Making an automation test in Trilleon is easy. The process is parallel to writing a Selenium test, or even a unit test. Learning how to implement your new framework, and prepare to write your first automation test, could not be simpler.
Test Rails Integration
Have your tests automatically communicate results to your Test Rails suite whenever they are run outside of the Unity Editor. Existing wrappers make it easy to customize and use any other test suite tools instead of Test Rails.
Explore The Tools At Your Disposal
You have a menagerie of options to assist you in interacting with every type of object in your game, Unity or custom-made. The most immediately accessible are those under the Automation Custodian game object. In the driver (not a real driver), examples include “Find”, “SendKeys”, “WaitForCondition”, “Click”, “ClickAndHold”, “KeyPress”, “Scroll”, etc. There's a custom assertion class that allows you to interact with your game objects in a multitude of ways. All of these tools are extendable, allowing you the flexibility to add platform or game-specific functionality.
View Your Test Results
View your test results in a visually-appealing, functional way. The results of your test runs are presented in an HTML report with appealing CSS and functional JS. This allows you to filter and view all aspects of failures, passes, skips, and ignores. Get useful information on each failure that saves significant time in debugging failures. You have easy-access to all the information you need in the debugging process.
Custom Editor Windows
Single Window Accessibility Template (SWAT)
Trilleon editor windows use a custom template that unifies most automation windows into a single, organized, and accessible location. This is akin to a SPA, or single page application in web development. Additionally, it looks more modern, and strives to be reactive. A custom tooltip system is integrated into the SWAT model that is much more reliable than the default GUIContent tooltip system.
Note: The window screenshots following this gif have yet to be updated. Some of the appearances are outdated, despite using the SWAT model shown above.
The Nexus window displays all tests in the framework in an organized manner with the last status of the test in its most recent run. One can launch tests based on their status, individually by clicking on the test in the window, by category, or launch all tests in the suite by pressing play. Additionally, one can launch subsets of any category.
This window has several tabs. In addition to the tests list, there is a results view that displays all Html reports for previous test runs. From here, they can be opened or deleted.If a subset of tests is launched, all dependencies will be calculated, and an updated list of tests will be presented for confirmation. One can ignore these dependencies, or add them to the requested test run.
Another tab contains the Trilleon Console. This console separates all Trilleon-related logging from the main Unity log. From here, additional log entry information can be viewed, and individual test results can be opened up in an Html report format for viewing during a test run.
See example trailer for this feature
The Assistant is an Editor Window tool that allows you to select any game element in the Unity hierarchy view. The tool then displays all automation-relevant properties for the selected element; including references to this object in other object's MonoScripts. Using this information, you can select an action to take on it, and generate a line of Trilleon code to represent that action. In addition to the line of test code, a TestObject property is generated to make the code reusable. If no TestObject file exists for the current object's top level parent, then the code will offer to generate a file for you.
See example trailer for this feature
FULL SIZE DEMO GIF LINK:
Using the recorder, you can naturally play your game in the editor; all the while generating code for each action taken. When you are complete, you can insert the code into a test method, and replay the recorded test immediately.
Dependency Web Viewer
View all tests marked with "DependencyWeb" attributes a visualized web of relationships that can be easily manipulated to see how tests depend on one another.
See example trailer for this feature
Dependency Architecture Viewer
View all DependencyTest and DependencyClass entities in the framework.
BuddySystem Relationship Viewer (Feature Incomplete)
This window allows you to view the relationships between all BuddySystem tests in your framework on a class-to-class basis. All classes containing these tests appear here, and one can cycle through each one to visualize the flow. Errors in these relationships are displayed for the user, and the issue can be fixed in the editor window, directly updating the C# file containing that test.
Edit all of your settings from one window. This window also allows you to add new keys to the settings file, and to remove non-required keys. Changes to important keys will also perform updates wherever necessary. For example; changing the global string delimiter in the configuration file will automatically update all other text files that store data using that supplied delimiter.
Unit Test Support (NEW!)
You can now test all of your coroutines and context-specific logic using the Trilleon "Unity" (Unit-like) tests. They do not run on compile time like traditional Unit test frameworks. They allow you to set any scenario and test your game logic the way you see fit.
Input of any kind is appreciated. One thing in particular that I am looking for is suggestions for features not mentioned, along with opinions on which of the features are most interesting, especially for incomplete features.
For example, here are several incomplete features:
1) Real-time Multiplayer Testing (Not started)
2) Unit Testing and Performance Testing framework integration (Not Started)
Write First Test
Customize & Extend
Todo & In-Progress
The First Full Integration Automation Framework For Unity!