Let’s Talk - Quality

On May 6th, 26 days ago, we switched off our 3rd Open Beta and we wanted to start by thanking each and every one of you that took part. Especially those who slayed Ortorg or managed to clear pre-nerf Upper Blackrock Spire.

You helped find issues across the full spectrum of the game from simple item stat oddities, quest bugs, dungeon scripting issues and general technical hurdles. In total more than 11,700 reports were filed in game. With this we knew we needed to jump into action and begin the process of whittling this down to a reasonable number.

The Initial Cull

The first step in this process began on the 7th of May with the creation of a dead simple automation script that:

• Goes through every report in each category (Spells, Quests, Items, Achievements, etc).
• Finds duplicates by matching ID's and merges into one single report.
• Based on Category, and whether it is a Blizzard ID or an Epoch ID, assign labels to a report.
• Submit the final report via Github API to our bug tracker.

Due to rate limits on the Github API from start to finish the above process took approximately 4 days however was a far more efficient way of doing things than we have in prior betas as they were primarily filtered through entirely manually. This gave us 8,276 open issues on our bug tracker ready to move to the next stage of triage.

Generic Grand Canyon

The "Generic" category for reported bugs in game is considered a catch all for when something doesn't quite fit into any other category. The challenge with these is finding all the various duplicates and in some cases struggling to identify the context of a report. In every beta so far this has been the largest category, and this beta was no exception. Of the remaining 8,276 issues following the above step 61% of them are within the Generic category.

This is the first time we've had this particular category exceed more than half off all issues. This of course has knock on effects in trying to identify, merge duplicates, confirm and close issues. This greatly impacted decisions to be later explained in this post.

QA Dream Team

For prior beta's both the types of reports and quantity of reports were more granular. With Generic category vastly exceeding our expectations and in general us receiving more players than ever leading to more reports we knew the time had come where this is a sincere waste of time for developers to sift through. We needed to do something that up to now we haven't on Project Epoch, expand outside of the Design and Development team.

Behind the scenes however this required some rapid refactoring of how our internal realm is set up and prepping for expanding access to more people. This included rewriting how our Launcher distributes patches to be able to support giving users an access key to swap to internal realm, automating creation of accounts that can be easily revoked, setting up a new Discord server and adjusting the contents of shipped files themselves to hide some things we definitely didn't want a chance to leak.

Once we were confident that infrastructure wise we were ready to expand we opened up applications for our QA & Bug Triage team on the 19th of May. We received more than 30 applicants in this short window which closed on the 23rd of May.

From those we selected one to be our first test case of easy of getting access to internal realm and bug tracker etc which went very well. Starting after this post goes out we will be reaching out to up to 5 more additional members and we can properly kick off this new section of the team.

Current State

Of the 8,276 issues within our bug tracker we have closed 15%. This could be due to them being invalid, duplicates, not reproducible or being fixed. This leaves us with 7,044 remaining open. Of those 152 are confirmed and either awaiting testing or awaiting a developer. The other 6,811 are still awaiting triage and classed as Not Confirmed.

The priority of fixing issues will be Classes > Dungeons > Key Loot > Everything else. This influenced our QA team selection process too based on users who had actively been farming dungeons or were knowledgable in particular classes that we knew were going to be problematic.

Big Gremlins

Some of the critical issues we've been looking at over this time are as follows.

Instance ID Shenanigans

This encompasses anything that can lead to a player being seperated from their group and respawning into a different instance ID. This is by far the most critical bug we have and a fix for this is awaiting testing. The challenge with this however is that it's not consistently reproducible and just seems to "happen". This means any fix we have is currently theoretical and will require stringent testing on our internal realm and may also lead to a test weekend where we throw you guys at dungeons again.

Fatigue & Evade

Across essentially every coastline there have been issues with fatigue flagging and evading. These are both caused by a legacy issue that caused corruption in map assets many years ago. We were waiting until we were confident we'd covered all new world terrain content in development before bulk fixing it. This has now been resolved globally by manually painting new fatigue boundaries across all coasts afflicted.

Instance Saving

In an ideal world a server would not crash however we do not live in an ideal world. When a server does crash it makes it swiftly painfully obvious what instance state is or is not being actually saved to the database, and the frequency of what triggers those saves. Most dungeons functioned well with Blackrock Depths being pencilled in for a pass to correct some door state problems. However, see below for the worst offender.

Stratholme

There is no other way to describe the state of Stratholme in Beta 3 than this; a hack built on top of hacks. With absolutely no instance state being saved to the database 1 server restart is enough to brick any run in this dungeon. This is of course not acceptable.

Alongside the above were many issues related to bosses not spawning, duplicates of bosses spawning, missing role play events and more. With that in mind we decided we needed to commence a ground up re-implementation of this dungeon ensuring both accuracy and reliability. This has lead to needing to expand the bounds of our modding tool TS-WoW to better support writing and reading from state and hooks into when creatures and game objects get created when an instance is created or loaded.

Using these new capabilities we have almost entirely rewritten the Undead side of Stratholme and will shortly be moving on to the Live side. This includes things such as door states being correctly saved, ziggurats remaining destroyed and missing functionality such as Magistrate Barthilas sprinting away if you enter from the Service Gate.

We will be tweaking Blackrock Depths, Baradin Hold and Glittermurk Mines to use some of these new features to resolve instance state issues within them too.

Pooling Woes

Pooling is responsible for keeping groups of objects spawned together. Most commonly used for Creature spawns and Node / Chest / Quest Object spawns. We've received hundreds of reports that we know are related to this 1 aspect of the core with it simply not restocking them over time. This leads to fewer and fewer node / chest spawns the longer the server uptime. Eventually quest objects may not spawn at all.

This is reproducible with a test case and is considered critical priority. We will likely contract out a fix for this issue to ensure speed of resolution. This should allow us to close hundreds of issues on our tracker.

Next Steps

To put it simply. It isn't cooked. You know it isn't, we know it isn't. And we do not want to skimp on quality or push to the point of burn out trying to rush something. Monthly logs will return bringing you at least some update on progress with development and fixes.

Before the next monthly log, or with it, we intend to open our bug tracker publicly so that you as our community can see even more up to date information about the latest state and what we consider high priority.

We are likely to need a handful of weekend long tests to validate some issues have been resolved at a larger scale such as Instance ID issues and pooling.

Project Epoch has grown so much further than we ever anticipated both in content quantity and community scale. This has naturally lead to both more reported issues and a broad spectrum of confirmed bugs. We are as ever appreciative of each and every one of you who has tested in our betas. We hope that you can keep in mind that for us this is a passion project that is entirely done as no ones full time role and be patient with us while we narrow down these issues.

When will it release? When it's ready.