Support NeoGAF

[LectureMaster]

Only 23% of Developers Believe They've Conducted 'Enough' QA for New Games

It was revealed in a recent study that a whopping 77% of developers feel they're not conducting enough QA ahead of releasing games.

insider-gaming.com

---

In a recent study published by modl.ai, it was revealed that a whopping 77% of developers polled felt like they didn’t conduct enough QA for their studio’s most recent releases. Additionally, 50% of developers surveyed felt that QA budgets aren’t ‘growing fast enough’ to keep up with the ‘increasing complexity of modern games’.

The study was pieced together to take stock of the quality control ecosystem in the world of gaming development. It was focused primarily on gauging the adoption and anticipation of artificial intelligence in QA processes and procedures.

---

Not Ready But Released​

In recent years, we’ve seen some big-budget games released in a woeful state. More than a few titles have hit the open market in a paltry condition, boasting bugs across the board and performance issues left and right.

Last year, STALKER 2 was released to an eager reception but it was hampered by a stack of bugs – particularly on console platforms. In 2020, Cyberpunk 2077 was released in such a bad state that it faced legal concerns from investors.

In the report published by modl.ai, it was revealed that 94% of developers (extrapolated) believe that AI will ‘play an important role in the future of game QA’. But with that being said, it comes as a surprise that just 18% of developers feel they’re ‘fully prepared to implement AI in all QA processes’.

They recognise that it’s the future of QA and testing, but they’re not ready to adopt it just yet.

In the numbers, 70.3% of survey respondents said they’d never launched a bug-free game.

In a statement on the data, Dajana Dimovska, the CEO of Indium Play, said:

You’re never going to catch every bug, but AI can get you closer. It can take on early testing responsibilities, reducing the need for large QA teams throughout the production phase.
As the game reaches a more playable and polished state, manual QA testers step in — working alongside AI to identify and address quality issues. This collaboration between experienced testers and AI ensures a better final product.

Many are ready to place stock in the implementation of AI-based QA. In one slide in the report from modl.ai, it was revealed that 21% of developers feel that AI testing is much more effective than traditional methods.

Presently, the blockers preventing industry-wide adoption include the complexity of setting up and operating AI-based systems, the costs of doing so, and the resistance to change – and fear of facing redundancy.

---

 

[LectureMaster]

 

[GymWolf]

I guess you are not aware of the new branch in the empire military corp, the storm twerkers

 

[Angry_Megalodon]

I have grown very little sympathetic to layoffs and this just reinforces my position.

 

[Guardian Monkey]

100% of gamers are aware of this.

 

[Laptop1991]

Yeah i know, they have saved me quite a bit of money not doing Q and A cos i haven't bought their dodgy games lol.

 

[Fake]

They still blame gamers somehow.

 

[Killer8]

I used to work in QA and every time I read some fucking spaz online saying "how did they not catch this bug???" I want to bash their skull in with a claw hammer.

QA catches a massive amount of bugs. The problem is not QA. You don't need AI to catch them. The problem is the programmers a) not having enough time to fix everything, b) not having enough manpower to fix everything, c) needing to fix something much more important instead, or d) simply not giving enough of a shit ("by design").

If you think you're being smart by pointing out an issue online, trust me, they spotted it and there's probably been a JIRA entry about it for months - but it was probably just a little bit more important that the programmer's time was spent making sure the game fucking boots at all.

 

[Saber]

Shocker, color me surprised

Who needs to waste money in QA when we have dumb people who do this for free. The next Battlefield its making a whole game out of fan feedback.

 

[Topher]

 

[StreetsofBeige]

They still blame gamers somehow.

In my example, I wouldnt classify it as blame the gamer. But more like make the user the tester himself.

Unrelated to game making, doesn't sound much different than corporate ERP program issues or getting hold of the IT guy to fix data.

Something goes wrong, you send a detailed IT ticket to fix it. A few days later they say they fixed it and go check the data. Data still shows zero like a few days ago, so you get hold of the guy again saying nothing fixed.

Then he says ok, he'll fix it again and maybe it fixes, maybe it doesnt again requiring another go around.

Well, maybe if you retards checked the data to see if it really shows up first before stamping the ticket complete, things would go smoother. It's like they just do a fix and assume it works without even checking themselves. The dumbest thing is when they stamp it complete and it's still wrong. Sometimes you just resubmit reopening to fix. But then sometimes the asstard will say "Sorry, it's closed can you open up a new IT ticket again?"


.

 

[GoldenEye98]

I could see modern AAA games be a nightmare for testing.

Think about how many different scenarios there could be...especially in a massive open world...

 

[ReBurn]

I used to work in QA and every time I read some fucking spaz online saying "how did they not catch this bug???" I want to bash their skull in with a claw hammer.

QA catches a massive amount of bugs. The problem is not QA. You don't need AI to catch them. The problem is the programmers a) not having enough time to fix everything, b) not having enough manpower to fix everything, c) needing to fix something much more important instead, or d) simply not giving enough of a shit ("by design").

If you think you're being smart by pointing out an issue online, trust me, they spotted it and there's probably been a JIRA entry about it for months - but it was probably just a little bit more important that the programmer's time was spent making sure the game fucking boots at all.

 

[Rentahamster]

 

[Digital-Aftertaste]

Here would be an avenue where AI tools could possibly flourish and subsequently make the entirety of QA sweat profusely.

 

[rm082e]

I think you could ask this same question of just about any software team and they would have similar opinions. Maybe in medical and military weapons technology you get an amount of QA and testing that the whole team feels comfortable with because lives are on the line. But the average software shop can't afford to spend a ton of time on QA and testing if the software can be patched over the interwebs.

 

[Halo is Back]

No shit, 90% of "AAA" game studios do not care if thier game is broken at launch. The state of game releases today largely reflects this.

 

[Spyxos]

Yeah, you can clearly see that Monster Hunter Wilds will come out unfinished.

 

[Dr_Ifto]

Ive worked in a few tech companies and I can say this is across the board, not isolated to game companies.

 

[K' Dash]

Any person working on software development can tell that for current games it is impossible to test everything, also the amount of edge cases and combinations of things that you can do to break the game could be infinite, I'm actually surprised that most games release on a playable state.

I'm glad I don't work for the games industry, working on Fintech apps is less punishing.

 

[yogaflame]

But, but, but, they value DEI more than QA, that is more important

 

[Gambit2483]

Figures. They have enough money to pay these crappy consultant companies but not enough to have actual gamers playtest to tell them how shit some of this stuff they are greenlighting is.

Everything makes sense now.

 

[Loomy]

Well I for one am shocked.

 

[Yoboman]

I have grown very little sympathetic to layoffs and this just reinforces my position.

You realise QA is first to go with layoffs, which is probably why devs feel this after the past few years of layoffs

 

[Loomy]

Ive worked in a few tech companies and I can say this is across the board, not isolated to game companies.

Yup. QA teams dismantled and devs are expected to test/QA their own work before release. It's the dumbest thing ever.

 

[ReyBrujo]

That's one of the downsides of the gaming industry, once you set a release window delaying will only bring free negative reviews so it's always better to launch it buggy on time than delaying it. Just look at Cyberpunk 2077, nobody remembers its terrible launch and it sold over 30 million copies. Gamers will swallow it all.

 

[StereoVsn]

Yup. QA teams dismantled and devs are expected to test/QA their own work before release. It's the dumbest thing ever.

It’s really dumb. MS, for example, has been cutting their QA teams for ages on their Enterprise products.

Which means tons of issues from Windows to their Cloud services. All to save a few $ when they make tens of billions $ a year.

 

[LavitzSlambert]

I used to work in QA and every time I read some fucking spaz online saying "how did they not catch this bug???" I want to bash their skull in with a claw hammer.

QA catches a massive amount of bugs. The problem is not QA. You don't need AI to catch them. The problem is the programmers a) not having enough time to fix everything, b) not having enough manpower to fix everything, c) needing to fix something much more important instead, or d) simply not giving enough of a shit ("by design").

If you think you're being smart by pointing out an issue online, trust me, they spotted it and there's probably been a JIRA entry about it for months - but it was probably just a little bit more important that the programmer's time was spent making sure the game fucking boots at all.

99% of the time when someone say "their QA sucks" online, they aren't directing it right at the QA team itself. It's almost always the QA process. In Microsoft's case, it's the QA team they got rid of years ago. In Arrowhead's case, it's that they clear nobody was testing anything at all at one point.

 

[Rentahamster]

I used to work in QA and every time I read some fucking spaz online saying "how did they not catch this bug???" I want to bash their skull in with a claw hammer.

QA catches a massive amount of bugs. The problem is not QA. You don't need AI to catch them. The problem is the programmers a) not having enough time to fix everything, b) not having enough manpower to fix everything, c) needing to fix something much more important instead, or d) simply not giving enough of a shit ("by design").

If you think you're being smart by pointing out an issue online, trust me, they spotted it and there's probably been a JIRA entry about it for months - but it was probably just a little bit more important that the programmer's time was spent making sure the game fucking boots at all.

I think the spirit of this sentiment is more along the lines of "How did they not feel like it was necessary to devote the resources into fixing this bug???"

 

[Boss Man]

Maybe games aren’t tested well enough, but this poll isn’t very meaningful IMO. You’d get the same results from any software engineer. Can always test more.

 

[DeepEnigma]

 

[ShadyAcshuns]

All the more reason to give them $79 per game now.

 

[Radical_3d]

This is low key awesome.

 

[ChazAshley]

Posted this in another thread but very appropriate:

 

[Skifi28]

On the contrary, there has never been so much QA testing in the history of gaming. We're talking about potentially millions of QA testers per game.

 

[Ozzie666]

All that QA budget went into DEI and re-conditioning their employees. Partialliy kidding.

No Publisher wants to spend money on QA if they can make short term profits on disasters, game and reputation means almost nothing to anyone outside of Nintendo. At least on a consistent basis. Most companies have that PC mentality, games can be patched to greatness. Risk vs reward. Games like Star Wars Outlaws and even Cyberpunk should set them straight, but it doesnt. Enough shit games make profits with bugs.

I miss cartridges because if you fucked up, you really fucked up and it caused much more pain.

 

[xrnzaaas]

There's an even worse scenario - QA testing has been paid for, but the developer or (more likely) publisher refuses to take extra time to repair the reported bugs. The game shipping as planned takes priority over everything.

There are many cases where delaying a game even by a week or two would mean much better reception. But no, release a broken or barely working game that gets bad Steam reviews that lead to worse sales.

 

[Gp1]

Let me see if I understood that right.

AI doing QA = necessary
AI doing programming, script, script doctoring, art, Sound etc.= Industry Doom!

Right...

 

[Gp1]

That's one of the downsides of the gaming industry, once you set a release window delaying will only bring free negative reviews so it's always better to launch it buggy on time than delaying it. Just look at Cyberpunk 2077, nobody remembers its terrible launch and it sold over 30 million copies. Gamers will swallow it all.

That's the Achilles heel in Agile project management with all those fluid requirements.

 

[Hudo]

 

[Esppiral]

More news at eleven.

Btw that figure seems low, I would say 99%

 

[MrTroubleMaker]

 

[WX3]

The remaining 23% do no testing at all.

 

[Spiral1407]

I can tell

 

[Dorago]

Labor costs in America and Europe have made game dev unaffordable at least in the context of giving recent college grads competitive compensation vs other industries.

Outsource has gotten much more expensive as well because of demand and increasing wages in India, China, and Indonesia.

Finally 40% cumulative inflation in 3.5 years has really screwed things up in ways that are just being felt.

You can't just pay people less either because rent on garbage apartments has gotten cartelized thus has risen double what inflation would have indicated.

Bottom line our favorite hobby is an entertainment product that became too expensive to be profitable due to factors outside of any dev studio's or publisher's control.

If you don't think this is true look a the struggles MacDonalds or WalMart are having.

Devs HQ'd in third world countries and indie hobbyists can probably survive, but they still need consumers to be able to afford anything and I don't see ANY relief for this in the near term.

DOG will trim some gov spending but will do nothing for the 3 trillion needed for SS, 2.5 for Military, 2 for Medicare/Medicaid, and 2 for debt servicing that need to be paid every year before ANY government function takes place.

So much employment is a direct function of government subsidy that trimming even wasteful/useless programs removes hundreds of people from the consumer pool who would be buying vidya games or Toyota Carollas or what not.

Seems like the same thing that happened to the USSR happend to us but 36 years later and there's nothing we can do about it.

 

[StreetsofBeige]

Labor costs in America and Europe have made game dev unaffordable at least in the context of giving recent college grads competitive compensation vs other industries.

Outsource has gotten much more expensive as well because of demand and increasing wages in India, China, and Indonesia.

Finally 40% cumulative inflation in 3.5 years has really screwed things up in ways that are just being felt.

You can't just pay people less either because rent on garbage apartments has gotten cartelized thus has risen double what inflation would have indicated.

Bottom line our favorite hobby is an entertainment product that became too expensive to be profitable due to factors outside of any dev studio's or publisher's control.

If you don't think this is true look a the struggles MacDonalds or WalMart are having.

Devs HQ'd in third world countries and indie hobbyists can probably survive, but they still need consumers to be able to afford anything and I don't see ANY relief for this in the near term.

DOG will trim some gov spending but will do nothing for the 3 trillion needed for SS, 2.5 for Military, 2 for Medicare/Medicaid, and 2 for debt servicing that need to be paid every year before ANY government function takes place.

So much employment is a direct function of government subsidy that trimming even wasteful/useless programs removes hundreds of people from the consumer pool who would be buying vidya games or Toyota Carollas or what not.

Seems like the same thing that happened to the USSR happend to us but 36 years later and there's nothing we can do about it.

Game studios can surely cut costs if they cut out a lot of cut scenes, diologue which a lot of gamers probably skip and uber texture details and artwork only DF microscopic analysis would reveal.

Problem is it's an industry that always has to go bigger and better no matter what. It's not too often a sequeled game will be shorter or worse in production values than an old game.

 

[Bernardougf]

And since people continues to pre-order and day 01 buy this games to practically beta test them... nothing is going to change.. is standard practice now.

Early adopters are beta testers. And should be ok with it.. its not a secret.

 

[Jesb]

Gaming is past its prime. Been saying this for a while.

 

[StreetsofBeige]

And since people continues to pre-order and day 01 buy this games to practically beta test them... nothing is going to change.. is standard practice now.

Early adopters are beta testers. And should be ok with it.. its not a secret.

Never understood the preorder craze. I did it way back when there were buy 2 get 1 free deals and I knew COD and NHL were familiar territory. And for uber collectors, preordering a $200 deluxe pack with a plastic figurine and cheesy poster may require it or else it runs out.

But probably 95% of preorders are just the standard game with no extra perks.

The internet has day one reviews, YT videos, streamers gabbing about the game etc.... By the time someone gets home from work or school and checks metacritic after dinner, there's probably 30 reviews already to sift through. A high profile game will probably have 50 reviews in the next 48 hours.

Yet, gamers can never help the itch but preorder A DIGITAL DOWNLOAD as if it's the old days of no early reviews, a magazine review might be next month and people were relegated to going to a store and buying a disc blind hoping it's good.

 

[SHA]

Devs don't quit the industry, you need to dig to find what you like under no name studios and publishers. Sometimes they don't want to talk about their next games.