“Can we meaningfully predict how things go?” How AI could shape how publishers invest

This article is part of AI Week.

In January, analyst Joost van Dreunen published a white paper on how small language models (SLMs) – as opposed to large language models (LLMs) – have the potential to transform business intelligence.

Van Dreunen co-founded the market intelligence firm SuperData Research in 2010, which was sold to Nielsen in 2018, and he subsequently launched analysis firm Aldora. Here, he explains to GamesIndustry.biz how Aldora AI could transform how games companies interact with market data, even helping to predict future trends.

What is the problem facing business intelligence?

The fundamental issue in the context of AI is that big companies like Nielsen have lots of survey data, but these are large-scale, complicated processes.

I grew up as a researcher in a world where it’s an almost ceremonial process, where we issue a request for a proposal, and then seven different vendors are invited to make their best offer. And then we’re going to go through weeks of review: we’re going to review the survey questions, and then Timmy and Joey and Jessica, they’re all going to review the different slides, and we’re going to go through iterations of the deck, and then we’re going to present it on-site.

It’s this cumbersome process, and by the time these numbers come out, the world has moved on. So part of the issue is that the existing organizational models for research firms just don’t match the speed of business. You have these overnight hits like Grow A Garden or Steal A Brainrot, and nobody sees it coming except some of the smaller fringe companies that specialize in them. But large companies are just unable to detect meaningful signals in the market anymore, because they’re just not organized appropriately. It’s a structural issue more than anything else.

How exactly could AI help?

AI is often described as a content generation tool: I think that that’s why a lot of people have some scepticism around it. The way we see it is as a distribution tool.

AI can help you distribute information and access information more easily, allow you to run multiple scenarios, and look at the problem in a more agile, iterative way.

Our focus is particularly around small language models. They are different from large language models in that they are smaller, but as a result also cheaper to maintain. They are more energy efficient: they run on an edge compute basis, so I can run it off a smartphone as opposed to an online connection, and I can combine it with private data because they’re secure – as opposed to me as a large Fortune 500 company having to share my internal data with OpenAI to run some analysis.

So the small language models basically allow you to have expert level access to information in a mode of operation that is safe, secure, and efficient, as opposed to throwing it into a large language model, which is less efficient and less secure.

You don’t need all that firepower. You don’t necessarily need 700 billion parameters when a 7 billion parameter model can suffice.

So an SLM is basically doing one thing really well rather than trying to do everything?

Exactly right. The nature of the games industry is specialisation, so I don’t need to know everything else that happened, I just need to answer this subset of questions.

You can even narrow it down and say, well, all we’re making as a studio is a shooter game and we operate in the mobile industry, so I don’t need to know all this other stuff. You can be much more specific about your needs.

Can you give me an example of how this kind of system would work?

On the base level, you can just do prompts and say, Hey, what are the biggest [games]? What are the best ones? What’s the next step? Those sort of things.

Over time, when you do reinforcement learning as you train it on your internal data, you can also [use it to take on] the job of an analyst. An analyst’s job is to anticipate the right or next question, to hear their boss’s boss raise a question and then say, ‘The question you’re really trying to answer is this other one’. With reinforcement learning, you can do exactly that. If you know enough of the subject matter, you can say, ‘What you’re really asking me is not about user acquisition, it’s about gun variety in your shooter game’. You can take different approaches to the same problem set.

One problem that people often bring up with AI is that it can be confidently wrong, and that it wants to please you. How can that be accounted for in the kind of model you’re talking about?

As AI would say, that’s a great question. There has to be human operators in the mix, which makes it difficult and expensive to some degree, but I think that it’s a necessary, mandatory inefficiency that’s going to prevent some of these AI illnesses.

What it does, however, is it flattens the number of layers between the provider and the consumer of information.

So it sounds like the people who are using this tool need to be trained, and the onus is on them to make sure that the answer is correct by drilling down into the data.

Yeah, absolutely. In the same way that a decade ago in business intelligence, you had people that were using Excel, and then there was a breed of researchers and business intelligence analysts that were already upgrading it to a SQL [structured query language] level. So they would do regression analysis using SQL, and they were just more effective.

In many ways there’s a metaphor in that we can do a regression analysis all day long, but if you don’t know that the data that went into it was structured appropriately, the result is meaningless – that would be the equivalent of AI hallucination.

It sounds like the advantage for companies is that you can ask multiple questions rapidly rather than waiting weeks for an analysis to come back.

And have an expert model at your disposal. My personal experience is that I get asked into an organization by the CEO, but I can’t possibly speak to every manager in an organization. But what if they could all have one-to-one access, not just to one version of me, but a council of experts in video games, and you are able to run your questions by them based on case studies, research, any kind of information that they’ve put out in the past. So now we have these heavyweight counsellors basically helping you think through a problem.

How close are we to this happening? Presumably this is something you’ve been working on at Aldora, right?

Correct. We’ve been spending the last two years aggregating both structured and unstructured data sets to build the training data.

How close are you to launching it as a product?

The launch will be summer: Q2, Q3. I’m expecting it to be Q3, but I’m hoping for Q2.

Where is the data coming from? Is this your own data or are you getting companies to give you the data? And are there separate SLMs for different companies?

I’ve been aggregating data as I go over the years, and so I’ve been able to establish a large data set over the last 10 years that is both structured in terms of earnings reports and a bunch of transcripts of stuff that the CEOs and CMOs would say…

So this is publicly available information?

Public, but of course as you go, you build analysis and market models around these things. All of it costs time, but all of it goes into the sausage in the end.

The unstructured data comes from when you take a step back and say, well, what’s being written out there? What are the academic articles? What are the case studies? What are the business cases? What are the interviews? What are the conversations out there with specific people that are tailored to certain companies or problem sets relevant to the games industry? In that sense, then all of a sudden it becomes a more complex task, but you get a much better insight.

Like, a CEO of a large publisher today, they started their career somewhere. What was their job? What did they do? What was their big moment? Why did they get elected to be CEO? As you track all these things, you get much better insight into what is likely to be their next decision, what is going to be their number one or two choice, because they’ve spoken about these things in the past.

It’s almost like an Oracle predicting what someone’s going to do.

It would be an Oracle if it was pulled out of a hat in a smoky grotto, but people are pretty consistent, I find, even in a creative industry. If somebody was the producer for a certain title or franchise, and their big success was making everything blue, you know that as a CEO later in their career, they’re going to advocate for blue more than any other colour.

The thing that I always try to bring it back to is, can we meaningfully predict how things go? The games industry is directed by only a few thousand people, really. So those are the kinds of questions that we look at.

How is this going to work from a practical standpoint? Are people going to ask you questions and then you feed it to the machine, or are you going to provide a version of this software for companies to use?

The initial rollout will be access to a localized small language model that feeds off of our data. So we create this whole data lake, we have all this market information, and we use that on the product side to build a leaderboard.

We say, OK, what are the biggest IPs out there across all of the gaming channels? Where do they live today, and what’s the effective reach? So we still express it in conventional metrics, but then very quickly, it [becomes] what does the audience look like? What’s the next step here?

You’ll have a localized small language model as well as more conventional PDFs just to kind of inform and get people on board slowly. And the SMS will be more prompt based. You’ll say, OK, my next projects for the coming three months are X, Y, Z. Here’s some of the details. You can do that and ask questions around your own project without it going back to some server in the cloud where we would see any proprietary information. So it would be totally secure.

So you can experiment in the safety of your own organization, but of course that requires up-to-date information.

So a company could purchase or subscribe to your software, and then have a local version of it which they can feed with their own data?

Exactly. It would be sort of like a secure data room. So you can just say, well, here’s our idea, what do you think? And then it runs it against all of what we know. And you can do that on either an incidental basis – when, let’s say, you are deciding on a big launch or a big investment – or you just do your weekly, ‘what’s going on in category X, Y, Z?’

In the white paper, I drafted some case scenarios. You and I might be very well versed [on the games industry], and we’re sort of endemic to it. But you see non-endemic companies looking at gaming in the same way that they looked at social media a decade ago. You have all these brand managers saying, I want to know how to deploy my brand in a gaming circumstance. And they don’t know where to go.

So we’re trying to cater to all these non-endemic companies as they recognize that gaming is a bigger piece of their overall business and an important marketing channel. So it’s both, we cater to the existing industry, of course, as well as the non-endemics that are entering this space.

It reminds me of the investment talk at Pocket Gamer Connects, when they were talking about private equity firms moving into gaming, but with many not really knowing anything about gaming.

Exactly. It’s been interesting to see how that goes, because for SuperData, a large part of its success was built on dealing with Wall Street. And look, they’re all nice people, but they only really know EA and maybe a few others. EA was the big one because all of these MBA degree financial investors, they all played FIFA in college. They played World of Warcraft, maybe some Call of Duty, but the more exotic stuff, the indies, would be totally outside of their scope.

So we spent a lot of time explaining how games work, just basic information, to these bankers that would invest billions of dollars with very little information about the industry at large. We see the same thing now with non-endemics, with private equity, with venture capital, it’s just the next generation of all this.

This interview has been edited for length and clarity.