But Is It Research

I came across an interesting article about World of Warcraft a few days ago. Blizzard CEO Mark Morhaime revealed, in an interview, that only about 30% of players in World of WarCraft actually go past the level 10 mark in the game. In fact, directly quoting him from the article:

"Our research shows that trial players who play World Of Warcraft past level 10 are much more likely to stick with the game for a long time. Currently, only about 30% of our trial players make it past this threshold. So anything we can do to improve the new player experience is a huge opportunity for us," he said.

Since this blog is about research and that claim is about research, I thought it might be worthy of comment. That is an interesting statistic and I don't doubt that it is true. But is it research? And what can you do with it?

First, on the question of "is it research", I would say no. It is not research. Earlier in this blog I referred to a concept introduced by Francis Bacon called Idols of the Marketplace. It is the common place usage of terms that have more precise definitions in the advancement of knowledge. Research is one of those terms. In common place usage, we might say, "I did a lot of research before I bought my car." In common place meaning that is perfectly acceptable. We understand it to mean that we gathered information before making a decision. However, in the world of research this term has a very different meaning. Research must advance knowledge. Typically, this means a well defined claim, based upon well defined concepts where the claim is either supported or refuted advancing our understanding of the concepts involved.

Since this is not research, it is unclear what should be done, if anything, about the derived number.

What Blizzard has done is some data analysis which yielded a parameter the meaning of which is entirely unclear. The quote suggests that the problem can be addressed by improving new player experience. But is that the case? Is there really a problem? Let's say the 70% who don't make it to level ten are regular players who download the trial version to amuse themselves while they are in a lab, or when their regular server is down, or for a class, or a demo. If that is the case, nothing needs to be done. What if the 70% include people who use trial version to get used to the game. So, they set up a trial account, use it for ten days, set up a new one and so on. Let's say, for the sake of argument, that each player does this three times before achieving a level 10 and converting to a regular account. If this is the case then the trial period should be extended. Finally, let's say that the 70% are people who play some other game and are trying WoW just so they can say they tried it. If this is the case then nothing should be done.

The point here is that we think the 30% parameter is meaningful information and it is not. We don't know what it means and we don't know what was measured. Hence, we really do not know what to do. If Blizzard were worried about retention, ease of use or some other factor then it would be possible to set up a research study to get to the bottom of it. But as it is, they don't have research. They only have a number, the meaning of which is entirely unclear.