Nubia defends its ethics in REDMAGIC benchmark scandal

Most of the modern smartphone industry has blessedly moved on from the numbers games that plagued its early days, when manufacturers would hype up their phones with the highest megapixel cameras, or most CPU cores. And while we’re much more focused on features and the experience today, benchmark numbers still manage to attract attention — especially when we’re dealing with performance-geared gaming phones. That’s tempted more than one company to game the system by artificially boosting those benchmark results, and today we’ve got some updates to share on the latest scandal to hit.

Earlier this week, we shared with you the saga of Nubia’s REDMAGIC 11 Pro phones. 3DMark parent UL Solutions delisted the REDMAGIC 11 Pro and Pro+ from benchmarks with the explanation that the phones were not following its benchmark rules. Typically, we see this when a company configures its devices to recognize when a benchmark app is running, and then disables things like power or thermal throttling, giving it an unfair advantage.

In some tests with the REDMAGIC 11 Pro, the phones would get so hot during those unthrottled benchmarks that they had to shut down before even completing tests.

At the time, Android Authority reached out to Nubia for comment, and the manufacturer seemed to defend its practices. To an extent, it has a point: This is real performance the phone is showing, after all. But doesn’t delivering better-than-normal performance during benchmarks come across as a bit deceitful?

We got back in touch with Nubia, asking the company specifically about the ethical concerns involved here. And now we have our responsive, with Nubia telling us:

We do not view our performance profiles as unethical. Rather, we see them as transparency toward the hardware’s true capabilities.

Unlike other smartphones, REDMAGIC is engineered with an internal physical fan and liquid cooling specifically to handle extreme thermal loads.

Regarding “real-world conditions,” these performance levels are fully accessible to our users. Through “Game Space” and by enabling settings like “Diablo Mode,” players have the manual “key” to unlock the same high-wattage performance and thermal ceiling used in benchmarks for any demanding application, such as high-end PC emulation.

We believe the user should decide how to balance power and heat. Our benchmarks simply reflect what the hardware can achieve when those user-accessible cooling and performance features are fully engaged.

But is that really what’s going on here? It would be one thing if the phone were asking the user to manually make all the decisions about performance throttling, but there are clearly different profiles in place for when the phone recognizes specific apps.

We also reached out to UL Solutions in an attempt to verify exactly how the REDMAGIC phones were supposed to be violating 3DMark rules. The company replies:

We can confirm that we have delisted the REDMAGIC 11 Pro and REDMAGIC 11 Pro+ after verifying this ourselves for breaking our Benchmark Rules and Guidelines.

Further, UL Solutions points us to a statement it published on the REDMAGIC delisting, where it confirms that the 11 Pro phones performed up to 24% better on the public 3DMark app than they did on an internal, camouflaged version — indicating the phone was recognizing that it was being benchmarked and manipulating performance settings.

In our testing, this higher-performance behavior was enabled by default when the phone detected 3DMark running, and we were unable to find a way to disable it when running 3DMark. As a result, 3DMark cannot produce results for these models that reflect normal, real-world gaming behavior.

Basically, we’re at a bit of an impasse. UL Solutions considers this kind of behavior to be incompatible with its benchmarking, and Nubia seems content with the way the REDMAGIC 11 Pro scales performance. Considering the outcome, it feels like a self-defeating measure from Nubia — what’s even the point of boosting your numbers, if benchmarks are just going to lock you out?

Despite this outcome, we would not be surprised in the least if this isn’t the last time a phone maker tries to get away with shenanigans like this.