ATi & nVidia in driver "optimizations" race

July 26, 2003 / by Leonidas / page 1 of 9

This translation has been made possible by our community who wanted me to provide this articles' information to more than just the german-speaking audience. We'd like to apologize for our language skills.

   Intro

During the course of preparing our upcoming graphics cards roundup of the current highend chips of nVidia and ATi, we wondered, and still wonder which benchmarks to use. This situation is unprecedented. Due to the last information about driver "optimizations" we unfortunately cannot be sure for which benchmarks the chip manufactures have optimized, either legitimately or not.

The main cause of this uncertainty is an article by Digit-Life which proves that ATi and nVidia use special driver optimizations for 3DMark2001. Some benchmark results from 3DCenter Forum raised even more suspicion. These were generated with the AntiCheat scripts Digit-Life have provided along the aforementioned article. The scripts remove application specific optimizations from drivers and we've seen a clear performance disparity between the original and modified drivers in 3DMark03, Unreal Tournament 2003 and Codecreatures benchmarks.

To get some confidence in our own benchmarks for our next graphics chip roundup, we'll largely follow Digit-Life's testing methology for our benchmarks, using both the original drivers (nVidia 44.65 and ATi 03.4, just like in Digit-Life's article) and the AntiCheat script modified versions.

We're perfectly aware that the AntiCheat scripts may not be sufficient to defeat all optimizations - however at this time they're the best tool for the job, by virtue of being the only one available. These scripts may also have the unwanted side effect of removing perfectly legal optimizations - in this case we can only detect that there is an optimization. Whether it's valid or not can only be judged by inspecting screenshots.

This article relies on Digit-Life's AntiCheat scripts completely deactivating all application specific optimizations without unwanted side effects. Should this assumption be proven to be false, some statements of this article will become obsolete.

Unfortunately, these AntiCheat scripts work only for Direct3D, so we used Serious Sam: The Second Encounter and also Counter-Strike with their Direct3D API. A test of the OpenGL versions of these games and of pure OpenGL games is not possible with these scripts.

The observations in this article apply to the following driver sets, unless explicitly stated otherwise: nVidia 44.03, 44.65, 44.67, 44.68, 44.71, 44.90, 45.23 and 45.24 + ATi 03.4, 03.5 and 03.6. Initially this article was started using the 44.65 nVidia drivers and 03.4 ATi drivers, the 44.03, 44.67, 44.68, 44.71 and 44.90 nVidia drivers + 03.5 und 03.6 ATi drivers were checked later. August 14 2003 we've also checked nVidia driver versions 45.23 and 45.24, but these perform just like the other drivers we tested earlier.

   nVidia Optimizations

In the following benchmarks we used 8x anisotropic filtering to ensure graphics chip limited scenarios, so driver optimizations become more apparent:

There clearly is a remarkable impact on 3DMarks and Unreal Tournament 2003 scores. The nVidia driver definitely contains application specific optimizations for these games. For all other games it does not seem to be the case - or rather the AntiCheat script can't recognize any application specific optimizations for the other games, we unfortunately cannot entirely dismiss this possibility.

The observation that application specific optimization is present for 3DMark2001, 3DMark03 and Unreal Tournament 2003 does not yet prove that these optimizations are unallowed - but there's a catch. Application specific optimizations should not affect theoretical benchmarks like the 3DMarks because the majority thinks these are unallowed optimizations. The hardware should only do what the developer of the theoretical benchmark intended.

Any simplification, even if it does not change image quality, contradicts this principle and actually circumvent the purpose of a theoretical benchmark. The 3DMark developers Futuremark share the same opinion ("3DMark03 Audit Report", PDF, 1 MB). Regarding theoretical benchmarks, any application specific optimization is an "unallowed optimization", an opinion we strongly agree with.

We should add that all our 3DMark03 testing has been performed with build 330, which was published by Futuremark to disable the optimization of nVidia and ATi. Unfortunately Futuremark haven't discovered all of nVidia's "optimizations" - and apparently they didn't discover the "optimizations" for 3DMark2001 at all.