nVidia and FutureMark’s 3DMark03

Introduction

Unless you have been out of the loop recently, you probably have heard of a new rise in the battle between nVidia graphics drivers and their use with 3DMark03. Let me give you the brief story:

On January 19th, Martin Luther King Jr. Day, I received an email from FutureMark asking that I kindly remove my 3DMark03 scores because I used a driver that was not listed on their "Approved" Drivers List. More information to follow on the "Approved" Drivers List. Nate and I did some quick scrambling to find out why FutureMark would ask such a question and what grounds they had to do so.

The first place we looked: the EULA for 3DMark03.

The End-User License Agreement:

When thinking about the situation legally, we decided to first look at the End-User License Agreement (EULA) to see if we had any leg to stand on if we decided to refuse. Low and behold, the second paragraph in the EULA states the following:

"PLEASE NOTE that Section 11 of this Agreement contains terms, which restrict Your right to publish any Results obtained by You using drivers that do not fulfil Futuremark Optimization Guidelines requirements. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."

Since everyone who uses 3DMark03 agrees to this EULA, we really had no way to refuse their request. So five minutes later, FutureMark’s request was completed, but our quest to find out why had just begun. To conclude our findings within the EULA, the following is information pertaining to the above statement from Section 11:

"Notwithstanding foregoing, You are allowed to publish only such Results that are obtained using drivers that fulfil requirements of Futuremark Optimization Guidelines. Updated Optimization Guidelines and list of drivers verified by Futuremark can be found on Futuremark’s website at www.futuremark.com."

The "Approved" Drivers List:

In case you never pay attention to the EULA and never heard of FutureMark’s Optimization Guidelines, here is the link to FutureMark’s "Approved" Drivers List. Here is the description of this page from FutureMark’s website:

"Here we have listed all WHQL drivers which are approved to be used only with Build 340 of 3DMark®03. If you haven’t got Build 340 of 3DMark03, you can download the patch here. By using these drivers & Build 340 you will get a valid and fully comparable 3DMark03 result. We will update the driver information continuously as new WHQL certified drivers are reviewed. Please note that the approved drivers listed below have been tested only with the 3DMark03 default benchmark settings!"

Another interesting point is the "note" given after showing that the nVidia 52.16’s are the only Approved Driver for use.

"The NVIDIA ForceWare 52.16 Drivers have been tested with the GeForce3 series, GeForce4 series and the GeForceFX series. The drivers have not been tested with the MX-series! There are no Win9x or WinME WHQL drivers available supporting all the GeForce series. The 52.16 drivers have 3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between nvidia cards."

So, FutureMark is actually informing everyone that you can use these drivers for an accurate 3DMark Score, but not when comparing tests using just the Pixel Shader 2.0 test. According to a response from FutureMark given in Adrian’s editorial on the matter, the usage of Pixel Shader 2.0 in the Nature game test is not affected by this known optimization in the 52.16 driver.

Since it seems that FutureMark is legitimately attempting to keep their benchmark fair and unbiased, let’s next look towards nVidia’s drivers to find out why the 52.16’s are approved and the 53.03’s are not.