Intel graphics drivers employ questionable 3DMark Vantage optimizations 3DMurk all over again?

In the early days of GPUs, application-specification performance optimizations in graphics drivers were viewed by many as cheating. Accusations were hurled with regularity, and in some cases, there was real cheating going on. Some optimizations surreptitiously degraded image quality in order to boost performance, which obviously isn't kosher. Optimizations that don't affect an application's image quality are harder to condemn, though, especially if you're talking about games. If a driver can offer users smoother gameplay without any ill effects, why shouldn't it be allowed?

The situation gets more complicated when one considers optimizations that specifically target benchmarks. Synthetic tests don't have user experiences to improve, just arbitrary scores to inflate. Yet the higher scores achieved through benchmark-specific optimizations could influence a PC maker's choice of graphics solution or help determine the pricing of a graphics card.

Futuremark's popular 3DMark benchmark has been the target of several questionable optimizations over the years. Given that history, it's not surprising that the company has strict guidelines for the graphics drivers it approves for use with 3DMark Vantage. These guidelines, which can be viewed here (PDF), explicitly forbid optimizations that specifically target the 3DMark Vantage executable. Here's an excerpt:

With the exception of configuring the correct rendering mode on multi-GPU systems, it is prohibited for the driver to detect the launch of 3DMark Vantage executable and to alter, replace or override any quality parameters or parts of the benchmark workload based on the detection. Optimizations in the driver that utilize empirical data of 3DMark Vantage workloads are prohibited.

No ambiguity there, then: Vantage-specific optimizations aren't allowed.

Intel may not be playing fair, though. We recently learned AMD has notified Futuremark that Intel's 15.15.4.1872 Graphics Media Accelerator drivers for Windows 7 incorporate performance optimizations that specifically target the benchmark, so we decided to investigate.

We tested 3DMark Vantage 1.0.1 with these drivers on a G41 Express-based Gigabyte GA-G41M-ES2H motherboard running the Windows 7 x64 release-to-manufacturing build, a Core 2 Duo E6300, 4GB of DDR2-800 memory, and a Raptor WD1500ADFD hard drive.

We first ran the benchmark normally. Then, we renamed the 3DMark executable from "3DMarkVantage.exe" to "3DMarkVintage.exe". And—wouldn't you know it?—there was a substantial performance difference between the two.

Our system's overall score climbs by 37% when the graphics driver knows it's running Vantage. That's not all. Check out the CPU and GPU components of the overall score:

The GPU score jumps by a whopping 46% thanks to Intel's apparent Vantage optimization. At the same time, the CPU score falls by nearly 10%. Curious.

Next, we ran a perfmon log of CPU utilization during each of 3DMark's CPU and GPU component tests. Vantage takes its sweet time loading each test, so our start and end times aren't perfectly aligned for each run. However, the pattern is pretty obvious.

In the GPU tests, the system's CPU utilization is much higher with the default executable than with the "3DMarkVintage" executable. There isn't much difference in CPU utilization in the CPU tests, though.

What's really going on
Intel appears to be offloading some of the work associated with the GPU tests onto the CPU in order to improve 3DMark scores. When asked for comment, Intel replied with the following:

We have engineered intelligence into our 4 series graphics driver such that when a workload saturates graphics engine with pixel and vertex processing, the CPU can assist with DX10 geometry processing to enhance overall performance. 3DMarkVantage is one of those workloads, as are Call of Juarez, Crysis, Lost Planet: Extreme Conditions, and Company of Heroes. We have used similar techniques with DX9 in previous products and drivers. The benefit to users is optimized performance based on best use of the hardware available in the system. Our driver is currently in the certification process with Futuremark and we fully expect it will pass their certification as did our previous DX9 drivers.

This CPU-assisted vertex processing doesn't appear to affect Vantage's image quality. However, Intel is definitely detecting 3DMark Vantage and changing the behavior of its drivers in order to improve performance, which would appear to be a direct contravention of Futuremark's guidelines.

At present, Intel's 15.15.4.1872 graphics drivers don't appear on Futuremark's approved driver list for 3DMark Vantage. None of the company's Windows 7 drivers do. The 7.15.10.1554 Windows Vista x64 drivers that are on the approved list don't appear to include the optimization in question, because there's no change in performance if you rename the Vantage executable when using those drivers.

Violating Futuremark's driver optimization guidelines is one thing, but Intel also claims it's offloading vertex processing to enhance performance in games. Indeed, the very same INF file that singles out 3DMarkVantage.exe also names other executables.

HKR,, ~3DMarkVantage.exe, %REG_DWORD%, 2 HKR,, ~3DMarkVantageCmd.exe, %REG_DWORD%, 2 HKR,, ~CoJ_DX10.exe, %REG_DWORD%, 2 HKR,, ~Crysis.exe, %REG_DWORD%, 2 HKR,, ~RelicCoH.exe, %REG_DWORD%, 2 HKR,, ~UAWEA.exe, %REG_DWORD%, 2

One of the games on the list for detection, Crysis Warhead, should have no problem saturating an integrated graphics chipset, to say the least. We tested it with the executable under its original name and then renamed to Crisis.exe, using FRAPS to collect real-world frame rate data with the game running at 800x600 and minimum detail levels.

Intel's software-based vertex processing scheme improves in-game frame rates by nearly 50% when Crysis.exe is detected, at least in the first level of the game we used for testing. However, even 15 FPS is a long way from what we'd consider a playable frame rate. The game doesn't exactly look like Crysis Warhead when running at such low detail levels, either.

Our Warhead results do prove that Intel's optimization can improve performance in actual games, though—if only in this game and perhaps the handful of others identified in the driver INF file.

How do Intel's driver optimizations affect the competitive landscape? To find out, we assembled an AMD 785G-based system that's pretty comparable to the G41 rig we used for testing: Athlon II X2 250 processor, Gigabyte GA-MA785GPMT-UD2H motherboard, the same Raptor hard drive, and 4GB of DDR3 memory running at 800MHz with looser timings than the Intel system. We even disabled the board's dedicated sideport graphics memory, forcing the GPU to share system RAM like the G41.

With the Futuremark-approved Catalyst 9.9 drivers, the AMD 785G-based system scored 2161 in 3DMark Vantage—nearly the same score as the 2132 3DMarks the G41 gets when it's playing by the rules, but well below the 2931 the score the G41 posts with optimizations enabled. (Renaming the Vantage executable on the AMD system had no notable effect on benchmark scores.) The app-specific optimization gives the G41 a definitive lead in 3DMark Vantage.

Here's the tricky part: the very same 785G system managed 30 frames per second in Crysis: Warhead, which is twice the frame rate of the G41 with all its vertex offloading mojo in action. The G41's new-found dominance in 3DMark doesn't translate to superior gaming performance, even in this game targeted by the same optimization.

That's no great shock, really. We've seen Intel's integrated graphics solutions thoroughly outclassed by rivals from AMD and Nvidia on multiple occasions.

All of which brings us back to the perils of using 3DMark Vantage as a substitute or proxy for testing real games. Those perils are well established by now. PC makers and others in positions of influence would do well to re-train their focus on real applications—especially for testing integrated graphics solutions, which have no need of advanced graphics workloads based on the latest version of DirectX to push their limits. 3DMark's traditionally purported role as a predictor of future game workloads makes little sense in the context of these modest IGPs.

We're curious to see what FutureMark will make of Intel's Windows 7 graphics drivers. As far as we can tell, the latest GMA drivers are in violation its rules. We've asked Futuremark for comment on this issue, but the company has yet to reply.