Interview with Futuremark

September 20, 2003 / by aths / page 1 of 1

Our interview partner is Patric Ojala from Futuremark.

1) Hello, and welcome to 3DCenter. Please introduce yourself to our readers, and tell us about your job at Futuremark Corporation, if you'd be so kind.

Futuremark: Thank you, it's good to be here!

My name is Patric Ojala, Senior Manager of Benchmark Development at Futuremark. I'm the producer of the 3DMark benchmark product series, and I'm also the head of the 3D development team at our company. Together with my highly talented team, we develop the popular 3DMark benchmarks, featuring some of the most forward looking and best looking real-time 3D graphics available on the PC.

2) Can you give us an idea about the size of Futuremark Corporation, and the differences between the US and Finnish offices?

Futuremark: We are 28 employees all in all, with 3 marketing and sales persons in Saratoga, CA. The remaining 25 are here in Finland, handling R&D (Research and Development) and management.

3) Futuremark, Remedy and the Bitboys all originated from the "Future Crew" group. Both 3DMark2001 and Max Payne take advantage of "Max FX" technology. Do Futuremark and Remedy still cooperate in terms of 3D engines, and if yes, in what respect?

Futuremark: We still cooperate, but we don't share the engine anymore. All Max Payne IP was sold to the publisher as far as I know, and the rapid pace of the 3D technology encouraged us to implement a modern DX wrapper engine. That kind of engine is just a thin program layer on top of DirectX, that loads the artwork and the shaders. DirectX handles the rest. You could say that 3DMark03 basically measures DirectX performance, and not the performance of a certain "3D engine".

Remedy's and our offices are still next door, and the door between our companies is always open. We borrow resources from each other, and otherwise too try to help eachother, but we are two separate companies.

4) Personally, I like the included demos of 3DMark 99 MAX and 3DMark 2000 better than the 2001 and 03 demos. "XL-R8R" could almost be considered a straight demo, even. Will there be a return of those "old" demos with continuous storylines, will Futuremark focus more on game test demos as found in the latter 3DMarks, or scrap demos alltogether?

Futuremark: Back in the 3DMark 99 and 2000 days, our goal was to gain popularity on the market by releasing amazing demos. We therefore invested more in making the demos, and less in making benchmark tests. As we have become a more important player in the benchmarking industry, all our resources are needed in perfecting the tests. Even though it would be fun, there is little time left over for pure demo development. I don't think we'll ever scrap the demo, but we will spend relatively more time on the game tests than on the demo, compared to the 3DMark 99 or 2000 development days.

5) In THAT audit report, Futuremark accused nVidia of cheating, using the word "cheat" itself. Later on, Futuremark backed down and changed the wording to "optimisations" instead. Our readers asked whether you consider the original audit report to have been inaccurate, or whether Futuremark opted for the new term "optimisation" for purely semantic reasons? Or was there any other reason to change the wording?

Futuremark: The real substance of the audit report was correct and still is. But we realized that there was no clear definition of terms. Thus, using the 'C-word' was perhaps a little bit inappropriate. Many people misunderstood the purpose of the paper and thought that it was meant to be something else than to defend 3DMark. We have had a long and productive relationship with nVidia, just like we have had with the other major PC hardware manufacturers.

6) Some aspects of 3DMark03 were critisised quite heavily by members of the online 3D community as well as a certain IHV. We would be glad if you gave us your personal opinion on the following two: a) Game Test 1 used a lot of single texturing, whereas nearly all actual, current games are using multitexturing. b) It appears as if 3DMark03 contained only one set of DirectX 9 shaders for all DX9 cards, but it seems likely that "real" games will contain CineFX-optimised shaders, also.

Futuremark: I haven't heard about the issues of that "Lab Report" in the last half a year, not even from that 'certain IHV' itself :-).

a) Most polygons (my guess is over 90%) in that test are quad textured (4 texture layers). Then again, it is typical for a flight game to have most pixels on screen in the sky. Skyboxes almost always have been and mostly still are single textured. There is no reason to artificially add texture layers to achieve multitexturing, since games don't do that either. During development we tried adding a second texture layer on the sky, but the performance readings remained within 3%. Game test 1 was designed to be the light weight DX7 compatible test, giving decent framerates also on value hardware. The flight scene we have in game test 1 fulfills its requirements very nicely.

b) The higher level language of DirectX is HLSL, and that is what we will use. I don't see how an impartial benchmark could use IHV specific APIs, unless all competing companies have a chance to do the same. We have gotten strong support from the key IHVs supporting the usage of HLSL in 3DMark.

7) Other people claim that 3DMark03's framerate is too low considering the actual visuals. Rumours abound that Futuremark decided to augment the workload with additional, "unnecessary" (because "invisble") calculations to make the benchmark more stressful to current and especially future graphics boards. Should this be true, how did Futuremark make sure that the balance of different types of calculations are comparable to future DX9 game engines?

Futuremark: Again we're going back to the 6 month old discussion raised by this 'lab report'. I think everyone remembers that already back then we proved all claims in that report to be baseless. Later on we have witnessed that the same IHV has product launch demos implementing shadows the very same way we did. One way to implement stencil shadows is to do less calculations, but slower on the CPU. This implementation also sets some restrictions on the artwork.

The other alternative is to do more work but faster on the GPU, also giving more freedom regarding the artwork. We chose the latter, since the DX9 hardware available during 3DMark03 development gave quite decent frame rates with the more GPU intensive implementation (~30fps). Since the 3DMark03 game tests above all test the graphics performance, there should be no question about which implementation to use.

8) 3DMark03's results listing also contains clock frequencies for both GPU and CPU as well as FSB. But it is fairly easy to wangle the results by changing clock frequencies just before the release of the test results. Is Futuremark aware of this, and are any procedures planned or already in effect that prevent such adulteration?

Futuremark: This should be prevented already in our build 330 of 3DMark03. The benchmark client might in some cases show this incorrectly, but our online result database reads it correctly. Then again, if you intentionally try to produce these incorrect results, and feel hacker satisfaction when it is displayed incorrectly just on your local system, you're only fooling yourself :-).

9) 3DMark measures the performance of 3D accelerators, but image quality is not included as a comparison bias in the final results for a default run. Image quality features such as anti-aliasing and anisotropic filtering have become more and more important even to casual gamers, but unfortunately for comparative IQ analysis and benchmarkers, the techniques used differ considerably between major IHV. How will Futuremark address this problem in the future, and will there be a comfortable way to compare the performance with visually (near-)identical image quality settings?

Futuremark: This is one of my favourite subjects and I guess I could write a whole book about it. The problem is how to implement an automatic IQ (image quality) tool for features like AA and advanced texture filtering. If there were a reliable implementation available, we would definitely have used it, but so far no good solution has been found.

It is therefore up to the user to manually confirm that the IQ is what it should. 3DMark03 does contain the tools for such investigations, but it does require some knowledge of the user to be able to carry out that exercise. As for texture filtering, 3DMark03 Pro has a separate tool highlighting the differences in various texture filtering methods. Please read the 3DMark03 help file for more info on the usage of that tool and on how to interpret the results.

10) Thank your for your time. Is there anything you'd like to add and tell our readers?

Futuremark: There has been a lot of controversial discussions around 3DMark03, but I would like you to keep one thing in mind. 3DMark03 was developed in cooperation with all the major PC hardware manufacturers, just like 3DMark2001. That enabled us to create a product that was ahead of its time. We understand that many people did not believe the performance figures when 3DMark03 was published back in February. Only now we are beginning to see DX9 games emerge and what do you know, their performance readings seem to correlate well with what 3DMark03 showed you already in February :-).