Go Back   HWBOT forum > News > HWBOT Frontpage News

HWBOT Frontpage News Official HWBOT articles and frontpage news

Reply
 
Thread Tools Display Modes

3DMark Time Spy-Gate: In Summary

What do you get if take a pair of new GPU architectures, add a new API and a new benchmark?. Answer? A whole load of debate and plenty of discordant noise. It all started when some clued up people on the Overclock.net started debating the relative pros and cons of Futuremark?s decision to not implement any vendor specific architectural optimizations in its latest Time Spy benchmark which was rolled out earlier this week.

Claims from the AMD fanboy estate fiercely point out that AMD has invested a great deal in making sure that its architecture is optimized for the arrival of DX12, and more specifically, ?asynchronous computing?. Asynchronous computing in DX12 allows for applications to make specific choices about how the workload is distributed across GPU cores. This means that a 3D video game application can a) know which GPU is being used and b) make workload and queuing decisions to optimize the experience.

One example that is used to illustrate just how AMD have worked hard and succeeded at making its Polaris GPUs optimized in a specific title is the massively improved experienced in Doom when using the Vulkan API (an alternative to DX12 which also supports asynchronous computing). According to German tech reviewers Computebase, a true implementation of asynchronous compute would give AMD a significant performance boost, whereas Nvidia would see significantly less improvement.

So why did Futuremark decide to not implement vendor specific optimizations for asynchronous computing? After all it is a key feature of DX12, and Time Spy is billed as the first DX12 benchmark? A statement released by the company states:

?Asynchronous compute is one of the most interesting new features in DirectX 12?. The implementation is the same regardless of the underlying hardware. In the benchmark at large, there are no vendor specific optimizations in order to ensure that all hardware performs the same amount of work. This makes benchmark results from all vendors comparable across multiple generations of hardware. Whether work placed in the COMPUTE queue is executed in parallel or in serial is ultimately the decision of the underlying driver.?

So Futuremark are clearly trying to dispel any accusations of bias, instead arguing that using vendor specific optimizations would in fact be unfair. Users commenting on a reddit thread on the subject tend to disagree however:

??they just confirmed it's not a proper DX12 benchmark due to it not utilizing the benefits of DX12 low level optimization, all in the sake of "fairness" they used a single path... the path that fits Pascal architecture capabilites.?

Reading the forum thread on OC.net and other comments around web, it?s clear that emotions between green and red camps can certainly run high. My view is perhaps that the customer should ultimately have a choice. If I want to assess how well a GPU vendor is doing in terms of low level optimizations to ?get closer to the metal? of a GPU, why shouldn?t a benchmark app provide me with that opportunity? Likewise, if I am of the opinion that a single, common path or compute implementation is fairer, perhaps I should have that option too.

Yup. I vote for an on/off switch. Please add your thoughts in the forum thread below.


Jul 20, 2016 - article - overclock.net
  #1  
Old 07-20-2016
sdougal sdougal is offline
robot overlord
 
Join Date: Apr 2015
Location: Taiwan
Posts: 1,254
sdougal
Default 3DMark Time Spy-Gate: In Summary

Looking forward to all your views and opinions....
__________________
Reply With Quote
  #2  
Old 07-20-2016
K404's Avatar
K404 K404 is offline
nVidia FTW!
 
Join Date: Jan 1970
Location: Glasgowvic, Scotistan. Average Kingdom of Un-United Britain..
Posts: 3,855
K404
Default Re: 3DMark Time Spy-Gate: In Summary

PhysX is not allowed for Vantage (HWBot,) Tess can not be disabled for 3DM11 (FM) (am I right in saying that?), now this.

It's a synthetic bench. Normal rules of what's appropriate don't really come into it. If people want a better 3DMark score, they should buy the card that gives it, as the rules stand. or, if they're going to play actual games, they should buy the card that, overall, gives better performance in the games that they play.

I don't see what the fuss is about. This is only important uintil the next cards come out, or a new driver comes out with improvement X.

A lot of us use LN2 and voltmods. People aren't lining up to defend those as part of an equal playing field


First-world problems. "Boo hoo. I'm not allowed to tick/untick the box in my e-peen waving campaign"
__________________
Quote:
Originally Posted by Hondacity View Post
i believe in K404! :P
https://twitter.com/K404ExtremeOC

I do not believe in 1st place by any means, at any cost.

Quote:
Originally Posted by der8auer View Post
Knowing workarounds to get a benchmark to run is not a skill. It's a combination of knowledge and wasting time.
Reply With Quote
  #3  
Old 07-20-2016
FM_Jarnis FM_Jarnis is offline
robo cop
 
Join Date: Oct 2010
Location: FINLAND
Posts: 234
FM_Jarnis
Default Re: 3DMark Time Spy-Gate: In Summary

Sooo... Let's see: hypothetical "AMD optimized" and "Nvidia optimized" switches... who decides when they are "properly optimized"? Who programs the codepaths? If Futuremark does, you do know that as soon as the numbers are in, one or both "teams" yell how their codepath is not "properly optimized", right?

AMD, NVIDIA and Intel engineers have all seen the source and visited Futuremark offices in the past six months, discussing the implementation, offering their optimizations etc. and anything that helps while not hindering any other architecture is generally accepted. We still strongly doubt that game developers will spend the time & money to do vendor specific code paths (with the exception of sponsored games).

Also AMD, NVIDIA, Intel and Microsoft all have indicated that they do not want vendor-specific paths in 3DMark as it would devalue it as an useful neutral benchmark to them.

How about using games to figure out who has the best team of engineers for optimizing for their architecture (see green or red logo for the team responsible for each game) and 3DMark as the "neutral ground"?
__________________
Reply With Quote
  #4  
Old 07-21-2016
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,577
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default Re: 3DMark Time Spy-Gate: In Summary

How about each "team" commits their own codepath? The source code is available to all BDP members, so any trickery can be spotted. Wouldn't the situation be similar to the PCMark8 where you have the choice between a default and an accelerated benchmark? In this case the Time Spy benchmark can be used to show how fast the GPUs are without optimization and how fast they are with optimization.

That doesn't sound like a bad situation for the end-user?
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.
Reply With Quote
  #5  
Old 07-21-2016
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,577
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default Re: 3DMark Time Spy-Gate: In Summary

Oh, also, thumbs up for joining PCPer podcast

__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.
Reply With Quote
  #6  
Old 07-21-2016
FM_Jarnis FM_Jarnis is offline
robo cop
 
Join Date: Oct 2010
Location: FINLAND
Posts: 234
FM_Jarnis
Default Re: 3DMark Time Spy-Gate: In Summary

Quote:
Originally Posted by Massman View Post
How about each "team" commits their own codepath? The source code is available to all BDP members, so any trickery can be spotted. Wouldn't the situation be similar to the PCMark8 where you have the choice between a default and an accelerated benchmark? In this case the Time Spy benchmark can be used to show how fast the GPUs are without optimization and how fast they are with optimization.

That doesn't sound like a bad situation for the end-user?
Uh, any vendor-specific optimizations would, by definition be "trickery". The biggest issue is that as soon as you have separate paths, how do you truly enforce that the both do the exact same work. You really can't.

Most vendor-specific optimizations in games involve actually changing the work subtly to trade off some slight differences in image quality, or to use different algorithm for similar-but-not-identical output. Fine for games, when all you really care about is the framerate and that the output looks acceptable visually.

In a benchmark that makes a honest effort to be reproducible and fair, doing vendor-specific optimizations would rapidly turn into two benchmarks (then we'd have those mythical beast, "AMDmark" and "NVIDIAmark") and results from neither would be directly comparable.

PCMark 8 is very different - there the difference between Conventional and Accelerated is "do we use OpenCL for compute?". This would be comparable to a very very early benchmark offering "software renderer or DirectX or OpenGL?". Scores from those two are not comparable against each other.

3DMark is never designed to be the guy that squeezes every single vendor-specific processor cycle out of each GPU. In the real world, where game development schedules and budgets say you can't spend an year to gain 3% on one vendor tend to agree as well. It is quite well optimized, but in a generic way - optimizations that benefit on various hardware, but do not harm the performance on others.

I know HWBot "use case" is very very different from normal benchmarking, but even here the fact that each GPU from each vendor is actually pushed to do the same work should definitely matter.
__________________
Reply With Quote
  #7  
Old 07-22-2016
speed.fastest's Avatar
speed.fastest speed.fastest is offline
robo cop
 
Join Date: May 2010
Location: Surakarta, Indonesia
Posts: 439
speed.fastest is on a distinguished road
Default Re: 3DMark Time Spy-Gate: In Summary

Quote:
Originally Posted by FM_Jarnis View Post
In a benchmark that makes a honest effort to be reproducible and fair, doing vendor-specific optimizations would rapidly turn into two benchmarks (then we'd have those mythical beast, "AMDmark" and "NVIDIAmark") and results from neither would be directly comparable.
Maybe i'm blind, what in real world DirectX12 or Vulkan gaming, RX 480 is faster than GTX 1060 even at stock clock. But in 3DMark Time Spy GTX 1060 is faster, i don't know if Futuremark create 3DMark to emulate current gaming situation or make 3DMark is not biased to "AMDmark" or "Nvidiamark". But who i am
__________________
Reply With Quote
  #8  
Old 07-22-2016
FM_Jarnis FM_Jarnis is offline
robo cop
 
Join Date: Oct 2010
Location: FINLAND
Posts: 234
FM_Jarnis
Default Re: 3DMark Time Spy-Gate: In Summary

Quote:
Originally Posted by speed.fastest View Post
Maybe i'm blind, what in real world DirectX12 or Vulkan gaming, RX 480 is faster than GTX 1060 even at stock clock. But in 3DMark Time Spy GTX 1060 is faster, i don't know if Futuremark create 3DMark to emulate current gaming situation or make 3DMark is not biased to "AMDmark" or "Nvidiamark". But who i am
Have you considered how many of those DX12 games are actual DX12 engines, built DX12 first, and how many are DX11 games with DX12 renderer fitted in after-the-fact?

Unfortunately there is really no way to "prove" this either way until we have considerably more DX12 games. Could we take another look in, say, 12 months from now? I'm venturing an educated guess, with the backing of our engine team, that Time Spy will track the performance of DX12 titles over a larger sample-size quite well.

(adding Vulkan to DX12 comparisons gets from me "Objection, relevance!" since it is a different API, plus the only Vulkan game out there has AMD-optimized codepath only at this very moment, so using it for amd vs. nv comparisons is not the greatest idea)
__________________
Reply With Quote
  #9  
Old 07-22-2016
speed.fastest's Avatar
speed.fastest speed.fastest is offline
robo cop
 
Join Date: May 2010
Location: Surakarta, Indonesia
Posts: 439
speed.fastest is on a distinguished road
Default Re: 3DMark Time Spy-Gate: In Summary

Maybe that not "really dx12 engine", but from this review all i can say is even DX12 from Nvidia Games and "not true dx12", RX 480 is faster than GTX 1060. The source that i use for comparison : HARDOCP - Introduction - NVIDIA GeForce GTX 1060 Founders Edition Review

And i hope Time Spy is the next Sky Diver, not the next Firestrike
__________________

Last edited by speed.fastest; 07-22-2016 at 08:24.
Reply With Quote
  #10  
Old 07-22-2016
buildzoid's Avatar
buildzoid buildzoid is offline
robo cop
 
Join Date: Jan 1970
Location: United Kingdom
Posts: 368
buildzoid
Default

If games use different code paths and vendor specific optimizations benchmarks should do that too. Especially with a low level API like DX12 or Vulkan which were created to allow programmers to play to the strengths of each architecture. The way I see it 3Dmark should have an Nvidia path and an AMD path and implement all available speed increasing tricks in each path so long that the tricks don't impact visual quality.
__________________
Keep calm and raise Vcore


My Blog
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
jiccman1965 - GeForce GTX 570 - 25655 marks 3DMark - Cloud Gate [GF]Duke Result Discussions 5 12-06-2013 18:46
Europe to allow gate-to-gate electronics use by the end of November hwbotrss Tech News RSS 0 11-14-2013 11:30
{SAS}TB - 3x Radeon HD 7970 - 38487 marks 3DMark - Cloud Gate ObscureParadox Result Discussions 1 02-08-2013 23:43
3DMark Cloud Gate Trailer Posted hwbotrss Tech News RSS 0 12-17-2012 20:11


All times are GMT +1. The time now is 15:44.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright HWBOT 2004 - 2015