Go Back   HWBOT forum > HWBOT Information > HWBOT Software and Apps > HWBOT Unigine Heaven Benchmark

HWBOT Unigine Heaven Benchmark HWBOT's first own benchmark application using the Unigine Heaven engine ...

Reply
 
Thread Tools Display Modes
  #11  
Old 02-05-2011
1Day's Avatar
1Day 1Day is offline
transformer
 
Join Date: Jun 2007
Location: Outside of London
Posts: 621
Thanks: 0
Thanked 0 Times in 0 Posts
1Day is on a distinguished road
Default

Welcome and nice first post. LOL

And what do you suggest as a solution?
__________________
Still the best benchmark of them all.

Reply With Quote
  #12  
Old 02-05-2011
David's Avatar
David David is offline
kitchen robot
 
Join Date: Jun 2010
Location: UNITED STATES
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
David
Default

Quote:
Originally Posted by OSWiz View Post
My BS detector just went off!
Its obvious to me anyway, that HWBot has a double standard going on here. I can't submit a Vantage score with PhysX turned on, but I can can submit a score with tessellation turned off in the Heaven bench. Just because your bot can't determine whether or not tessellation is on or off does not excuse the double standard. I realize that a lot of scores are submitted where cheats have been used, I prefer NOT to cheat!

What can I say, other than "Amen". It would appear that HWBOT has just sanctioned a legal viable cheat. I can no longer feel comfortable submitting my benches to HWBOT.
__________________
_____________
__________
Cooler Master HAF 932 - Corsair AX1200 - Asus Rampage III Formula - i7 930 @ 4.0GHz 24/7 - ThermalRight Ultra 120 Rev. C
6GB G.Skill 6-6-6-18 1T - eVGA GTX 580 SLI - GTX 460 Dedicated PhysX - Intel X25-M 80GB Boot Drive - 3x500GB WD Spinners
Sony & Asus Optical Drives - NZXT Sentry LX and Scythe Kaze Master Pro 5.25 Fan Controllers
Samsung 2494SW 24" Display - Logitech G15, G13, G500, Performance MX, and C310
Reply With Quote
  #13  
Old 02-05-2011
David's Avatar
David David is offline
kitchen robot
 
Join Date: Jun 2010
Location: UNITED STATES
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
David
Default

Quote:
Originally Posted by 1Day View Post
Welcome and nice first post. LOL

And what do you suggest as a solution?
A solution will take considerable thought, however, HWBOT could at least not sanction this protocol. At the very least, make it clear that the modification of any of a benchmarks defaults by the use of driver software, or any other means is considered cheating, and will not be tolerated. Certainly HWBOT can do that much.
__________________
_____________
__________
Cooler Master HAF 932 - Corsair AX1200 - Asus Rampage III Formula - i7 930 @ 4.0GHz 24/7 - ThermalRight Ultra 120 Rev. C
6GB G.Skill 6-6-6-18 1T - eVGA GTX 580 SLI - GTX 460 Dedicated PhysX - Intel X25-M 80GB Boot Drive - 3x500GB WD Spinners
Sony & Asus Optical Drives - NZXT Sentry LX and Scythe Kaze Master Pro 5.25 Fan Controllers
Samsung 2494SW 24" Display - Logitech G15, G13, G500, Performance MX, and C310
Reply With Quote
  #14  
Old 02-05-2011
Christian Ney's Avatar
Christian Ney Christian Ney is offline
Single Point of Contact
 
Join Date: Apr 2010
Location: Geneva, Switzerland
Posts: 7,490
Thanks: 3
Thanked 369 Times in 168 Posts
Christian Ney is on a distinguished road
Default

There is no way to know what was used in the drivers
Reply With Quote
  #15  
Old 02-05-2011
David's Avatar
David David is offline
kitchen robot
 
Join Date: Jun 2010
Location: UNITED STATES
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
David
Default

Quote:
Originally Posted by Christian Ney View Post
There is no way to know what was used in the drivers
Granted, I certainly won't argue that point. All I am saying is HWBOT should not condone it. As it stands now they are condoning it by saying that they don't care what you do with drivers, it's ok, it's not an issue. That seems to be the Bot's position as of now, and that's a green light. That's all I'm trying to say.
__________________
_____________
__________
Cooler Master HAF 932 - Corsair AX1200 - Asus Rampage III Formula - i7 930 @ 4.0GHz 24/7 - ThermalRight Ultra 120 Rev. C
6GB G.Skill 6-6-6-18 1T - eVGA GTX 580 SLI - GTX 460 Dedicated PhysX - Intel X25-M 80GB Boot Drive - 3x500GB WD Spinners
Sony & Asus Optical Drives - NZXT Sentry LX and Scythe Kaze Master Pro 5.25 Fan Controllers
Samsung 2494SW 24" Display - Logitech G15, G13, G500, Performance MX, and C310
Reply With Quote
  #16  
Old 02-05-2011
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,579
Thanks: 808
Thanked 1,136 Times in 557 Posts
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default

Quote:
Originally Posted by OSWiz View Post
My BS detector just went off!
Its obvious to me anyway, that HWBot has a double standard going on here. I can't submit a Vantage score with PhysX turned on, but I can can submit a score with tessellation turned off in the Heaven bench. Just because your bot can't determine whether or not tessellation is on or off does not excuse the double standard. I realize that a lot of scores are submitted where cheats have been used, I prefer NOT to cheat!
PhysX has been prohibited because it affects a CPU benchmark by using GPU features. Additionally, Futuremark has also banned PhysX from their official HOF rankings for the exact same reason. In other words, it's not really a double standard (although I can understand why you see it that way).

Being able to change the levels of tesselation through the drivers is, although annoying since we spend so much time on this, nothing more than adjusting the AA, AF or LOD settings. Please also note that when we became aware of this option, we instantly tried to find out if it's possible to detect override the driver settings. Sadly enough, that's not possible.

By the way, adjusting tesselation levels also affects the 3DMark11 score.
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.
Reply With Quote
  #17  
Old 02-05-2011
jjjc's Avatar
jjjc jjjc is offline
robo cop
 
Join Date: Aug 2009
Location: Australia
Posts: 389
Thanks: 0
Thanked 0 Times in 0 Posts
jjjc
Default

I think that the general answer from everyone here is that we can't detect it so it's all fine by us. While it may be undetectable by the bench, it still shouldn't be condoned and should be condemned, it puts AMD cards at an unfair advantage, particularly when we're talking about global points.

Perhaps HWBOT should go the way of Futuremark, and require drivers to be approved before they can be used in rankings. Surely, drivers that allow certain settings to be forced off with no notification to the program wouldn't be approved and therefore this wouldn't be an issue.

Just an idea.
__________________
Reply With Quote
  #18  
Old 02-05-2011
David's Avatar
David David is offline
kitchen robot
 
Join Date: Jun 2010
Location: UNITED STATES
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
David
Default

Quote:
Originally Posted by Massman View Post
...Please also note that when we became aware of this option, we instantly tried to find out if it's possible to detect override the driver settings. Sadly enough, that's not possible.

By the way, adjusting tesselation levels also affects the 3DMark11 score.
Thanks for your input, I am enjoying all points of view on this issue. Yes, I am aware of the advantages of turning off Tessellation in 3dmark11, and of course disagree with that as well. Personally I find Heaven 2.1 the most annoying because some people are turning off Tessellation in a Tessellation benchmark, which makes it especially absurd.
__________________
_____________
__________
Cooler Master HAF 932 - Corsair AX1200 - Asus Rampage III Formula - i7 930 @ 4.0GHz 24/7 - ThermalRight Ultra 120 Rev. C
6GB G.Skill 6-6-6-18 1T - eVGA GTX 580 SLI - GTX 460 Dedicated PhysX - Intel X25-M 80GB Boot Drive - 3x500GB WD Spinners
Sony & Asus Optical Drives - NZXT Sentry LX and Scythe Kaze Master Pro 5.25 Fan Controllers
Samsung 2494SW 24" Display - Logitech G15, G13, G500, Performance MX, and C310
Reply With Quote
  #19  
Old 02-05-2011
David's Avatar
David David is offline
kitchen robot
 
Join Date: Jun 2010
Location: UNITED STATES
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
David
Default

Quote:
Originally Posted by jjjc View Post
...Perhaps HWBOT should go the way of Futuremark, and require drivers to be approved before they can be used in rankings. Surely, drivers that allow certain settings to be forced off with no notification to the program wouldn't be approved and therefore this wouldn't be an issue...
Now that sounds like something to at least consider, does it not?
__________________
_____________
__________
Cooler Master HAF 932 - Corsair AX1200 - Asus Rampage III Formula - i7 930 @ 4.0GHz 24/7 - ThermalRight Ultra 120 Rev. C
6GB G.Skill 6-6-6-18 1T - eVGA GTX 580 SLI - GTX 460 Dedicated PhysX - Intel X25-M 80GB Boot Drive - 3x500GB WD Spinners
Sony & Asus Optical Drives - NZXT Sentry LX and Scythe Kaze Master Pro 5.25 Fan Controllers
Samsung 2494SW 24" Display - Logitech G15, G13, G500, Performance MX, and C310
Reply With Quote
  #20  
Old 02-05-2011
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,579
Thanks: 808
Thanked 1,136 Times in 557 Posts
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default

That is definitly a possibility. Actually, we have taken that into consideration more than once in the past. There are some problems with validating only specific drivers, though.

1) it's very time intensive
2) need for detailed driver qualifation guidelines
3) who are we to decide what drivers show true performance
4) it's annoying not to be able to use the latest drivers

One of the biggest issues perhaps is that it will annoy people. When new drivers are released, a lot of overclockers test to see if they get a small boost or not. Sometimes that boost is a few hundred points, sometimes it's around 1000 (depending on the optimalisations). Testing new drivers is part of the overclocking experience and I would argue that it's a part that should not be removed by prohibiting people to use non-approved drivers. One of the things I don't like about the Futuremark ORB is that it takes so very long to approve drivers. Sometimes we are forced to use out-dated drivers just because they haven't been validated yet.

Another problem is more ideological/philosophical. As drivers are the connection between theoretical performance (hardware-level) and the practical performance in games, who are we to decide which drivers show the 'true performance' of a graphics card. Software optimalisations of drivers are a part of the performance equation and disallowing manufacturers to optimize end-user experience through software performance updates is, in my humble opinion, not correct.

Last but not least, it wouldn't solve the issue of adjusting tesselation levels. Just like any other option you can find in the Control Panel, it can also be adjusted through the registry. This means that, even if the option would not be in the CP, end-users would still be able to (de-)activate it if they know the registry key.
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:41.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright HWBOT 2004 - 2015