Go Back   HWBOT forum > News > HWBOT Frontpage News

HWBOT Frontpage News Official HWBOT articles and frontpage news

Reply
 
Thread Tools Display Modes
  #1  
Old 08-18-2013
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,579
Thanks: 808
Thanked 1,136 Times in 557 Posts
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default Windows 8 Benchmark Records No Longer Accepted At HWBOT - Benchmark Result Veracity Compromised

Since this is all very new to the staff, we are still looking into the consequences and possible solutions for this problem.

//edit: seems like Wordpress is broken due to traffic. I'll cross-post the article below.

How to recreate this issue

- Boot in OS at 130MHz BCLK
- Reduce BCLK frequency to 122MHz at runtime (in OS)
- Check benchmark before and after the downclock

Affected

- Windows 8 + Haswell = YES
- Windows 7 + Haswell = NO
- Windows 8 + Trinity = NO
- Windows 8 + LGA775 = YES
- Windows 8 + AM3 = NO
- Windows 8 + Ivy Bridge = YES
- Windows 8 + Sandy Bridge = YES

Quote:


As the result of weekend-time research, the HWBOT staff has decided to invalidate all benchmark records established with the Windows8 operating system. Due to severe validity problems with the Windows8 real time clock ("RTC"), benchmarks results achieved with Windows8 cannot be trusted. The main problem lies with the RTC being affected when over- or underclocking under the operating system. The operating system uses the RTC as reference clock, and benchmarks use it to reference (benchmark) time.

Background Information - Remember Heaven?

At the moment of writing, we do not have the full technical what's and how's figured out. Since this problem affects everyone who is passionate about overclocking, it is important to provide an explanation. It is far from the complete story, but it should be enough for you to understand why we have decided to ban Windows 8 from HWBOT.

Do you remember the history of Unigine Heaven at HWBOT? About three years ago, we launched a wrapper for the back-then brand new DX11 benchmark software Unigine Heaven. The wrapper featured an easy and secure way to submit benchmark scores to HWBOT via data files. On November 2, 2010 we posted a response to an on-going discussion about downclocking in Windows affecting the benchmark score. To make a very long story short, by downclocking in Windows the Heaven benchmark time runs slower than it really is. One second in the Heaven benchmark is equal to 1,x second of real time. Because there is more time within a second, the system can render more frames. The benchmark itself is unaware of all this - for the benchmark, one second is still exactly one second. In the end, the system renders a higher amount of frames in a longer timeframe. In the result calculation for frames per second ("FPS"), the "frames" have increased but the "second" remains the same. You get a higher score.

Later that week, we published a V1.03b version of the wrapper, which fixed the downclock issue. Without going too much into detail, the wrapper uses a second source to verify the benchmark duration and takes the relative measured time difference into account when calculating the final score. Problem solved.

Windows8 - "Support for all devices"

As you know, Microsoft is trying to come up with a unified operating system and user interface for a wide range of devices, including tablets, smartphones, Xbox One, and the desktop PC. Building this unified platform is not easy. It is not just a matter of creating an interface that can be used with a multitude of input devices (finger, mouse, controller), but it also needs to support as many devices as possible. Getting the software to run out of the box on as much hardware as possible is the challenge they are facing.

Of the many aspects to fine-tune, one feature in particular is causing Windows8 to be practically useless for (competitive) overclocking: the RTC. Quoting Wikipedia, "A real-time clock (RTC) is a computer clock (most often in the form of an integrated circuit) that keeps track of the current time. Although the term often refers to the devices in personal computers, servers and embedded systems, RTCs are present in almost any electronic device which needs to keep accurate time." Sparing you the details of development process, compared to Windows7 and previous versions, Microsoft made changes to how it measures time to be compatible with embedded or low cost PCs that do not have a fixed RTC clock. After all, having a fixed RTC clock adds cost to a platform.

Your PC system uses the RTC for many things. For example, it ensures the Windows Time on your machine is accurate. For most benchmark applications, the RTC is used as reference clock when executing the benchmark code. By synchronizing with the RTC, the benchmark knows exactly how much time has passed, and takes that value into account when calculating the performance of your system.

It's all relative (part2)

In a previous editorial ("Critical thinking - Should Maximus V Extreme be ..."), we already touched the topic of the relativity of time in our overclocking universe. "The concept of time on a PC configuration is, if not synced via network or internet, an arbitrarily defined constant designed to ensure that the configuration is running in sync with the real world. In other words: hardware and software engineers ensure that one second on your PC equals one second in real time. One of the reasons why its so important to have the PCs timer line up with the real world time is to ensure that your PC can produce accurate measurements and predictions." The points we brought up in that editorial are relevant again. To ensure that the arbitrarily defined constant of 'time' is the same on everyone's benchmark system, we rely on the OS and hardware. This worked quite well, until Windows8 came around.

The problem builds on the problems we faced with Heaven. When downclocking the system under Windows8, the Windows RTC is affected as well. The biggest difference between Windows7 and Windows8 is that now all benchmarks (no exception) are affected.

Examples: Benchmarks and Windows Time.

Let us make this more practical. On our Haswell test system we downclocked the BCLK frequency by about 6% from 130 MHz to 122MHz. Using a CPU ratio of respectively 32x and 34x, the resulting CPU frequency remains 4160MHz. Then we ran comparison benchmarks. Here are a couple examples:



http://news.hwbot.org/Articles/9824_...lowed/7.14.jpg
http://news.hwbot.org/Articles/9824_...llowed/122.jpg
http://news.hwbot.org/Articles/9824_...llowed/130.jpg

http://news.hwbot.org/Articles/9824_...llowed/cg1.jpg
http://news.hwbot.org/Articles/9824_...llowed/cg2.jpg
http://news.hwbot.org/Articles/9824_...llowed/fs1.jpg
http://news.hwbot.org/Articles/9824_...llowed/fs2.jpg

http://news.hwbot.org/Articles/9824_...llowed/141.jpg
http://news.hwbot.org/Articles/9824_...llowed/141.jpg
http://news.hwbot.org/Articles/9824_...llowed/150.jpg

As you can see, even the latest 3DMark is affected by this. In Windows7, most benchmarks are unaffected by this kind of "overclocking".

We can also demonstrate the Windows8 RTC problem with the Windows Time. Have a look at the videos below. In one of the videos, we underclocked the base clock frequency by 6%. After 5 minutes, Windows Time was already 18(!) seconds behind real time. When overclocking the base clock frequency, we can see the opposite effect. We overclock by roughly 4%, and after two minutes, Windows Time is 3 seconds ahead of real time.

http://www.youtube.com/embed/brkDKgvXKko
http://www.youtube.com/embed/XUi0zpRrtMM

Conclusive lines.

First, we would like to apologise if this editorial seems put together in a hurry. If it seems so, it is because it is. As we hope you gather from the data, we provided in the article, it is not possible for HWBOT to accept any benchmark results or records achieved using Windows8. Simply no benchmark - not even 3DMark - is unaffected by Microsoft's RTC design decisions. As a result, it is impossible to verify the veracity of a system performance indicative in Windows8. The resulting score of any benchmark is relative to the RTC bias of that system.

In order to compare we need the RTC bias to be equal for all systems. With Windows8, we can only use question marks.

Practically speaking, it is not allowed to:

- exploit the Windows8 RTC for benchmark purposes
- submit any benchmark results with RTC bias

The HWBOT staff will:

- Block any seemingly out-of-line Windows8-based benchmark results
- Block any Windows8-based benchmark record, even if the score seems in line with the expectations

We hope that everyone is willing to adhere to these new guidelines. The HWBOT Staff is as passionate as you are about overclocking, and this kind of external problem is affecting our hobby in a way that makes we cannot easily resolve. We are friends amongst each other, and respect the effort your fellow overclockers put into overclocking results.

Thank you for your understanding.

The HWBOT Staff.
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.

Last edited by Massman; 08-19-2013 at 23:43.
Reply With Quote
  #2  
Old 08-18-2013
Atlas Rush's Avatar
Atlas Rush Atlas Rush is offline
construction bot
 
Join Date: Nov 2008
Location: Italy, L1 Cache
Posts: 81
Thanks: 1
Thanked 0 Times in 0 Posts
Atlas Rush
Send a message via Skype™ to Atlas Rush
Default

What if my Windows 8 is always connected to the Internet? What if I just overclock using multipliers, not modifying BCLK neither straps? Is there a way to check the RTC without the need of blocking all of the Win8 results?
I think it's a too drastic decision to block all of the results made with W8, it means that Windows 7 will be the last supported OS for Overclock, since Microsoft's path will keep working on modified versions of Windows 8, not on 7's ones. There should be a way to check if a score is "tricked" or not, maybe with a video, or something like that, like "If you wanna use Windows 8, don't use BCLK clocking/downclocking and post a video proving that you didn't modified clocks before taking the screenshot", it's a viable compromise, I think. Besides, top league clockers don't use W8 for WRs, they use tweaked versions of XP, 7 and S2003, since 8 is not really a bench system...
__________________
Reply With Quote
  #3  
Old 08-18-2013
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Taipei
Posts: 22,579
Thanks: 808
Thanked 1,136 Times in 557 Posts
Massman has a spectacular aura aboutMassman has a spectacular aura aboutMassman has a spectacular aura about
Send a message via MSN to Massman
Default

To clarify, we don't want to take the position of "everyone is guilty, until proven otherwise". The problems with the Windows8 RTC are of such nature that RTC bias (incorrect clock) can happen without the user knowing. The matter of the fact remains that any score with a biased timer is an false result.

We are looking into ways to get Win8 back on the approved list. But given even Futuremark's 3DMark is affected, I think this will be quite a complicated issue. Thank you for your suggestions Ciruzzo!
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

For all HWBOT community related questions, contact Christian Ney or Websmile. For any other questions, contact me at pieter@hwbot.org.
Reply With Quote
  #4  
Old 08-18-2013
Mr.Scott's Avatar
Mr.Scott Mr.Scott is offline
Fully capable of playing games
 
Join Date: Jan 2009
Location: UNITED STATES
Posts: 2,197
Thanks: 847
Thanked 361 Times in 185 Posts
Mr.Scott is on a distinguished road
Default

Nice work.

Thanks for the info.
__________________

Warp9-systems, the Antichrist of competitive benchmarking.
Loved by thousands, hated by millions.
Reply With Quote
  #5  
Old 08-18-2013
borandi's Avatar
borandi borandi is offline
robo cop
 
Join Date: Feb 2009
Location: UNITED KINGDOM
Posts: 477
Thanks: 0
Thanked 0 Times in 0 Posts
borandi
Default

This obviously makes an issue for regular reviewers benchmarking in Win8 + pre-overclocked systems.
__________________

Quote:
"who cares about Christian Ney, this is overclocking" - John Lam
Reply With Quote
  #6  
Old 08-18-2013
Atlas Rush's Avatar
Atlas Rush Atlas Rush is offline
construction bot
 
Join Date: Nov 2008
Location: Italy, L1 Cache
Posts: 81
Thanks: 1
Thanked 0 Times in 0 Posts
Atlas Rush
Send a message via Skype™ to Atlas Rush
Default

Quote:
Originally Posted by Massman View Post
To clarify, we don't want to take the position of "everyone is guilty, until proven otherwise". The problems with the Windows8 RTC are of such nature that RTC bias (incorrect clock) can happen without the user knowing. The matter of the fact remains that any score with a biased timer is an false result.

We are looking into ways to get Win8 back on the approved list. But given even Futuremark's 3DMark is affected, I think this will be quite a complicated issue. Thank you for your suggestions Ciruzzo!
You're welcome Pieter!
Quote:
Originally Posted by borandi View Post
This obviously makes an issue for regular reviewers benchmarking in Win8 + pre-overclocked systems.
That's exactly what's my point, since I'm one of them :\
__________________
Reply With Quote
  #7  
Old 08-18-2013
redmax redmax is offline
grunt bot
 
Join Date: Mar 2009
Location: UNITED STATES
Posts: 116
Thanks: 1
Thanked 0 Times in 0 Posts
redmax
Default

good work! Pretty sure this will cause issues throughout the industry, hardware review and otherwise.
__________________
Reply With Quote
  #8  
Old 08-18-2013
Christian Ney's Avatar
Christian Ney Christian Ney is offline
Single Point of Contact
 
Join Date: Apr 2010
Location: Geneva, Switzerland
Posts: 7,490
Thanks: 3
Thanked 369 Times in 168 Posts
Christian Ney is on a distinguished road
Default

Quote:
Originally Posted by Saito View Post
Can't we patch windows 8 maybe? and remove RTC from win 8?
Tell Microsoft
Reply With Quote
  #9  
Old 08-18-2013
Saito Saito is offline
maintenance bot
 
Join Date: Dec 2010
Location: SOUTH AFRICA
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Saito
Send a message via Skype™ to Saito
Default

Can't we patch windows 8 maybe? and remove RTC from win 8?
__________________
Reply With Quote
  #10  
Old 08-18-2013
Fastvedub1 Fastvedub1 is offline
kitchen robot
 
Join Date: Jul 2012
Location: United States
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Fastvedub1
Default

Wow this is a bummer,I just started doing some benches with windows 8,As i have switched over to windows 8 on my machines....
__________________
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:41.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright HWBOT 2004 - 2015