Go Back   hwbot.org > News > HWBOT Frontpage News

HWBOT Frontpage News Official HWBOT articles and frontpage news

Reply
 
Thread Tools

(UPDATE 03/09) Windows 8 Results (Temporarily) Not Accepted At HWBOT - Benchmark Result Veracity Compromised

Author: Pieter-Jan Plaisier

As the result of weekend-time research, the HWBOT staff has decided to invalidate all benchmark records established with the Windows8 operating system. Due to severe validity problems with the Windows8 real time clock (“RTC”), benchmarks results achieved with Windows8 cannot be trusted. The main problem lies with the RTC being affected when over- or underclocking under the operating system. The operating system uses the RTC as reference clock, and benchmarks use it to reference (benchmark) time.

Update 03/09 – Revised Moderation Guidelines Concerning Windows 8, Futuremark Benchmarks Allowed (with new SI)

Following our recent news item regarding Futuremark’s solution for the Windows 8 RTC bug, we have updated our HWBOT moderation guidelines. They now include exceptions for Futuremark benchmarks with ORB link and System Info 4.20. The updated rules:

  • All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • Existing Windows 8 based benchmark submissions will be blocked if seemingly out of line or world record or top score.
  • Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above
  • Existing Windows 8 based benchmark submissions will not be blocked or removed if they were made on an AMD system (New ones will be blocked)

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove that result.

Exceptions to the Windows 8 rule:

  • non-overclockable notebooks may be used in combination with Windows 8
  • 3DMark and PCMark runs on Windows 8 if an ORB Link is included, and SystemInfo 4.20 (unofficial download) is used.

Update 24/08 – Revised Moderation Guidelines Concerning Windows 8

Following the findings of Ocaholic.ch (reference), the HWBOT moderating team has revised its rules regarding the usage of Windows 8 for benchmark submission at HWBOT. The major change is that the combination of AMD and Windows 8 is also not allowed, as in contrary to previous findings the platform is affected too. The revised rules can be found below.

  • 1) All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • 2) Existing Windows 8 based benchmark submissions will be blocked if seemingly out of line, world record, or top score.
  • 3) Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove that result.

Exceptions to the Windows 8 rule:

  • 1) non-overclockable notebooks may be used in combination with Windows 8

Update 21/08 – Updated Moderation Guidelines Concerning Windows 8

Following up on the recent developments on the topic of Windows 8 RTC, read the updated news item here, Head of Moderating Christian Ney has updated the moderation rules regarding the usage of Windows8. Here is a list of the updated rules:

  • 1) All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • 2) Existing Windows 8 based benchmark submissions will be blocked if
    • seemingly out of line
    • world record, or top score
  • 3) Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove the result.

Exceptions to the Windows 8 rule:

  • 1) AMD-based systems may be used in combination with Windows 8
  • 2) We are still debating on how to address non-overclockable notebooks

We also received a couple of emails debating the topic of which is to blame, Windows 8 or Intel CPUs. Based on the information we gathered from clock drift problem on various operating systems and hardware platforms, we are still of the opinion that Windows 8 is the root cause of the problem. Windows 8 seems to be using a different source to fetch “real time” on an Intel based system compared to an AMD based system. It’s still guess why and which at the moment, but it seems to be related to the power saving features integrated in Windows 8. With Windows 7 unaffected, and Windows 8 affected on all Intel platforms (including old generations), the root of the problem lies with Windows 8. Having the Intel system unaffected when coming from and AMD Windows 8 installation indicates that the correct timers are available on Intel too, but unused by Windows 8.

Thank you for your understanding.

The HWBOT Staff.

Update 20/08 – Windows 8.1 Affected, AMD Problem-Free(?)

We have continued our testing on more platforms. Windows 8.1 is, just like its older brother, affected by at-runtime base clock frequency adjustments. Intel platforms back to LGA775 are affected. From the looks of it, AMD systems are unaffected. In fact, the clock drifting on Intel based systems seem to resolve itself when switching from an Intel based system, to AMD, and back to Intel. Later today Christian Ney, Head Moderator, will publish an update to the rules at HWBOT concerning the usage of Windows 8. Stay tuned!

The following combinations of platforms have been tested

Operating SystemPlatformRTC Bias / Clock Drifting
Windows 8.1HaswellYES
Windows 8HaswellYES
Windows 7Haswellno
Windows 8Ivy BridgeYES
Windows 8Sandy BridgeYES
Windows 8GulftownYES
Windows 8WolfdaleYES
Windows 8Trinityno
Windows 8Piledriverno

How does it work and how to spot it?

  • Boot in OS at 130MHz BCLK and 32x CPU Ratio
  • Reduce BCLK frequency to 122MHz at runtime (in OS). Set CPU ratio at 34x to maintain the same frequency of 4160 MHz
  • Check benchmark before and after the downclock

You can also easily spot the RTC problem using Windows Clock (see video below)

More information


Background Information – Remember Heaven?

At the moment of writing, we do not have the full technical what’s and how’s figured out. Since this problem affects everyone who is passionate about overclocking, it is important to provide an explanation. It is far from the complete story, but it should be enough for you to understand why we have decided to ban Windows 8 from HWBOT.

Do you remember the history of Unigine Heaven at HWBOT? About three years ago, we launched a wrapper for the back-then brand new DX11 benchmark software Unigine Heaven. The wrapper featured an easy and secure way to submit benchmark scores to HWBOT via data files. On November 2, 2010 we posted a response to an on-going discussion about downclocking in Windows affecting the benchmark score. To make a very long story short, by downclocking in Windows the Heaven benchmark time runs slower than it really is. One second in the Heaven benchmark is equal to 1,x second of real time. Because there is more time within a second, the system can render more frames. The benchmark itself is unaware of all this – for the benchmark, one second is still exactly one second. In the end, the system renders a higher amount of frames in a longer timeframe. In the result calculation for frames per second (“FPS”), the “frames” have increased but the “second” remains the same. You get a higher score.

Later that week, we published a V1.03b version of the wrapper, which fixed the downclock issue. Without going too much into detail, the wrapper uses a second source to verify the benchmark duration and takes the relative measured time difference into account when calculating the final score. Problem solved.

Windows8 – “Support for all devices”

As you know, Microsoft is trying to come up with a unified operating system and user interface for a wide range of devices, including tablets, smartphones, Xbox One, and the desktop PC. Building this unified platform is not easy. It is not just a matter of creating an interface that can be used with a multitude of input devices (finger, mouse, controller), but it also needs to support as many devices as possible. Getting the software to run out of the box on as much hardware as possible is the challenge they are facing.

Of the many aspects to fine-tune, one feature in particular is causing Windows8 to be practically useless for (competitive) overclocking: the RTC. Quoting Wikipedia, “A real-time clock (RTC) is a computer clock (most often in the form of an integrated circuit) that keeps track of the current time. Although the term often refers to the devices in personal computers, servers and embedded systems, RTCs are present in almost any electronic device which needs to keep accurate time.” Sparing you the details of development process, compared to Windows7 and previous versions, Microsoft made changes to how it measures time to be compatible with embedded or low cost PCs that do not have a fixed RTC clock. After all, having a fixed RTC clock adds cost to a platform.

Your PC system uses the RTC for many things. For example, it ensures the Windows Time on your machine is accurate. For most benchmark applications, the RTC is used as reference clock when executing the benchmark code. By synchronizing with the RTC, the benchmark knows exactly how much time has passed, and takes that value into account when calculating the performance of your system.

It’s all relative (part2)

In a previous editorial (“Critical thinking – Should Maximus V Extreme be …”), we already touched the topic of the relativity of time in our overclocking universe. “The concept of ?time? on a PC configuration is, if not synced via network or internet, an arbitrarily defined constant designed to ensure that the configuration is running in sync with the real world. In other words: hardware and software engineers ensure that ?one second? on your PC equals ?one second? in real time. One of the reasons why it?s so important to have the PC?s timer line up with the real world time is to ensure that your PC can produce accurate measurements and predictions.” The points we brought up in that editorial are relevant again. To ensure that the arbitrarily defined constant of ‘time’ is the same on everyone’s benchmark system, we rely on the OS and hardware. This worked quite well, until Windows8 came around.

The problem builds on the problems we faced with Heaven. When downclocking the system under Windows8, the Windows RTC is affected as well. The biggest difference between Windows7 and Windows8 is that now all benchmarks (no exception) are affected.

Examples: Benchmarks and Windows Time.

Let us make this more practical. On our Haswell test system we downclocked the BCLK frequency by about 6% from 130 MHz to 122MHz. Using a CPU ratio of respectively 32x and 34x, the resulting CPU frequency remains 4160MHz. Then we ran comparison benchmarks. Here are a couple examples:

BenchmarkResult at defaultResult after underclockDifference
Wprime32M5.6055.235+ 7.07%
Prime9513.62213.423+ 1.48%
Heaven DX111417.9821522.567+ 7.38%
SuperPI 32M7min 337min 10.799+ 5.10%
PiFast15.7814.88+ 6.05%
Aquamark3351,160371,902+ 5.91%
3DMark Ice Storm141,977150,526+ 6.02%
3DMark Cloud Gate2046221676+ 5.93%
3DMark Fire Strike47275012+ 6.03%

As you can see, even the latest 3DMark is affected by this. In Windows7, most benchmarks are unaffected by this kind of “overclocking”.

We can also demonstrate the Windows8 RTC problem with the Windows Time. Have a look at the videos below. In one of the videos, we underclocked the base clock frequency by 6%. After 5 minutes, Windows Time was already 18(!) seconds behind real time. When overclocking the base clock frequency, we can see the opposite effect. We overclock by roughly 4%, and after two minutes, Windows Time is 3 seconds ahead of real time.

Conclusive lines.

First, we would like to apologise if this editorial seems put together in a hurry. If it seems so, it is because it is. As we hope you gather from the data, we provided in the article, it is not possible for HWBOT to accept any benchmark results or records achieved using Windows8. Simply no benchmark – not even 3DMark – is unaffected by Microsoft’s RTC design decisions. As a result, it is impossible to verify the veracity of a system performance indicative in Windows8. The resulting score of any benchmark is relative to the RTC bias of that system.

In order to compare we need the RTC bias to be equal for all systems. With Windows8, we can only use question marks.

Practically speaking, it is not allowed to:

  • exploit the Windows8 RTC for benchmark purposes
  • submit any benchmark results with RTC bias

The HWBOT staff will:

  • Block any seemingly out-of-line Windows8-based benchmark results
  • Block any Windows8-based benchmark record, even if the score seems in line with the expectations

We hope that everyone is willing to adhere to these new guidelines. The HWBOT Staff is as passionate as you are about overclocking, and this kind of external problem is affecting our hobby in a way that makes we cannot easily resolve. We are friends amongst each other, and respect the effort your fellow overclockers put into overclocking results.

Thank you for your understanding.

The HWBOT Staff.


Aug 18, 2013 - News, benchmarks, Editorials, Articles - HWBOT
  #1  
Old 08-18-2013, 15:13
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Waregem, Belgium
Posts: 19,228
Send a message via MSN to Massman
Default Windows 8 Benchmark Records No Longer Accepted At HWBOT - Benchmark Result Veracity Compromised

Since this is all very new to the staff, we are still looking into the consequences and possible solutions for this problem.

//edit: seems like Wordpress is broken due to traffic. I'll cross-post the article below.

How to recreate this issue

- Boot in OS at 130MHz BCLK
- Reduce BCLK frequency to 122MHz at runtime (in OS)
- Check benchmark before and after the downclock

Affected

- Windows 8 + Haswell = YES
- Windows 7 + Haswell = NO
- Windows 8 + Trinity = NO
- Windows 8 + LGA775 = YES
- Windows 8 + AM3 = NO
- Windows 8 + Ivy Bridge = YES
- Windows 8 + Sandy Bridge = YES

Quote:


As the result of weekend-time research, the HWBOT staff has decided to invalidate all benchmark records established with the Windows8 operating system. Due to severe validity problems with the Windows8 real time clock ("RTC"), benchmarks results achieved with Windows8 cannot be trusted. The main problem lies with the RTC being affected when over- or underclocking under the operating system. The operating system uses the RTC as reference clock, and benchmarks use it to reference (benchmark) time.

Background Information - Remember Heaven?

At the moment of writing, we do not have the full technical what's and how's figured out. Since this problem affects everyone who is passionate about overclocking, it is important to provide an explanation. It is far from the complete story, but it should be enough for you to understand why we have decided to ban Windows 8 from HWBOT.

Do you remember the history of Unigine Heaven at HWBOT? About three years ago, we launched a wrapper for the back-then brand new DX11 benchmark software Unigine Heaven. The wrapper featured an easy and secure way to submit benchmark scores to HWBOT via data files. On November 2, 2010 we posted a response to an on-going discussion about downclocking in Windows affecting the benchmark score. To make a very long story short, by downclocking in Windows the Heaven benchmark time runs slower than it really is. One second in the Heaven benchmark is equal to 1,x second of real time. Because there is more time within a second, the system can render more frames. The benchmark itself is unaware of all this - for the benchmark, one second is still exactly one second. In the end, the system renders a higher amount of frames in a longer timeframe. In the result calculation for frames per second ("FPS"), the "frames" have increased but the "second" remains the same. You get a higher score.

Later that week, we published a V1.03b version of the wrapper, which fixed the downclock issue. Without going too much into detail, the wrapper uses a second source to verify the benchmark duration and takes the relative measured time difference into account when calculating the final score. Problem solved.

Windows8 - "Support for all devices"

As you know, Microsoft is trying to come up with a unified operating system and user interface for a wide range of devices, including tablets, smartphones, Xbox One, and the desktop PC. Building this unified platform is not easy. It is not just a matter of creating an interface that can be used with a multitude of input devices (finger, mouse, controller), but it also needs to support as many devices as possible. Getting the software to run out of the box on as much hardware as possible is the challenge they are facing.

Of the many aspects to fine-tune, one feature in particular is causing Windows8 to be practically useless for (competitive) overclocking: the RTC. Quoting Wikipedia, "A real-time clock (RTC) is a computer clock (most often in the form of an integrated circuit) that keeps track of the current time. Although the term often refers to the devices in personal computers, servers and embedded systems, RTCs are present in almost any electronic device which needs to keep accurate time." Sparing you the details of development process, compared to Windows7 and previous versions, Microsoft made changes to how it measures time to be compatible with embedded or low cost PCs that do not have a fixed RTC clock. After all, having a fixed RTC clock adds cost to a platform.

Your PC system uses the RTC for many things. For example, it ensures the Windows Time on your machine is accurate. For most benchmark applications, the RTC is used as reference clock when executing the benchmark code. By synchronizing with the RTC, the benchmark knows exactly how much time has passed, and takes that value into account when calculating the performance of your system.

It's all relative (part2)

In a previous editorial ("Critical thinking - Should Maximus V Extreme be ..."), we already touched the topic of the relativity of time in our overclocking universe. "The concept of ‘time’ on a PC configuration is, if not synced via network or internet, an arbitrarily defined constant designed to ensure that the configuration is running in sync with the real world. In other words: hardware and software engineers ensure that ‘one second’ on your PC equals ‘one second’ in real time. One of the reasons why it’s so important to have the PC’s timer line up with the real world time is to ensure that your PC can produce accurate measurements and predictions." The points we brought up in that editorial are relevant again. To ensure that the arbitrarily defined constant of 'time' is the same on everyone's benchmark system, we rely on the OS and hardware. This worked quite well, until Windows8 came around.

The problem builds on the problems we faced with Heaven. When downclocking the system under Windows8, the Windows RTC is affected as well. The biggest difference between Windows7 and Windows8 is that now all benchmarks (no exception) are affected.

Examples: Benchmarks and Windows Time.

Let us make this more practical. On our Haswell test system we downclocked the BCLK frequency by about 6% from 130 MHz to 122MHz. Using a CPU ratio of respectively 32x and 34x, the resulting CPU frequency remains 4160MHz. Then we ran comparison benchmarks. Here are a couple examples:



http://news.hwbot.org/Articles/9824_...lowed/7.14.jpg
http://news.hwbot.org/Articles/9824_...llowed/122.jpg
http://news.hwbot.org/Articles/9824_...llowed/130.jpg

http://news.hwbot.org/Articles/9824_...llowed/cg1.jpg
http://news.hwbot.org/Articles/9824_...llowed/cg2.jpg
http://news.hwbot.org/Articles/9824_...llowed/fs1.jpg
http://news.hwbot.org/Articles/9824_...llowed/fs2.jpg

http://news.hwbot.org/Articles/9824_...llowed/141.jpg
http://news.hwbot.org/Articles/9824_...llowed/141.jpg
http://news.hwbot.org/Articles/9824_...llowed/150.jpg

As you can see, even the latest 3DMark is affected by this. In Windows7, most benchmarks are unaffected by this kind of "overclocking".

We can also demonstrate the Windows8 RTC problem with the Windows Time. Have a look at the videos below. In one of the videos, we underclocked the base clock frequency by 6%. After 5 minutes, Windows Time was already 18(!) seconds behind real time. When overclocking the base clock frequency, we can see the opposite effect. We overclock by roughly 4%, and after two minutes, Windows Time is 3 seconds ahead of real time.

http://www.youtube.com/embed/brkDKgvXKko
http://www.youtube.com/embed/XUi0zpRrtMM

Conclusive lines.

First, we would like to apologise if this editorial seems put together in a hurry. If it seems so, it is because it is. As we hope you gather from the data, we provided in the article, it is not possible for HWBOT to accept any benchmark results or records achieved using Windows8. Simply no benchmark - not even 3DMark - is unaffected by Microsoft's RTC design decisions. As a result, it is impossible to verify the veracity of a system performance indicative in Windows8. The resulting score of any benchmark is relative to the RTC bias of that system.

In order to compare we need the RTC bias to be equal for all systems. With Windows8, we can only use question marks.

Practically speaking, it is not allowed to:

- exploit the Windows8 RTC for benchmark purposes
- submit any benchmark results with RTC bias

The HWBOT staff will:

- Block any seemingly out-of-line Windows8-based benchmark results
- Block any Windows8-based benchmark record, even if the score seems in line with the expectations

We hope that everyone is willing to adhere to these new guidelines. The HWBOT Staff is as passionate as you are about overclocking, and this kind of external problem is affecting our hobby in a way that makes we cannot easily resolve. We are friends amongst each other, and respect the effort your fellow overclockers put into overclocking results.

Thank you for your understanding.

The HWBOT Staff.
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.

Last edited by Massman; 08-19-2013 at 23:43.
Reply With Quote
  #2  
Old 08-18-2013, 15:28
Hades Hades is offline
construction bot
 
Join Date: Nov 2008
Location: L3 Cache
Posts: 76
Default

What if my Windows 8 is always connected to the Internet? What if I just overclock using multipliers, not modifying BCLK neither straps? Is there a way to check the RTC without the need of blocking all of the Win8 results?
I think it's a too drastic decision to block all of the results made with W8, it means that Windows 7 will be the last supported OS for Overclock, since Microsoft's path will keep working on modified versions of Windows 8, not on 7's ones. There should be a way to check if a score is "tricked" or not, maybe with a video, or something like that, like "If you wanna use Windows 8, don't use BCLK clocking/downclocking and post a video proving that you didn't modified clocks before taking the screenshot", it's a viable compromise, I think. Besides, top league clockers don't use W8 for WRs, they use tweaked versions of XP, 7 and S2003, since 8 is not really a bench system...
__________________
Reply With Quote
  #3  
Old 08-18-2013, 15:35
Massman's Avatar
Massman Massman is offline
In the distance
 
Join Date: Jul 2006
Location: Waregem, Belgium
Posts: 19,228
Send a message via MSN to Massman
Default

To clarify, we don't want to take the position of "everyone is guilty, until proven otherwise". The problems with the Windows8 RTC are of such nature that RTC bias (incorrect clock) can happen without the user knowing. The matter of the fact remains that any score with a biased timer is an false result.

We are looking into ways to get Win8 back on the approved list. But given even Futuremark's 3DMark is affected, I think this will be quite a complicated issue. Thank you for your suggestions Ciruzzo!
__________________
Where courage, motivation and ignorance meet, a persistent idiot awakens.
Reply With Quote
  #4  
Old 08-18-2013, 15:37
Mr.Scott's Avatar
Mr.Scott Mr.Scott is offline
Fully capable of playing games
 
Join Date: Jan 2009
Location: UNITED STATES
Posts: 1,248
Default

Nice work.

Thanks for the info.
__________________

"If you make people think they're thinking, they'll love you; but if you really make them think, they'll hate you."
Reply With Quote
  #5  
Old 08-18-2013, 15:40
borandi's Avatar
borandi borandi is offline
robo cop
 
Join Date: Feb 2009
Location: UNITED KINGDOM
Posts: 479
Default

This obviously makes an issue for regular reviewers benchmarking in Win8 + pre-overclocked systems.
__________________
Bro, do you even bench?



Quote:
"who cares about Christian Ney, this is overclocking" - John Lam
ReviewBros
Reply With Quote
  #6  
Old 08-18-2013, 15:47
Hades Hades is offline
construction bot
 
Join Date: Nov 2008
Location: L3 Cache
Posts: 76
Default

Quote:
Originally Posted by Massman View Post
To clarify, we don't want to take the position of "everyone is guilty, until proven otherwise". The problems with the Windows8 RTC are of such nature that RTC bias (incorrect clock) can happen without the user knowing. The matter of the fact remains that any score with a biased timer is an false result.

We are looking into ways to get Win8 back on the approved list. But given even Futuremark's 3DMark is affected, I think this will be quite a complicated issue. Thank you for your suggestions Ciruzzo!
You're welcome Pieter!
Quote:
Originally Posted by borandi View Post
This obviously makes an issue for regular reviewers benchmarking in Win8 + pre-overclocked systems.
That's exactly what's my point, since I'm one of them :\
__________________
Reply With Quote
  #7  
Old 08-18-2013, 16:00
redmax redmax is offline
grunt bot
 
Join Date: Mar 2009
Location: UNITED STATES
Posts: 106
Default

good work! Pretty sure this will cause issues throughout the industry, hardware review and otherwise.
__________________
Reply With Quote
  #8  
Old 08-18-2013, 16:08
Christian Ney's Avatar
Christian Ney Christian Ney is offline
I am the one that blocked your score!
 
Join Date: Apr 2010
Location: Geneva, Switzerland
Posts: 6,508
Send a message via Skype™ to Christian Ney
Default

Quote:
Originally Posted by Saito View Post
Can't we patch windows 8 maybe? and remove RTC from win 8?
Tell Microsoft
__________________
Memory reviews by TaPaKaH
Reply With Quote
  #9  
Old 08-18-2013, 16:09
Saito Saito is offline
maintenance bot
 
Join Date: Dec 2010
Location: SOUTH AFRICA
Posts: 16
Send a message via Skype™ to Saito
Default

Can't we patch windows 8 maybe? and remove RTC from win 8?
__________________
Reply With Quote
  #10  
Old 08-18-2013, 17:06
Fastvedub1 Fastvedub1 is offline
kitchen robot
 
Join Date: Jul 2012
Location: United States
Posts: 1
Default

Wow this is a bummer,I just started doing some benches with windows 8,As i have switched over to windows 8 on my machines....
__________________
Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:05.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright COLARDYN IT GCV 2004 - 2013