Critical thinking - Should Maximus V Extreme be banned from competitive overclocking?
Introduction
The world of overclocking and the community it’s founded on is in essence not so much different from any other field of interest or social group. We have our own language, our own set of principles, our own set of shared motives and goals and (virtual) places we visit. Just like any other social group, once in a while we are forced to face an existential crisis; a problem that pushes us to think about what overclocking and benchmarking is, or what it is supposed to be. Over the years, we’ve had lengthy (and interesting) discussions regarding the usage of extreme cooling, a debate on whether or not graphical artifacts should be tolerated, if adjusting LOD values should be considered legit and many many more. In a recent conversation, overclocking legend Sampsa Kurri brought up an interesting subject that could very well bring us back into the position of facing another existential crisis.
In the video he linked a new feature of the infamous ROG OC Key is demoed. Let’s have a look.
As you can see in the video, this new feature of OC Key (currently without a marketing name, so let’s call it ROG Pause) allows the user to completely freeze the entire system at any point during the benchmark, go for a walk or (more likely) cool down the system and continue the benchmark when ready. As far as we can judge based on the very limited documentation available on the web or in the video, it seems that the ROG Pause functionality is purely hardware-based and stops the system on a hardware level. If correct, it’s important for two reasons:
- It means that this new function is not affecting the benchmark’s perception of time directly as, for example, a speedhack software application would. It doesn’t speed up or slow down time, it just halts everything.
- Unlike what MSI did upon themselves with a BIOS feature called “lab burst mode” on a couple of Llano mainboards, the perception of time is not altered either. For those who don’t know, the Lab Burst Mode essentially was a guarantee for successful overclocking past 133FSB. As power users (such as The Stilt) later pointed out, this featured altered the PLL (clock generator) mode, which caused the reported frequency to be higher than it really was. The frequency was not really 133+ MHz, it was much lower, but because it’s a hardware-level feature none of the software could detect this. The benchmark results with Lab Burst Mode are effectively hardware-based speedhacks.
Even though ROG Pause does not cheat the timer, on a fundamental level it’s affecting the benchmark’s legitimacy in quite a similar way. Not by affecting the timer of the hardware, but … by affecting real time!
That last sentence probably made you think I’m sort of a sci-fi lunatic. But bear with me for a few more lines as I’m going to explain why ROG Pause affects real time. But first, let’s think about the relativity of the PC’s timer and benchmark results.
It’s all relative …
‘Time’, so beautifully defined at Wikipedia as the continuing sequence of events occurring in apparently irreversible succession from the past through the present to the future, and a measure of the durations and frequencies of events and the intervals between them, is not an easy to grasp as a concept. We’re all able to work with it and somewhat understand the principles and the practical effects. Essential to understanding the implications of ROG Pause is understanding that the concept of ‘time’ on a PC configuration is, if not synced via network or internet, an arbitrarily defined constant designed to ensure that the configuration is running in sync with the real world. In other words: hardware and software engineers ensure that ‘one second’ on your PC equals ‘one second’ in real time. One of the reasons why it’s so important to have the PC’s timer line up with the real world time is to ensure that your PC can produce accurate measurements and predictions. After all, how would your PC be able to calculate the travel time between London and Paris if it does not have the correct definition of ’second’?
Okay, let’s make this a little more OC relevant. One of the most important measurement tools used to determine the performance of your system is FPS, Frames per second. As you know, the significance of any FPS indication is relative to two variables: the frame (“How many frames were calculated?”) and the time (“How much time has passed?”). In most 3DMarks the time is a constant, which means that for every benchmark the only variable is the amount of rendered frames. The decision to have the benchmark time constant is not a given, but merely an arbitrary decision made by the developers at Futuremark. In fact, when designing a benchmark that uses FPS as measuring tool, you need define both “frame” and “second”. We have to ask the questions “What constitutes a frame?” and “What is a second?”. Adjusting the definition of either parameter affects the significance of the performance measurement.
And this is something we, the overclocking community, already know!
- “What constitutes a frame?”, see “Lucid Virtu MVP: Revolution in benching?“.
- “What is a second?”, see “speedhack software” or see “Unigine Heaven Downclock Bug“
A small note to end this paragraph with: adjusting the definition of either variable has been judged illegal by the majority of the overclocking community.
Measured time versus real time
So, now that we’ve established the importance of defining all variables as well as the significance of having the PC’s time line up with the real world time, we can have a closer look at the implications of the ROG Pause software. As we’ve figured out earlier in the write-up, ROG Pause does not seem to change the system’s perception of time. In other words, every second on the M5E equals a ‘real’ second. So, technically, that would mean that any performance measurement is a reliable measurement. Or does it?
The “Bolt Marathon” Paradox
Shock in the world of athletics! Usain Bolt, 100M Dash WR holder, has challenged Patrick Makau, marathon WR holder, for a marathon duel, claiming he can easily beat Makau’s best time and win the race. Makau, fit and healthy as he is, accepts the challenge and agrees to run the marathon first. He finishes the race in a time that ties with the current world record; he needed 2 h 03 min 38 sec, averaging 20.5 km/h, to finish the 42,195 km. Impressive!
Usain Bolt, sneaky as he is, will be using a different strategy to run the 42,195 km as fast as possible. Instead of performing the full marathon in one straight run, he splits the marathon up in stages of 100m. After every 100m, he will take a 5 minute rest. Bolt, not so fit and not so healthy, runs every 100m of the marathon in a very poor 10 seconds. Still, he finishes the 42,195 km in a brilliant 1h 10 min 19.5 sec, averaging an incredible 36 km/h.
So, there we have it! Usain Bolt took the WR for the fastest ever 42,195 km (“marathon”)! Do you accept Bolt’s time as new WR for the marathon?
Probably not. And you are right not to! After all, Usain Bolt did not finish the marathon in 1h 10min 19.5sec, but actually used a lot more time. To be precise, he used 421,95x 5 minutes more as that’s the time he took to rest in between every 100m. Therefore, his real marathon time is 36h 20min 4.5sec!
ROG Pause and SuperPI
The same logic can be applied to the ROG Pause functionality with regards to any benchmark result. The easiest way to comprehend this is by using SuperPI, a benchmark which indicates how much time was needed to complete a specific calculation, as example. By enabling ROG Pause during a SuperPI benchmark, you do the exact same as what happened in the “Bolt’s Marathon” paradox. An example: there are two configurations, both assigned to complete the SuperPI 32M calculation as fast as possible. Both systems are configured exactly the same. System “Makau” will do the entire calculation in one run; System “Bolt” will be paused 1 second after every loop to cool down in order to ensure stability. The results are
- System “Makau”: 5min 0sec 0ms
- System “Bolt”: 5min 0sec 0ms
How much time did it take for both systems to finish the calculation? Well …
- It took System “Makau” exactly 5min 0sec to finish the entire SuperPI 32M calculation.
- It took System “Bolt” 5 minutes plus 25x 1 second, equaling 5min 25sec, to finish the calculation
As you figured out by now, the main conflict we have here is that there’s a difference between the measured time and the real time. And that’s where the existential crisis kicks in. Is it allowed to change the system’s perception of time to be altered during a benchmark? Is it allowed to split up any benchmark into an unlimited amount of sections? How long did it take to complete that 32M calculation?
3DMark01 and ROG Pause
Without any doubt, the first argument people will use in favor of the ROG Pause functionality is of course going to be 3DMark01. After all, the principle of starting and stopping a benchmark is essentially what makes 01 so different from all other benchmarks. And, yes, it’s true … even though 3DMark01 is supposed to be run as one long benchmark, overclockers still go subtest by subtest, segment by segment. But is it the same like ROG Pause? No, not at all. And here’s why:
- First of all, the legitimacy of splitting up the benchmark into different segments is a feature of the benchmark. The benchmark developers have determined that it’s allowed to go subtest by subtest to get an overall score.
- Secondly, the benchmark can only be split up in a pre-defined amount of subtests. Returning to the world of athletics, we could make the analogy with decathlon. In decathlon, there are also a pre-defined set of events in between which you can rest. It’s just part of the game.
- Thirdly, the measured time it took to render X amount of frames during each subtest is equal to real time. Unlike ROG Pause, having the ability to segment the 3DMark01 benchmark does not create a difference between measured and real time.
Therefore, 3DMark01’s application feature to split up the benchmark into different segments is not a valid argument in favor of tolerating ROG Pause.
Conclusive lines
Ultimately, there only a few ways to overcome this existential crisis. Either we allow a difference in measured time and real time or we don’t. Each way has their own set of implications: If allowed, we break with our previous policies against altering any of the definitions set by the benchmark developers. If disallowed, we have to find a way to detect the usage of ROG Pause to be able to ban it. As the entire system is halted, including any system timer, no software application can measure the real time it took to complete the benchmark, which means there’s no software tool to detect the usage of ROG Pause. Practically, there are two solutions.
One solution would be to develop a software tool that checks at what point a benchmark started, syncs with a calibrated off-system clock (e.g.: via the internet) and syncs again when the benchmark finishes. In this case, a significant difference between the measured off-system time and the measured on-system time would indicate tampering. This is an interesting solution, but it comes with a serious development cost (it has to be very secure) and obviously requires you to always have your bench system connected to the internet.
The second option, suggested to me by a fellow member of the overclocking community, is a lot more drastic: nullify all benchmark results done with the Maximus V Extreme (or any other mainboard supporting ROG Pause), which effectively bans the board from being used in a competitive overclocking environment.
In any case, as a community we will have to think about the validity of benchmark results done with pausing. As it does not only affect the principles of the competitive aspect of overclocking/benchmarking, but also affect a (commercial) benchmark’s business selling point as indication of stability, I assume that benchmark developers such as Futuremark will also have internal discussions on this subject. The main purpose of this write-up was to inform you about the problem and implications of the feature rather than effectively imposing a new policy or rule at HWBOT. Of course, at HWBOT, we’re very interested in hearing your opinion and thoughts, so feel free to comment in the forums!
