3DMark11 Physics test

Hi there guys, I need your help about a thing I ran through.

I think that the upgrade between version 1.0.1 and 1.0.2 brought some problems at consistency of the scores in Physics test.

I first found this in the Llano competition.

CPU@3,45ghz, ram@1200 9-11-9-27, 5.065 in that test.

After having 3dm11 patched to 1.0.2, with the same settings, the score went down to 4.833.

Same "fall" in points can be seen with a SB system.

If you pay attention to the fps rate during the test, you'll notice that with version 1.0.1, the fps are incresing througout the entire test.

With patch 1.0.2 installed, in the last 3/4seconds, the fps are descreasing a little.

So, the purpose of this thread is: can you guys please do comparative runs, using the same settings, first with 1.0.1 version, than patching it to 1.0.2 and having the second run.

Then post here your findings.

IF I'm right, there's a serious problem here, about the consistency of results with different patch levels

Thanks!!

This is normal. Bullet library was updated between patches and the scores of the Physics and Combined tests changed slightly. What you are seeing is exactly this.

The difference was considered small enough so that we did not invalidate results from older versions but for any contests etc. there should always be a rule to use the latest available version at the start of the competition or this kind of issue may crop up...

Well Jarnis, so you confirm I'm not crazy

Anyway, I propose to the staff to add the rule that, from now on, only 1.0.2 could be used.

It's not a huge difference in points, ok, but if someone's fighting for the WR in a category, a few points could mean first or second place

I agree with |ron on this, latest version should be used on all new scores

This is normal. Bullet library was updated between patches and the scores of the Physics and Combined tests changed slightly. What you are seeing is exactly this.

 

The difference was considered small enough so that we did not invalidate results from older versions but for any contests etc. there should always be a rule to use the latest available version at the start of the competition or this kind of issue may crop up...

I don't know what your definition of "small enough" is, but at the top level of 3DMark11 scores (>20K) it's easily over 1000 pts.

I thought in the past, scores were not supposed to change with different versions of Futuremark benchmarks. At the least there should be some more blatant way to invalidate scores done with 3DMark11 1.0.1.

I compared the two and when I was installing with patch 1.0.2, fps was decreasing little bit at final stage of it.

I don't know what your definition of "small enough" is, but at the top level of 3DMark11 scores (>20K) it's easily over 1000 pts.

 

I thought in the past, scores were not supposed to change with different versions of Futuremark benchmarks. At the least there should be some more blatant way to invalidate scores done with 3DMark11 1.0.1.

For 3dmark.com, you can see what benchmark version was used for each result.

Regarding FM Hall of Fame, our web team is in process of completely revamping that page and a new version should appear in Jan/Feb - included in that is enforcement that HoF only shows results from the latest version of each benchmark so for Hall of Fame purposes, old scores will be kinda-invalidated at that point.

Up to you how you deal with this in hwbot.org or in any other toplists etc. - the version used (Benchmark and SystemInfo) is visible always on 3dmark.com so it can be verified for any benchmark result that is submitted to the 3dmark.com service.

If you have further feedback on this, just toss me an email at info [at] futuremark.com and I'll happily forward it to the people who make decisions related to this at FM. All feedback is always welcome and will be considered. This is especially true for matters that relate to our benchmarks and any competitive benchmarking; we are making the product for our users and naturally want it to be relevant and fair.

For example, we may consider adding a line to the result validity warning box on 3dmark.com (the one that includes warnings for unapproved drivers / tessellation controls etc) for results obtained with older versions. Would that help?

then fix the issue with pcmark04 grammar check test and PCMark Vantage with AMD CPUs

 

For example, we may consider adding a line to the result validity warning box on 3dmark.com (the one that includes warnings for unapproved drivers / tessellation controls etc) for results obtained with older versions. Would that help?

This would be a good addition IMO.

then fix the issue with pcmark04 grammar check test and PCMark7 with AMD Dual core CPU

PCMark 04 is so old and unsupported, this will not be fixed. Sorry.

As for PCMark 7, I'm not sure what the problem you are referring to is? We do have PCMark 7 patch in development for (tentatively) early 2012 launch and there are some issues that are getting fixed but I'm not certain which one you refer to.

This would be a good addition IMO.

Will add that to the feature suggestion list internally and we'll take a look in January.

so for the time being even If I would be to use 3dmark 11 1.01 version along 280.26 drivers my score would still count for the hall of fame right?

PCMark 04 is so old and unsupported, this will not be fixed. Sorry.

The issue has been reported many times way before it has been discontinued but this you already know I alreday told you

As for PCMark 7, I'm not sure what the problem you are referring to is? We do have PCMark 7 patch in development for (tentatively) early 2012 launch and there are some issues that are getting fixed but I'm not certain which one you refer to.

oops sorry, was for PCmark Vantage not PCMark 7,

This one: http://www.yougamers.com/forum/showthread.php?t=138198&page=2

so for the time being even If I would be to use 3dmark 11 1.01 version along 280.26 drivers my score would still count for the hall of fame right?

1.02 at least. Not sure about 1.01.

But it won't count for very long, we are going to change it.

 

oops sorry, was for PCmark Vantage not PCMark 7,

 

This one: http://www.yougamers.com/forum/showthread.php?t=138198&page=2

This one is in the tracking database as a confirmed issue but I do not know if/when PCMark Vantage will receive a patch. There is also some uncertainty if the bug is actually in Microsoft code (naturally it could be worked around in our code, but that may require full revalidation and testing cycle for all benchmark results as such a workaround would affect the run on all platforms).

So at this point the answer is "I don't know". I'll poke the PCMark lead programmer when he gets back from the Christmas holiday and check if there is any news.

okay thanks for your heelp as always

so for the time being even If I would be to use 3dmark 11 1.01 version along 280.26 drivers my score would still count for the hall of fame right?

I already tried .

280.26 doesn't work with X79 properly: it doesn't allow 4Way SLI to be enabled.

FYI, the HOF rules do state that v1.0.1 is still allowed: "The lists are compiled from submitted default benchmark results of 3DMark 11 (Build 1.0.1 or newer), 3DMark Vantage (Build 1.0.2), 3DMark06 (Build 1.2.0)." (http://www.3dmark.com/hall-of-fame/)

For example, we may consider adding a line to the result validity warning box on 3dmark.com (the one that includes warnings for unapproved drivers / tessellation controls etc) for results obtained with older versions. Would that help?

What would even help more is add this information in the GUI.

Why don't you just disallow older versions altogether anyway? If the score is different, you can't compare anyway.

massman im a bit at a loss here, do I rerun my last submission just to get the extra physix points so i can compete with the others (will i look like a dush then??) or wait a few days and your gone bring in a rule that only latest 3d11 version is allowed on subs that are being made after that versions launch? for example every one who made 1.01 subs when that was the current version obviously are lucky bastards that get to have a high pt score, but people who submited 1.01 scores after 1.02 or 1.03 was out should rerun theres to make it fair with the rest?

Good question, I opened a discussion thread at the staff forum about it.

In essence, it's the same issue as with tesselation: we can't get any info from the GUI, so we can't be entirely sure based on a screenshot. IF we disallow 1.0.1, we have to enforce verification links (which is something we've never done before and comes with its own set of problems). If we just go with all versions allowed, just like we allow tesselation as tweak, we have no worries at all.

I wouldn't mind the latter; we've never really followed FM's policies historically speaking (allowing LoD, disabling SI, non-FM approved drivers, tesselation, etc) and I don't think it's HWBOT's ambition to only serve FM-allowed submissions. This site's mainly about enthusiasts pushing systems.

Well you've kind of required FM verification links for a while, they're required for Global top-20 results aren't they?

With FM changing their rules to say you have to use the latest revision for validation, you won't be able to get the required link for HWBot top-20 anymore. (assuming you've used an 1.0.1)

FM verification links were required for top-20 global, that's correct. But not for that other 99% of the submitted results.

I'm unsure how FM will change the rule exactly, but currently you can still get a verification link for 1.0.1, 1.0.2 and 1.0.3 results. If the older versions would not allow you to create a link anymore, it would be one step. But, it doesn't solve the issue for that other 99%.

You will be able to submit to 3dmark.com and get a link even on score that is not officially valid.

But it won't appear on FM.com hall of fame page unless it is deemed valid. The 3dmark.com result page will also have this box that tells how and why the score is not valid for Hall of Fame purposes - stuff like unapproved drivers, tessellation tweaking on AMD drivers or if the hardware isn't properly detected etc.

FYI, the HOF rules do state that v1.0.1 is still allowed: "The lists are compiled from submitted default benchmark results of 3DMark 11 (Build 1.0.1 or newer), 3DMark Vantage (Build 1.0.2), 3DMark06 (Build 1.2.0)." (http://www.3dmark.com/hall-of-fame/)

Correct. We plan to change that in January.

What would even help more is add this information in the GUI.

We can't. A lot of the validation is done on the 3dmark.com site. It is the only way to make sure the validation is not hackable. Any indicator on the GUI that would be based on what 3dmark.com says would be complex to implement and still hackable.

The only real way to ensure that the result is not tampered with is to check the result on 3dmark.com. I know you guys use screenshots and all that, but you really should also require a 3dmark.com result link which indicates that the result is valid.

Why don't you just disallow older versions altogether anyway? If the score is different, you can't compare anyway.

On 3dmark.com, we may do that on a later date. We don't want to do it instantly because there are millions of 3DMark 11 installations out there and people would get very very pissed if they run the benchmark and then as they go to submit (3dmark.com view is the only one on basic edition) it would tell that they would have to patch and rerun before getting a result.

Basically once large enough percentage of all submits are 1.0.3, we may disallow previous versions completely. If I recall right, at the moment 1.0.1 is the minimum for getting a result (1.0 would just tell you to upgrade).

But again, this will influence HoF sooner. The only reason this will take a bit is because it means a change in HoF code and we are already rewriting the whole page, so the change will come with the HoF page update (it will also have new features!)