Benchmarks for 2012

[r4gs]

Since everyone is nice and busy on this forum today, I think it will be a good idea to get your feedback on this before we get started on our tests.

Our current (tentative) config for this year's test rig is as follows...

Intel i7 3960x
Intel DX79SI
RAM ( Unconfirmed as yet as mysteriously, all our triple channel RAM kits died on us and we have to get replacements)
1080p display.
600GB velociraptor with an Intel 60GB SSD

Now, for benchmarking graphics cards,

We normally run the following tests

3D Mark 11
3D Mark 06
Unigene Heaven 2.5
Far Cry 2
STALKER: Call of Pripyat
Resident Evil
Crysis Warhead (Will be dropped from future tests as it scales more with the CPU than the GPU in our tests)
Dirt 3
Crysis 2


For 2012,

3D Mark 11
Unigene Heaven 2.5
Crysis 2 (with dx11 and High-res texture pack)
F1 2011 (in dx9 and dx11 modes)
Alien vs Predator (It is old, but scales well across cards)
Starcraft 2

Please add what other tests you would like to see on this list and why.

Remember the following when suggesting a benchmark:

• The benchmark should scale well across all GPUs
• The benchmark should not be nVidia or AMD specific (so no physx tests for example)
• Some games don't have an in-built benchmark, so unless there is a pre-recorded run-through or something that can be used, it would be of no use
• If you can, please provide a link as it will make our jobs that much easier. Not a pre-requisite though.
• Most important: Please explain why you want that benchmark as just mindlessly including game lists won't help.

 

[Skud]

Lost Planet 2
Stalker: Call of Pripyat

Both have standalone benchmark tools.

Also for mid-range, low-end card Resident Evil 5 & Street Fighter IV. These two also come with standalone tool.

Oh, before I forget Metro 2033 (can be used to replace SCOP). But you need the full game.

 

[skeletor]

Don't use Starcraft 2. It is highly CPU oriented.

Stick with Crysis: Warhead. Crysis 2 is biased - crippling performance on AMD cards via *excessive* tessellation, on purpose. Crysis 2 tessellation: too much of a good thing? - The Tech Report - Page 1

Battlefield 3. Metro 2033. Dirt 3. Civilization V. --> my suggestions.

 

[Skud]

Yeah BF3 should be included.

 

[thetechfreak]

Metro 2033 definitely is a game to run benches in.
I think Metro Last light will also be something to look out for to run a few benches. Crisis 2? No from my side.


Dirt 3 too I think is good. Battlefield 3 too should be in there

Crysis Warhead would be better than C2.I think Crysis 2 depends too much on GPU than CPU. Ran perfect
On gamer settings on my Celeron + 9500Gt rig giving 35 frames easily at gamer settings(yea it was at low resolution)
Crysis Warhead is still able to tax my Phenom II x4 840 + 9500Gt now. Hardly got 10 frames at 1024x768 with enthusiast settings. Yes I know my GPU is culprit here bit still wanted to make my point

 

[r4gs]

Don't use Starcraft 2. It is highly CPU oriented.

Stick with Crysis: Warhead. Crysis 2 is biased - crippling performance on AMD cards via *excessive* tessellation, on purpose. Crysis 2 tessellation: too much of a good thing? - The Tech Report - Page 1

Battlefield 3. Metro 2033. Dirt 3. Civilization V. --> my suggestions.

The article is informative yes, but it only indicates that AMD's cards are not as capable of handling extreme levels of tesselation and that flaw will be apparent across all games, not just Crysis 2, as mentioned here...

Unnecessary geometric detail slows down all GPUs, of course, but it just so happens to have a much larger effect on DX11-capable AMD Radeons than it does on DX11-capable Nvidia GeForces. The Fermi architecture underlying all DX11-class GeForce GPUs dedicates more attention (and transistors) to achieving high geometry processing throughput than the competing Radeon GPU architectures. We've seen the effect quite clearly in synthetic tessellation benchmarks. Few games have shown a similar effect, simply because they don't push enough polygons to strain the Radeons' geometry processing rates.

Now, how do we deal with this? Do we ignore the fact that AMD's architecture is deficient at handling extreme tesselation and recommend it anyway? Especially considering that tesselation is a factor when purchasing high-end cards?

Would you buy a 7990 when you know that it won't handle a Crysis 2 or HAWX 2 as well as, say, a 690 at the same price? It is a theoretical situation, but what do you guys think?

Regarding the other games,
Battlefield 3: Forgot about that. Yep, should be included. I don't think it has an in-built benchmark but we'll see what can be done.
Metro 2033: Our testing with this game showed very erratic performance, I don't think this should be included but let us see what more of you guys have to say.
Dirt 3: F1 2011 is already included, as it uses the same engine, benchmarking Dirt 3 seems kind of redundant.

Crysis Warhead: The problem with this game is that the scores just stagnated after a certain point and there wasn't much appreciable difference across cards, the game was obviously hitting a CPU limited wall. It is alright for low to mid-range cards though.

@thetechfreak: I actually don't get your point, you're comparing 2 different settings on 2 different setups. Also, how did you benchmark it?

 

[thetechfreak]

What I meant to say is that Crysis 2 is not good enough for benches for now. It I think scales the GPU too much than CPU. so Crysis warhead is to go for benches. Regarding how I benchmarked it? No I didn't run the benches for Crysis 2. Just played through the game. Posting on that experience

 

[skeletor]

The article is informative yes, but it only indicates that AMD's cards are not as capable of handling extreme levels of tesselation and that flaw will be apparent across all games, not just Crysis 2, as mentioned here...

Unnecessary geometric detail slows down all GPUs, of course, but it just so happens to have a much larger effect on DX11-capable AMD Radeons than it does on DX11-capable Nvidia GeForces. The Fermi architecture underlying all DX11-class GeForce GPUs dedicates more attention (and transistors) to achieving high geometry processing throughput than the competing Radeon GPU architectures. We've seen the effect quite clearly in synthetic tessellation benchmarks. Few games have shown a similar effect, simply because they don't push enough polygons to strain the Radeons' geometry processing rates.

Now, how do we deal with this? Do we ignore the fact that AMD's architecture is deficient at handling extreme tesselation and recommend it anyway? Especially considering that tesselation is a factor when purchasing high-end cards?

Would you buy a 7990 when you know that it won't handle a Crysis 2 or HAWX 2 as well as, say, a 690 at the same price? It is a theoretical situation, but what do you guys think?

You've missed the point. It's more like - pre-HD 7000 series AMD cards are not capable of handling *insane* level of tessellation.

Fact is, Crysis 2's *insane* level of tessellation adds nothing to visual quality as one would expect. Normal level of tessellation would have been fine. I don't see any point in heavily tessellating an already flat slab and running a tessellated water mesh below the ground for NOT particular reason. Tessellation here was used to cripple performance on competitor's GPU.

Let's take Battlefield 3 for example. Obviously involves tessellation, but as much as it should be. This game gives you the right pecking order.

HD 7970 > HD 7950/GTX 580 > HD 6970/GTX570 > HD 6950/GTX 560 Ti > GTX 560/HD 6870.

 

[r4gs]

What I meant to say is that Crysis 2 is not good enough for benches for now. It I think scales the GPU too much than CPU. so Crysis warhead is to go for benches. Regarding how I benchmarked it? No I didn't run the benches for Crysis 2. Just played through the game. Posting on that experience

Ummm.... Ok, but that isn't necessarily a reliable benchmark. From what I can make out, you claim the game was playable on gamer settings on your celeron and unplayable at enthusiast on your phenom. Also, how did you get the scores of 35 and 10? Fraps? Was that the average score across a certain playthrough or was it just you glancing at the overlay FPS on and off while playing?

My main problem with your post is just this, what makes you say that it "scales the GPU too much than CPU"?

If you remember the graphics card benchmark we conducted last year, we found that the game scaled very well across all cards (40 GPUs I think) and posted very consistent and usable results. At that time we didn't apply the DirectX11 or high-res texture patch though.

 

[skeletor]

What I meant to say is that Crysis 2 is not good enough for benches for now. It I think scales the GPU too much than CPU. so Crysis warhead is to go for benches. Regarding how I benchmarked it? No I didn't run the benches for Crysis 2. Just played through the game. Posting on that experience

didn't get you.

 

[Extreme Gamer]

I haven't seen whats included, but:

For GPU:

Crysis 2
Crysis 1
Battlefield 3
Batman: Arkham City (physx disabled)
Serious Sam 3: BFE
Metro 2033

CPU/Mobo/RAM:

COD (MW2 onwards)
Red Faction: Armageddon
Battlefield 3
Starcraft II
Civilization V
Crysis Warhead
StarCraft II

Synthetic:

3D Mark 11
3D Mark Vantage (Physx Disabled)
PC Mark 7
Cinebench
Unigine Heaven 2.5
Stone Giant
LinPack tests
WPrime

How is Crysis warhead not scaling well for you guys? As long as the right CPU-GPU combination is chosen, scaling should always cross 90% in multi-GPU scenarios. Besides, I thought it was common knowledge that Crysis enjoyed a beefy CPU more than a beefy GPU( in the sense that the fastest of GPUs will stutter if your CPU is a C2D): it does better Physics than any physx game of today off the CPU lol. With the SB-E chip you guys should have no problems testing GPU performance: My i7 960 at stock could handle the game with two 580s at a comparatively lower resolution (1680x1050), where things become more CPU-bound, especially in FPSes.

I would like to suggest a few changes to the test bench if possible:

A 2560x1600 monitor should be on your list apart from testing with 3 1080p monitors in surround if it is not already there. The 1080p monitors dont have to be identical- in benches you want the framerates and quality settings rather than "viewing pleasure", so it should be a plug and play process with monitor reuse.
The mobo, if possible should be changed. The board has less power phases than even cheaper X58 boards.
I do not find it surprising that literally all of your 3-ch RAM kits failed. DDR3 kits and controllers have always been very sensitive to even slight voltage changes in my experience- much more than DDR2 has ever been. On the upside, you guys can get 4 -ch kits to tread X79 "right" lol.

Spoiler

I'm sorry if this is off-topic, it is in response to previous posts opposing Crysis 2.

To those who oppose crysis 2:

Before crying about "excessive tess" in Crysis 2, dont forget that both Nvidia and AMD face the same runtime environments in the game. Fact is that Nvidia has a better tess engine in the GF100/110 line vs Northern Islands. Besides, DX9 mode is still meaty.

The fact that Crysis 2 implements tessellation in that manner is not completely Crytek's fault. Microsoft designed the API to treat whole objects rather than just portions. This was intended to ensure that the polygon reduction or increase scaled well with viewport movement. The DX11 API can behave so erratically, it even calls for the road to be tessellated when the floor is tessellated (causing ridge formation on the white paint on the road).
While a lot of hidden portions of water faces tessellation, that is a limitation of the editor/engine. Water is one entity in the game. Every bit of water in a level is a part of its "ocean," be it a fountain, a waterfall or a river (which is more often than not a visible portion of the ocean itself).
You're concerned because this is tessellation. Many games actually do this to standard geometry and objects.
The Sandbox Editor does not follow the traditional "quake" approach of building maps. When you call for a new map to be created in the editor, the first thing you see is a vast ocean . On top of that you add an island and then you model that island a bit like clay. You do not follow traditional approaches in the first few steps. The engine cannot understand which parts of the water hidden by the island will visible to the player, so it cannot choose to purposely not render some portions (that is beyond the logic of the code).
You do realize that sequel engines are most often not built from scratch? Assuming they could have "fixed" this if they'd built from the beginning, you would have to wait for at least another 3 years for Crysis 2, and the devs would have gone broke by then (at the very least EA, being EA, would have stopped funding Crysis 2 and the franchise would be terminated; between Crysis Warhead and Crysis 2, a significant amount of time had passed).

All this is not to say the devs were not half-lazy: the PC version released as a console port (albeit well optimized) with a Day 0 update, and they should have chosen not to tessellate the jersey barrier. When you deal with EA, this is what happens to you: you sink in the world of corporate suits who care about the bottom line more than their customers. Crysis 2 is a bad example of a Cryengine 3 game. The engine itself is awesome.

 

[skeletor]

While a lot of hidden portions of water faces tessellation, that is a limitation of the editor/engine. Water is one entity in the game. Every bit of water in a level is a part of its "ocean," be it a fountain, a waterfall or a river (which is more often than not a visible portion of the ocean itself).

Vindicates my reason to avoid this game.

Heavily tessellated water everywhere (for no reason) and you can't even see? What's the point? Done on purpose to "cripple"?

*img405.imageshack.us/img405/5954/img0032910.jpg

*img31.imageshack.us/img31/6109/img0032911.jpg

 

[Piyush]

yea...crysis 2 is not neutral. crysis/warhead is...better looking game as well imo and that's what should be there.

Crytek's decision to deploy gratuitous amounts of tessellation in places where it doesn't make sense is frustrating, because they're essentially wasting GPU power—and they're doing so in a high-profile game that we'd hoped would be a killer showcase for the benefits of DirectX 11. Now, don't get me wrong. Crysis 2 still looks great and, in some ways at least, is still something of a showcase for both DX11 and the capabilities of today's high-end PCs. Some parts of the DX11 upgrade, such as higher-res textures and those displacement-mapped brick walls, appreciably improve the game's visuals. But the strange inefficiencies create problems. Why are largely flat surfaces, such as that Jersey barrier, subdivided into so many thousands of polygons, with no apparent visual benefit? Why does tessellated water roil constantly beneath the dry streets of the city, invisible to all?

 

[Extreme Gamer]

Vindicates my reason to avoid this game.

Heavily tessellated water everywhere (for no reason) and you can't even see? What's the point? Done on purpose to "cripple"?

*img405.imageshack.us/img405/5954/img0032910.jpg

*img31.imageshack.us/img31/6109/img0032911.jpg

Did you read what I posted? The engine treats all the water as one entity. You cant cut off segments. This is not a quake engine deriviative.

 

[Joker]

another no here for crysis 2. but u can use unigine heaven "normal" and "extreme" tess if u want.

infact hd 7000 (the ones u are likely to review now) series has better tess than gtx 500 series. but benchmark set should be neutral.

Did you read what I posted? The engine treats all the water as one entity.

stupid thing to do. whole point is about heaving a neutral benchmark set.

 

[skeletor]

well, I had suggested Civilization V in my suggestions. That game also involves heavy tessellation and nVidia cards perform better in that game if you take HD 7000 series out of consideration. Point is, that game isn't shady in its implementation like Crysis 2 is. Crysis 2 cripples for no visual benefit. Not the case in Civilization V.

 

[r4gs]

Very interesting. I didn't know that about Crysis 2. The water bit I mean, will have to look into it further. Ok then, since so many of you guys are opposed to it, Crysis 2 will be dropped from the benchmarks. We shall still keep it for reference but not consider it for performance scores.

Considering that we are first going to kick off our benchmarking with an ASUS 7970, we will run all possible benchmarks just to be on the safe-side.

Sorry, can't change the board, it is the only one we could get for long-term use and also, the RAM modules went "kaput" before we got the X79 mobo.

@Extreme gamer:

3x1080p monitors? It isn't really feasible and to what effect? I don't know if even 1% of our readers have such setups.

Isn't Arkham City supposed to have problems with AMD cards? Something to do with tesselation again if I remember correctly.

 

[skeletor]

Isn't Arkham City supposed to have problems with AMD cards? Something to do with tesselation again if I remember correctly.

Nothing related to tessellation this time. Arkham City is fine. It's just that it's a buggy game, but still it's fine.

 

[Jaskanwar Singh]

dont forget to submit some results at ThinkDigit Forum @ hwbot. Link in my sig.

 

[Extreme Gamer]

Very interesting. I didn't know that about Crysis 2. The water bit I mean, will have to look into it further. Ok then, since so many of you guys are opposed to it, Crysis 2 will be dropped from the benchmarks. We shall still keep it for reference but not consider it for performance scores.

Considering that we are first going to kick off our benchmarking with an ASUS 7970, we will run all possible benchmarks just to be on the safe-side.

Sorry, can't change the board, it is the only one we could get for long-term use and also, the RAM modules went "kaput" before we got the X79 mobo.

@Extreme gamer:

3x1080p monitors? It isn't really feasible and to what effect? I don't know if even 1% of our readers have such setups.

Isn't Arkham City supposed to have problems with AMD cards? Something to do with tesselation again if I remember correctly.

3x1080p because then you are not bottlenecked by CPUs while benchmarking. If a card can average 30fps @medium settings in most games in that resolution, it is worth buying across all setups.

I never said that the RAM died because of X79 (because my experience is with X58 and P55 in DDR3)
Now you guys can get a few quad channel RipjawsZ 16GB kits from G.Skill (1600Mhz kits are decently priced).

Arkham City had issues across the board when it was launched with DX11. The issue is now fixed.

@Joker: The engine reuses code from CE2 for reasons I stated before. It works in a manner that is not very different to CE2 (where the water was rendered under the soil(only, it had not been tessellated). No, it is not stupid. It saves you a lot of effort when making maps, as can be exemplified by Crysis 1.
In Cryengine 3, it is not the dev's intention to crush systems, it performs better than Crysis across the board even in DX9 while maintaining visual fidelity. The fact that the water cannot be clipped is a coding byeproduct and to fix this they would probably have to rewrite whole sections.

From a financial standpoint it was better for the devs to not do that: another two years fixing this could cause heavy losses. Crytek did not have a game out for 3 years after Warhead.
Other games too show such behavior (excessive rendering). only when it is tessellation, people get angry.

____
You guys should note that Unigine Heaven 2.5, based on your logic, should not be included: it does 3x more tessellation work than CE3. Literally everything is tessellated. Does that not make the benchmark biased (based on your logic)?