During September we managed to get hold of some Haswell-EP samples for a quick run through our testing suite. The Xeon E5 v3 range extends beyond that of the E5 v2 with the new architecture, support for DDR4 and more SKUs with more cores. These are generally split into several markets including workstation, server, low power and high performance, with a few SKUs dedicated for communications or off-map SKUs with different levels of support. Today we are testing two 10 core models, the Xeon E5-2687W v3 and the Xeon E5-2650 v3.

Intel Xeon E5 v3: The Information

Our initial Haswell-EP coverage from Johan was super extensive and well worth a read for anyone interested in the Xeon platform. My focus here will be light in comparison, mentioning key points that as an ex-workstation user I find interesting. This will be the first of several reviews on the Xeon processors, which we have split up to focus more on each area.

The core layouts for each of the different levels of processor are from three designs, emulating the single and dual ring bus type arrangements depending on the number of cores in each SKU. As with the Xeon E5 v2 processors, the big block of cache is in the middle of the cores and data is transferred via the ring bus. From the core designs, pairs of cores can be disabled to make lower core count CPUs, and much like the previous generation, some low core / high cache models might be possible.

In the 10-12 core image above we essentially get two classes of cores – one in the big stack to the left and another to the right. The processor is designed to treat all cores equally, although the Cluster on Die snoop mode new to E5 v3 will organize the cache data into what acts like two big sections in a NUMA style-arrangement. This allows data relevant to cores that need it to stay close and hopefully reduce read/write latencies, but is all transparent to the user. Johan goes into more detail on this front in his review.

This column arrangement is also why we do not see the progressive jump in cores we would expect. In the consumer space, we have had 1, 2, 4, 6, 8 cores, and one might expect 12 and 16 on the horizon, but 10, 14 and 18 seem a little off canter, along witht the 15-core design from Ivy Bridge-EP. Using this column design, Intel has to balance the number of cores per ring and the number of cores per column. In the large 18 core design there are 10 cores in the secondary ring and six in a single column – ideally fewer columns would be preferable however more rings allows data to transfer more frequently. It becomes a bit of a balance in terms of design, efficiency, performance and yield at the end of the day, especially when dealing with up to 5.69B transistors in 662 mm².

CPU Specification Comparison
	CPU	Node	Cores	GPU	Transistor Count (Schematic)	Die Size
Server CPUs
Intel	Haswell-EP 14-18C	22nm	14-18	N/A	5.69B	662mm2
Intel	Haswell-EP 10C-12C	22nm	6-12	N/A	3.84B	492mm2
Intel	Haswell-EP 6C-8C	22nm	4-8	N/A	2.6B	354mm2
Intel	Ivy Bridge-EP 12C-15C	22nm	10-15	N/A	4.31B	541mm2
Intel	Ivy Bridge-EP 10C	22nm	6-10	N/A	2.89B	341mm2
Consumer CPUs
Intel	Haswell-E 8C	22nm	8	N/A	2.6B	356mm2
Intel	Haswell GT2 4C	22nm	4	GT2	1.4B	177mm2
Intel	Haswell ULT GT3 2C	22nm	2	GT3	1.3B	181mm2
Intel	Ivy Bridge-E 6C	22nm	6	N/A	1.86B	257mm2
Intel	Ivy Bridge 4C	22nm	4	GT2	1.2B	160mm2
Intel	Sandy Bridge-E 6C	32nm	6	N/A	2.27B	435mm2
Intel	Sandy Bridge 4C	32nm	4	GT2	995M	216mm2
Intel	Lynnfield 4C	45nm	4	N/A	774M	296mm2
AMD	Trinity 4C	32nm	4	7660D	1.303B	246mm2
AMD	Vishera 8C	32nm	8	N/A	1.2B	315mm2

Intel should be offering certain configurations with more L3 cache, given that in their press materials the one they labelled '10C-12C' will actually be offered as a cut down to six cores for release. These CPUs, whichever way you slice them, are still massive.

Today our review revolves around two of the 10 core options from Intel.

Intel Xeon E5 v3 SKU Comparison
Xeon E5	Cores/ Threads	TDP	Clock Speed (GHz)	Price
High Performance (35-45MB LLC)
2699 v3	18/36	145W	2.3-3.6	$4115
2698 v3	16/32	135W	2.3-3.6	$3226
2697 v3	14/28	145W	2.6-3.6	$2702
2695 v3	14/28	120W	2.3-3.3	$2424
"Advanced" (20-30MB LLC)
2690 v3	12/24	135W	2.6-3.5	$2090
2680 v3	12/24	120W	2.5-3.3	$1745
2660 v3	10/20	105W	2.6-3.3	$1445
2650 v3	10/20	105W	2.3-3.0	$1167
Midrange (15-25MB LLC)
2640 v3	8/16	90W	2.6-3.4	$939
2630 v3	8/16	85W	2.4-3.2	$667
2620 v3	6/12	85W	2.4-3.2	$422
Frequency optimized (10-20MB LLC)
2687W v3	10/20	160W	3.1-3.5	$2141
2667 v3	8/16	135W	3.2-3.6	$2057
2643 v3	6/12	135W	3.4-3.7	$1552
2637 v3	4/8	135W	3.5-3.7	$996
Budget (15MB LLC)
2609 v3	6/6	85W	1.9	$306
2603 v3	6/6	85W	1.6	$213
Power Optimized (20-30MB LLC)
2650L v3	12/24	65W	1.8-2.5	$1329
2630L v3	8/16	55W	1.8-2.9	$612

The E5-2687W v3 is an interesting model of the bunch, particularly due to the importance of the E5-2687W v2 from the previous generation. The v2 version was lauded due to the difference in peak frequencies compared to the higher core count models, but this changes with Haswell-EP.

For Ivy Bridge-EP:

- The 8-core E5-2687W v2 gave 3.6 GHz in full-load, TDP of 150W for $2108,
- The 12 core E5-2697 v2 gave 3.0 GHz in full-load, TDP of 130W for $2614

With Haswell-EP:

- The 10-core E5-2687W v3 gives 3.2 GHz for 160W at $2057,
- The 14-core E5-2697 v3 gives 3.1 GHz for 145W at $2702 or
- The 18-core E5-2699 v3 gives 2.8 GHz for 145W at $4115

If we compare the difference between the E5-2687W and E5-2697, first with v2 and then v3, it makes the new Haswell ‘W for Workstation’ CPU a little less enticing. Previously it was a trade-off between cores and frequency, and depending on the software having a high turbo mode helps with the v2 CPUs.

To make matters worse for the E5-2687W v3, if we compare single thread speeds, the E5-2697 v3 reaches 3.6 GHz compared to the E5-2687W v3 at 3.5 GHz, which puts the W processor at a disadvantage.

It is worth noting that Intel puts these two processors in different parts of the product stack, to technically they should not be 'competing' against each other:

The E5-2687W v3 is firmly for Workstations only, rather than servers, whereas the E5-2697 v3 should end up in 2U servers. 

The other processor in this review, the E5-2650 v3 sits in the ‘Advanced’ section in the SKU stack, giving 2.6 GHz at load or 3.0 GHz for single threaded speed, but lists at only 105W for $1166 tray price.

Using this information and a few SKUs that are off-roadmap, the turbo modes of the 10 core processors are:

All the 10 core processors reach their full-core turbo when five cores are in use, and are on the top turbo frequency when one or two cores are active.

The Chipset

When we reviewed a pair of the E5 v2 processors back in March, the main server based chipsets at the time revolved around the C600 series, codename ‘Patsburg’. For the v3 processors, this moves to the C610 series, also known as Wellsburg. The C612 chipset is the primary server component at this point, offering many of the features we have already seen in our X99 reviews:

- Up to 10 SATA 6 Gbps,
- 6 ports of USB 3.0,
- 8 ports of USB 2.0
- Up to 8 PCIe 2.0, with x1/x2/x4 supported

New features for C610 series include:

 - Reduced TDP, Average Power and Package (now 7W, 25mm x 25mm)
 - Intel SVT
 - USB 3.0 XHCI Debug
 - Support for MCTP Protocol and End Points
 - Support for Management Traffic over DMI
 - SPI Enhancements

Intel vPro, SPS 3.0, RSTe and CAS are also supported.

For the SATA/USB3/PCIe bencwidth combinations, Intel has implemented an extended from of Flex IO. It almost looks much the same at Z87 and Z97, offering 22 rather than 18 differential signal pairs. A certain amount of these pairs are fixed to USB3 / PCIe / SATA but two pairs are muxed:

This slide shows 18 signal pairs, although I mentioned 22. This is because the last four are from a secondary AHCI controller giving four more SATA 6 Gbps ports. Like X99, the downside of these secondary SATA ports is that they are not RAID capable due to limitations within the silicon.

MTCP over PCIe is also an interesting new addition to Wellsburg, allowing cross CPU communication from controllers attached to the other side of the system:

The DRAM

We still have a consumer class DDR4 review in the works, but the upgrade from DDR3 to DDR4 for Haswell-EP is more significant. The decrease in power consumption is often listed is the easiest-to-explain benefit, giving an approximate 2W saving at-the-wall per memory module:

One important aspect of DDR4 will be the higher memory frequency, especially when more DIMMs per channel are installed. It might also come to pass that some server motherboard manufacturers will end up supporting the DDR4-2133 at 3DPC, similar to some efforts made with Patsburg.

In a lot of Intel materials we received, it was worth noting that non-ECC UDIMM support is not often listed with the new Haswell-EP CPUs, but we can confirm that in our testing, all of our CPUs worked with standard consumer grade UDIMMs.

Competition and Market

As Johan mentioned in our initial Haswell-EP coverage, Intel’s main competition is with itself. Some other server CPU manufacturers focus on particular hardware and software combinations, while AMD has not updated their server line in over 2 years. ARM is making some inroads into the low end or the highly parallel market, but for the majority of workstations or servers, Intel has the market. The only downside to this strategy is that Intel has to convince that upgrading is worth it. One way to implement this is infrequent updates, although the customers demand a certain level of consistency over time such that updates are not super frequent.

Intel Xeon E5 v2 versus v3 2-socket SKU Comparison
Xeon E5	Cores/ Threads	TDP	Clock Speed (GHz)	Price	Xeon E5	Cores/ Threads	TDP	Clock Speed (GHz)	Price
High Performance (20 – 30MB LLC)					High Performance (35-45MB LLC)
					2699 v3	18/36	145W	2.3-3.6	$4115
					2698 v3	16/32	135W	2.3-3.6	$3226
2697 v2	12/24	130W	2.7-3.5	$2614	2697 v3	14/28	145W	2.6-3.6	$2702
2695 v2	12/24	115W	2.4-3.2	$2336	2695 v3	14/28	120W	2.3-3.3	$2424
					"Advanced" (20-30MB LLC)
2690 v2	10/20	130W	3-3.6	$2057	2690 v3	12/24	135W	2.6-3.5	$2090
2680 v2	10/20	115W	2.8-3.6	$1723	2680 v3	12/24	120W	2.5-3.3	$1745
2660 v2	10/20	115W	2.2-3.0	$1389	2660 v3	10/20	105W	2.6-3.3	$1445
2650 v2	8/16	95W	2.6-3.4	$1166	2650 v3	10/20	105W	2.3-3.0	$1167
Midrange (10-20MB LLC)					Midrange (15-25MB LLC)
2640 v2	8/16	95W	2.0-2.5	$885	2640 v3	8/16	90W	2.6-3.4	$939
2630 v2	6/12	80W	2.6-3.1	$612	2630 v3	8/16	85W	2.4-3.2	$667
2620 v2	6/12	80W	2.1-2.6	$410	2620 v3	6/12	85W	2.4-3.2	$422
Frequency optimized (15-25MB LLC)					Frequency optimized (10-20MB LLC)
2687W v2	8/16	150W	3.4-4.0	$2108	2687W v3	10/20	160W	3.1-3.5	$2141
2667 v2	8/16	130W	3.3-4.0	$2057	2667 v3	8/16	135W	3.2-3.6	$2057
2643 v2	6/12	130W	3.5-3.8	$1552	2643 v3	6/12	135W	3.4-3.7	$1552
2637 v2	4/12	130W	3.5-3.8	$996	2637 v3	4/8	135W	3.5-3.7	$996
Budget (15MB LLC)					Budget (15MB LLC)
2609 v2	4/4	80W	2.5	$294	2609 v3	6/6	85W	1.9	$306
2603 v2	4/4	80W	1.8	$202	2603 v3	6/6	85W	1.6	$213
Power Optimized (15 – 25MB LLC)					Power Optimized (20-30MB LLC)
2650L v2	10/20	70W	1.7-2.1	$1219	2650L v3	12/24	65W	1.8-2.5	$1329
2630L v2	6/12	70W	2.4-2.8	$612	2630L v3	8/16	55W	1.8-2.9	$612

 

Test Setup

For our testing, it is worth noting that our CPU samples arrived at different times. Due to the testing setup at those times, certain benchmarks were unable to be run due to updates required. As a result, we have no power data or single GPU benchmarks for the E5-2650 v3.

Test Setup
Processor	Intel Xeon E5-2687W v3 (160W), 10C/20T : 3.1 GHz (3.5 GHz Turbo) Intel Xeon E5-2650 v3 (105W), 10C/20T : 2.3 GHz (3.0 GHz Turbo)
Motherboards	ASUS X99-Deluxe ASRock X99 Extreme6
Cooling	Cooler Master Nepton 140XL Corsair H80i Thermalright TRUE Copper
Power Supply	OCZ 1250W Gold ZX Series Corsair AX1200i Platinum PSU
Memory	Corsair DDR4-2133 C15 4x8 GB 1.2V G.Skill Ripjaws 4 DDR4-2133 C15 4x8 GB 1.2V
Memory Settings	JEDEC @ 2133
Video Cards	MSI GTX 770 Lightning 2GB (1150/1202 Boost)
Video Drivers	NVIDIA Drivers 332.21
Hard Drive	OCZ Vertex 3 256GB
Optical Drive	LG GH22NS50
Case	Open Test Bed
Operating System	Windows 7 64-bit SP1

Many thanks to...

We must thank the following companies for kindly providing hardware for our test bed:

Thank you to OCZ for providing us with PSUs and SSDs.
Thank you to G.Skill for providing us with memory.
Thank you to Corsair for providing us with an AX1200i PSU and a Corsair H80i CLC.
Thank you to MSI for providing us with the NVIDIA GTX 770 Lightning GPUs.
Thank you to Rosewill for providing us with PSUs and RK-9100 keyboards.
Thank you to ASRock for providing us with some IO testing kit.
Thank you to Cooler Master for providing us with Nepton 140XL CLCs.

Load Delta Power Consumption

Power consumption was tested on the system while in a single MSI GTX 770 Lightning GPU configuration with a wall meter connected to the OCZ 1250W power supply. This power supply is Gold rated, and as I am in the UK on a 230-240 V supply, leads to ~75% efficiency > 50W, and 90%+ efficiency at 250W, suitable for both idle and multi-GPU loading. This method of power reading allows us to compare the power management of the UEFI and the board to supply components with power under load, and includes typical PSU losses due to efficiency.

We take the power delta difference between idle and load as our tested value, giving an indication of the power increase from the CPU when placed under stress.

The E5-2687W v3 is listed at 160W, which aside from a pair of AMD CPUs is the highest TDP for a CPU we have seen. Nevertheless, there are certain efficiencies exploited in the new platform and the v3 version of this CPU has a lower power delta than the v2 does, even with the higher TDP. Unfortunately due to limitations we were unable to measure power consumption while we had the E5-2650 v3 in for testing.

Overclocking...?

As per Intel's Xeon policy, the E5-26xx v3 processors are multiplier locked. For competitive overclockers, this is rather frustrating given that the Xeon processor line are often the better selected dies that also can pack a punch. So while multiplier overclocking is not possible, for motherboards with overclocking oriented BIOS options we can adjust the BCLK. While we never published the data at the time, the Ivy Bridge-EP processors we had in to test were good for 113 MHz (+13%), although 110 MHz had a good balance of overclock and stability.  

For this review, I put the E5-2687W v3 through its paces:

Moving up to 104 MHz is not a lot. It does afford some DRAM movement as well, but our system refused to POST at 105 MHz. This might purely be a result of the processor, so in our future Xeon reviews we will see if more movement is possible with other SKUs.

CPU Benchmarks

The dynamics of CPU Turbo modes, both Intel and AMD, can cause concern during environments with a variable threaded workload. There is also an added issue of the motherboard remaining consistent, depending on how the motherboard manufacturer wants to add in their own boosting technologies over the ones that Intel would prefer they used. In order to remain consistent, we implement an OS-level unique high performance mode on all the CPUs we test which should override any motherboard manufacturer performance mode.

HandBrake v0.9.9: link

For HandBrake, we take two videos (a 2h20 640x266 DVD rip and a 10min double UHD 3840x4320 animation short) and convert them to x264 format in an MP4 container. Results are given in terms of the frames per second processed, and HandBrake uses as many threads as possible.

Low quality conversion loves faster individual cores, hence the W processor wins out due to its higher full-load frequency. Nonetheless, the fast consumer grade processors win here by a large margin.

In full double-4K mode, the balance of cores, frequency and architecture upgrade puts the E5-2687W v3 above the 12-core E5-2697 v2.

Agisoft Photoscan – 2D to 3D Image Manipulation: link

Agisoft Photoscan creates 3D models from 2D images, a process which is very computationally expensive. The algorithm is split into four distinct phases, and different phases of the model reconstruction require either fast memory, fast IPC, more cores, or even OpenCL compute devices to hand. Agisoft supplied us with a special version of the software to script the process, where we take 50 images of a stately home and convert it into a medium quality model. This benchmark typically takes around 15-20 minutes on a high end PC on the CPU alone, with GPUs reducing the time.

Dolphin Benchmark: link

Many emulators are often bound by single thread CPU performance, and general reports tended to suggest that Haswell provided a significant boost to emulator performance. This benchmark runs a Wii program that raytraces a complex 3D scene inside the Dolphin Wii emulator. Performance on this benchmark is a good proxy of the speed of Dolphin CPU emulation, which is an intensive single core task using most aspects of a CPU. Results are given in minutes, where the Wii itself scores 17.53 minutes.

A single emulation instance benefits from a fast single core.

WinRAR 5.0.1: link

WinRAR seems to enjoy Haswell-EP over Ivy-EP, although it stills needs a high frequency to achieve top speeds.

PCMark8 v2 OpenCL

A new addition to our CPU testing suite is PCMark8 v2, where we test the Work 2.0 suite in OpenCL mode. 

Hybrid x265

Hybrid is a new benchmark, where we take a 4K 1500 frame video and convert it into an x265 format without audio. Results are given in frames per second.

Hybrid also takes advantage of the new architecture, giving a 5% advantage to the E5-2687W v3 despite two fewer cores.

Cinebench R15

3D Particle Movement

3DPM is a self-penned benchmark, taking basic 3D movement algorithms used in Brownian Motion simulations and testing them for speed. High floating point performance, MHz and IPC wins in the single thread version, whereas the multithread version has to handle the threads and loves more cores.

FastStone Image Viewer 4.9

FastStone is the program I use to perform quick or bulk actions on images, such as resizing, adjusting for color and cropping. In our test we take a series of 170 images in various sizes and formats and convert them all into 640x480 .gif files, maintaining the aspect ratio. FastStone does not use multithreading for this test, and results are given in seconds.

Web Benchmarks

General usability is a big factor of experience, especially as we move into the HTML5 era of web browsing. For our web benchmarks, we take four well known tests with Chrome 35 as a consistent browser.

Sunspider 1.0.2

Mozilla Kraken 1.1

WebXPRT

Google Octane v2

Gaming Benchmarks

F1 2013

First up is F1 2013 by Codemasters. I am a big Formula 1 fan in my spare time, and nothing makes me happier than carving up the field in a Caterham, waving to the Red Bulls as I drive by (because I play on easy and take shortcuts). F1 2013 uses the EGO Engine, and like other Codemasters games ends up being very playable on old hardware quite easily. In order to beef up the benchmark a bit, we devised the following scenario for the benchmark mode: one lap of Spa-Francorchamps in the heavy wet, the benchmark follows Jenson Button in the McLaren who starts on the grid in 22nd place, with the field made up of 11 Williams cars, 5 Marussia and 5 Caterham in that order. This puts emphasis on the CPU to handle the AI in the wet, and allows for a good amount of overtaking during the automated benchmark. We test at 1920x1080 on Ultra graphical settings.

Bioshock Infinite

Bioshock Infinite was Zero Punctuation’s Game of the Year for 2013, uses the Unreal Engine 3, and is designed to scale with both cores and graphical prowess. We test the benchmark using the Adrenaline benchmark tool and the Xtreme (1920x1080, Maximum) performance setting, noting down the average frame rates and the minimum frame rates.

Tomb Raider

The next benchmark in our test is Tomb Raider. Tomb Raider is an AMD optimized game, lauded for its use of TressFX creating dynamic hair to increase the immersion in game. Tomb Raider uses a modified version of the Crystal Engine, and enjoys raw horsepower. We test the benchmark using the Adrenaline benchmark tool and the Xtreme (1920x1080, Maximum) performance setting, noting down the average frame rates and the minimum frame rates.

Sleeping Dogs

Sleeping Dogs is a benchmarking wet dream – a highly complex benchmark that can bring the toughest setup and high resolutions down into single figures. Having an extreme SSAO setting can do that, but at the right settings Sleeping Dogs is highly playable and enjoyable. We run the basic benchmark program laid out in the Adrenaline benchmark tool, and the Xtreme (1920x1080, Maximum) performance setting, noting down the average frame rates and the minimum frame rates.

Battlefield 4

The EA/DICE series that has taken countless hours of my life away is back for another iteration, using the Frostbite 3 engine. AMD is also piling its resources into BF4 with the new Mantle API for developers, designed to cut the time required for the CPU to dispatch commands to the graphical sub-system. For our test we use the in-game benchmarking tools and record the frame time for the first ~70 seconds of the Tashgar single player mission, which is an on-rails generation of and rendering of objects and textures. We test at 1920x1080 at Ultra settings.

Intel Xeon E5-26xx v3 10 Cores Conclusion

Intel’s product stack for 2P capable CPUs is somewhat frustrating. The lower cost models always offer the best value for money, but getting a more expensive and faster CPU means that you end up with a faster unit. So if a user is buying purely on bang-for-buck, they might end up with a quad core.

The essence of the workstation is always centered on compute-limited throughput. I have mentioned this in a previous review – almost all computer usage can be split into idea-limited throughput or compute-limited. For the former, the user needs a faster brain, but for the latter a super-fast CPU is needed. Being able to get through a compute task even faster means the user is able to complete contracts quicker enabling more work and more money. Ultimately this means that if it can be justified in getting a higher core count processor, even at the expense of 100-200 MHz per thread, it might be worth investing in another $500.

In my previous existence requiring workstation CPUs, I was naïve and assumed that a 2P rig was the way to go – I even convinced my boss to invest in three for our simulation team. Our basic C++ simulations used threads, but no-one in the team understood about thread and cache management, let alone NUMA programming, because we were more chemists than computer scientists. I always encourage users to test their software on 1P and 2P workstations before convincing the people with the money to buy a machine – depending on the software, a big 1P system might have fewer cores but the cache management might increase throughput even more.

With this in mind, the Xeon E5 v3 workstation focused CPUs like the W range now sit in a more generalized form. Other CPUs, with more cores for 25% more in cost on paper might offer a 40% potential increase in throughput for less power. The E5-2687W v3 is a similar price to its last generation brethren, but the landscape around it has changed in favor of other processors – 160W is still a lot to take in, especially when 145W processors seem to offer more.  As a 10-core processor, the E5-2687W v3 still represents the best 10-core you can buy. But we have preliminary numbers in house for 12 core and 14 core CPUs, showing that a small increase in cost results in a better-than-cost increase in performance with lower power consumption.

The Xeon E5-2650 v3 represents part of the E5-2687W v3 problem. For just over half the price ($2057 vs. $1166), this CPU has two thirds of the TDP while only losing 500-600 MHz frequency across the power range. When paired up in a 2P system, two E5-2650 v3 CPUs against one E5-2687W v3 CPU will offer almost double the threads for only 13% extra CPU cost. For users that have software to take advantage of this, it makes a lot more sense while offering double the DRAM capabilities.

Over the coming weeks we hope to also supply reviews of 12 core and 14 core Xeon E5 v3 CPUs. Stay tuned for those! All our results will also be included in our CPU comparison section, Bench.