Every so often we are asked for more behind-the-scenes information on how we test products. So now that we've completed some updates to our SSD testing workflow, we're going to do just that, and give you a quick peek into our 2019 SSD testing setup.

As a bit of background to start things off, since December 2017, we have been using a Quarch XLC Programmable Power Module (PPM) to supply and measure power to SSDs during most of our benchmark runs. The XLC module is Quarch's second-generation power module, and while capable, it does have a few limitations as far as our SSD testing goes. Most of which, as it turns out, have been addressed in the XLC's successor, the third-generation HD Programmable Power Module.

So Quarch in turn has sent one of these new power modules our way, which we're using to upgrade our SSD testbed and refine our workflow. The newer power module allows us to take more accurate measurements of M.2 SSD power draw and gives us more flexibility in test configurations.

At a high level, the purpose of using a programmable power module is to allow us to record the power consumption of SSDs in great detail, with the power module feeding any SSDs we're testing. Complicating matters just a tad, these power modules are a bit larger than a standard 5.25" optical drive and require their own AC wall power input, so they don't fit easily inside any of the systems we use for SSD testing. Instead, they feed power to SSDs through cables that connect to any of several power injection fixtures—Quarch provides fixtures for almost all SSD form factors and connectors, including the very new EDSFF connector. These power injection fixtures pass through any data signals untouched, but connect the SSD's power supply lines to the Quarch power module instead of the host computer's power supply.

When it first launched, the HD Programmable Power Module added many convenience features over the XLC, but the core power supply and measurement capabilities we rely on were basically the same. Both modules provide two DC power rails: a 12V rail and one that can be configured for 3.3V or 5V depending on the type of drive being tested.

The power modules have several analog to digital converters (ADCs) to simultaneously measure current and voltage on both rails, and there's an extra low-range mode for the current measurements that allows for accurate measurement of sub-milliamp current drawn by SSDs in extremely low-power idle states. The ADCs are sampling at 250kHz, so the power modules can record very short-lived spikes in power draw. This for example means that when testing a mechanical hard drive, it is easy to observe individual seek operations separately from the read or write operations that occur after the heads are in the right place. The measurement resolution and sampling rate both fall somewhere between the typical values for bench multimeters and oscilloscopes, and for this specialized use case a Quarch programmable power module is able to replace a very large pile of traditional electronics lab equipment.

Last year, Quarch developed a hardware upgrade for the HD PPM to address a major limitation: When testing  M.2 SSDs, the power module has to supply 3.3V over a cable that's about two feet long. High-power M.2 SSDs can draw in excess of 3A of current, which leads to significant voltage drop along the cable—and SSDs typically only tolerate about a 5% drop below nominal voltage before they stop working. The power modules allow their output rails to be set above nominal voltage, but some M.2 PCIe SSDs still pushed the limits of the XLC module's current delivery capabilities. Quarch has now upgraded the HD module to support 5A continuous and up to 9.2A for short term (10 seconds). More importantly, the HD PPM now automatically adjusts its output voltage to compensate for voltage drop along the cable, so that it regulates the programed voltage measured at the drive instead of at the module's output connector.

200mV drop on 3.3V supply with XLC PPM while writing 16GB test file

With the XLC power module, we were unable to feed high-performance M.2 PCIe SSDs 3.3V directly without running into problems, and instead we have been conducting performance testing with the drives mounted in a PCIe to M.2 adapter card that does its own 12V to 3.3V regulation. That adds some overhead to our power measurements, but this is easy to correct for after the fact. For idle power testing, voltage drop isn't a problem and we have always been using a direct 3.3V supply. The upgraded capabilities of the HD power module allow us to start using a direct 3.3V supply even during performance testing, and to use Quarch's M.2 power injection fixture to connect drives through the host system's M.2 slots instead of just through full-size PCIe card slots.

Powering M.2 SATA or PCIe SSDs with M.2 injection fixture

The net impact of all of this is that we're now able to measure M.2 SSD power draw in a laptop computer (albeit with some extra cables hanging out the bottom). So, for example, we can get proper power measurements from the Optane Memory H10 that isn't supported by any of our current desktop testbeds. We can also now measure power draw with the deepest PCIe link state power management modes that are not usable on most desktop systems. This gets most recent M.2 NVMe SSDs well below 10mW at idle, compared to the 50+mW that is the best most NVMe SSDs can achieve on our usual Skylake desktop test system. We will be re-testing idle power for all M.2 SSDs in our collection and will start reporting three sets of idle power measurements:

• Active idle with all NVMe and PCIe power management features disabled, as is typical for most desktop systems
• Desktop idle representing the best idle power achievable on most desktop systems with appropriate BIOS and operating system settings configured to save power
• Laptop idle representing the best-case that can be expected from modern laptops (and the occasional desktop with the right settings)

We still don't have the ability to test DevSleep idle power for SATA drives, which is the closest equivalent to the laptop idle that we are now measuring for M.2 PCIe SSDs. DevSleep is generally impossible on desktops because it requires signaling through the SATA power connector rather than the data connector, and the HP Spectre x360 13t notebook we are using with the Optane Memory H10 is an NVMe-only system. The notebook market in general has reached a tipping point and OEMs are rapidly switching from SATA to NVMe, so SATA DevSleep is becoming less relevant.

Going forward, all new M.2 SSDs will be tested with the HD Programmable Power Module for both idle and performance benchmarks. We will re-run performance testing for older M.2 drives as time permits, after updating all of the idle power measurements. On the software side, nothing is changing with our test procedures, and we will still be using the same Skylake desktop system for most of our tests except when testing drives that are not supported on that system (eg. the Optane Memory H10). We are currently not planning to make major changes to our SSD test procedures or host systems until it's time to move to PCIe 4.0.

Preliminary Results

Below are some of our first idle power measurements conducted with the new setup using the HD PPM and the M.2 power injection fixture to feed a regulated 3.3V supply from the power module to the drive under test. The host system for these measurements is our Coffee Lake desktop, which is the only desktop SSD testbed in our collection that has fully functional PCIe Active State Power Management (ASPM). This allows us to use one system to represent both the Desktop and Laptop idle power scenarios by varying OS settings.

The desktop idle power numbers are generally consistent with our previous measurements conducted with the XLC power module and our Skylake desktop. Several of the active idle power numbers are significantly lower than previously measured, so we'll be double-checking that our scripts really are successfully disabling all power management features.

Some NVMe controllers have very odd behaviors when it comes to toggling the Autonomous Power State Transitions feature on or off, such as requiring new transition latency data to be provided even when turning the feature off, or requiring some real disk IO to wake the drive up before it will actually stop using the now-disabled idle states. Even when the host system's power management bugs have been resolved, NVMe power management continues to be quite the minefield. Directly measuring SSD power consumption is the only reliable way to determine which settings have actually taken effect.

Our new Laptop idle power numbers give us the first measurements that actually match what most of the spec sheets for these drives promise: single-digit milliwatt consumption for most recent controllers. The outliers so far are the MyDigitalSSD SBX with the entry-level Phison E8 controller, and the Gigabyte Aorus SSD with RGB lighting.

Now that we're measuring two levels of sleep for NVMe drives, we need to measure two different wake-up latencies. For the drives based on the Phison E12 controller, wake-up from the Desktop idle settings is much faster than from the Laptop idle settings. However, the rest of the drives we've tested so far show similar wake-up latencies for both scenarios despite drastic differences in actual power consumption while sleeping.