Do You Need to Buy The Fastest NVMe?

| Comments

Do you want the easy answer? No! You almost definitely do not need the fastest NVMe available. Most of us probably won’t even notice the difference between the slowest and the fastest NVMe drives.

NOTE: The XPG Gammix S70 isn’t literally the absolute fastest NVMe available, but it is definitely very near the top of list, and it is the super-fast drive I most often see good deals on. Even if you manage to exhaust the S70’s large write cache in one go, it is still quite fast, and it often goes on sale for $100 per terabyte.

I am also absolutely certain that there is someone out here with a very particular use case that would truly benefit from 7 GB per second reads or writes. Most of us don’t even have software that can keep that up for more than a fraction of a second.

I don’t have a budget! I am just going to buy the fastest thing!

If you truly have no budget, then you should absolutely buy what makes you happy. Most of us who say we don’t have a budget are still making choices based on price.

The price-to-performance graph for any piece of hardware in your computer tends to look like a hockey stick. The price of a component usually increases pretty linearly from the low-end to very nearly the high end, but it usually takes a sharp turn about 80% or 90% of the way to the end of the graph. You might have to pay three times as much to go from 80% to the very top of the performance graph.

This may not even be worth writing about because the fastest NVMe drives only cost twice as much as the no-name cheap drives. The cheapest no-name NVMe deal I’ve seen so far was $55 per terabyte, while some of the fastest NVMe drives go on sale for around $110 per terabyte. The middle-of-the-road drives with good warranties from reputable manufacturers are usually between $80 and $90 per terabyte.

I wonder how much cheaper these will while you are reading this in the future?!

This isn’t anywhere near as big a jump as going from the biggest Ryzen CPU to the smallest Threadripper. Even so, if my words mean you can move $60 from your NVMe to a slightly faster CPU or GPU, then it was worth my time!

I can’t max out my lower-end Samsung 980 NVMe

I can run a benchmark or spam some zeroes over the drive with dd and hit several gigabytes per second. I am running LUKS on top of my NVMe, and that layer of AES encryption seems to have me capped out at around 1.6 gigabytes per second. I haven’t found a use case that will register anywhere near that much bandwidth while monitoring with dstat.

I can hit these big numbers if I copy a huge directory of files from the NVMe to itself. This isn’t something most people do all that often.

The Internet says my drive will run out of write cache if I can write 300 gigabytes as fast as the drive can write. I don’t have any external sources that can supply data that fast. In practice, my 40-gigabit Infiniband network tops out at 13 gigabits per second because it is limited by my desktop computer’s PCIe slots. That’s roughly as fast as my encryption can go, but the drives on my server can only sustain about 60% of that under the very best conditions.

The most data I ever dump onto my computer comes from my cameras. It is normal for me to have one or two nearly full 64 GB microsd cards after filming. This could potentially fill up 1/3 of my Samsung NVMe’s write cache, but those cards only read at about 20 megabytes per second.

I edit lots of video, but that never needs more than a 100 or 150 megabytes per second of disk bandwidth.

I’ve been monitoring game loading times from my lvmcache. I haven’t found a game that has a bottleneck on disk operations, and I have yet to see a number higher than 180 megabytes per second in dstat while loading a game or new level in a game.

It is nice that my NVMe can manage hundreds of thousands of IOPS. That’s at least ten times more than my old SATA SSD, but my usual tasks don’t go any faster after my upgrade.

dstat doesn’t tell the whole story

Just because I am only seeing 180 megabytes per second in dstat doesn’t mean that I’m not benefiting from the 1.6 gigabyte or more than my NVMe is capable of providing. dstat is giving me a snapshot of my throughput at intervals of one second.

During that full second, whatever game was loading had read 180 megabytes from the disk. Odds are that this happened in a little over 100 milliseconds. My old SATA SSD would have also read 180 megabytes during that same second, but it would have taken nearly 500 milliseconds.

This improved latency is nice, and if software is blocked while waiting for that data, then hundreds of milliseconds saved here and there would add up to actual seconds. Something that took 20 seconds to load on the SATA SSD might now take 17 seconds.

The game loading times that I have managed to check don’t show such improvements. These games are likely still busy computing something important while waiting for more data.

Conclusion

I am certain that some of you reading this will actually benefit from a top-of-the-line NVMe. There are most definitely workflows out there can benefit from 7 gigabyte per second reads and writes. I haven’t run into one myself yet, and I’d bet that the majority of you won’t either.

When I upgraded from a mechanical disk that topped out at 200 IOPS to the ancient Intel X25-m with its 5,000 IOPS it was an absolute game changer. Everything seemed to load from disk instantaneously. Upgrading to the next SATA SSD with 50,000 IOPS didn’t feel much different, and neither does this NVMe with 500,000 IOPS.

We need some pretty serious changes in our hardware, operating systems, and software to really take advantage of the difference between 50,000 and 500,000 IOPS. Until then, we can definitely save a few bucks by skipping the upgrades to the fastest NVMe drives on the market.

Comments