Six Months of lvmcache on My Desktop

| Comments

I admit it. It hasn’t quite been six full months since I put a fresh NVMe into my desktop machine and turned on lvmcache. I am nearly three weeks short of that date as I am writing this sentence, but it will probably be another week before I finish this blog, and if I wait any longer I might miss the target by a few months!

I believe I only have good news to report. I’ve torn down, rebuilt, or reconfigured the cache at least three times: once when I installed the NVMe drive, once when I split my slow storage volume into two pieces, and again when I replaced the ancient 4 TB drive with a 12 TB drive.

Here’s the tl;dr!

The cache is fantastic. It works well enough to cache all the games that I play, and they load just as fast as they would if they were installed directly on the NVMe.

I am no longer the least bit concerned about wearing out my flash storage. It sure looks like I won’t run out of write endurance until 10 years after Samsung’s 5-year warranty expires.

I split my slow storage into two separately cached volumes

I did some math a few weeks after setting up my lvmcache. My lvmcache partition on the NVMe is 300 GB, and I process around 200 GB of video files each month. That much is just fine.

Quite a few of my Steam games are over 100 GB in size.

Testing says that the video files I am working on do indeed wind up nearly 100% cached. If we oversimplify the way lvmcache works, and we assume that the cache will be smart enough to always evict the older video files that I won’t be working on in the near future, this only leaves me enough room in cache for a single game.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

-------------------------------------------------------------------------
LVM [2.03.11(2)] cache report of given device /dev/mapper/zaphodvg-slow
-------------------------------------------------------------------------
- Cache Usage: 99.9% - Metadata Usage: 6.6%
- Read Hit Rate: 66.2% - Write Hit Rate: 56.7%
- Demotions/Promotions/Dirty: 27129/27165/0
- Feature arguments in use: metadata2 writeback no_discard_passdown
- Core arguments in use : migration_threshold 8192 smq 0
  - Cache Policy: stochastic multiqueue (smq)
- Cache Metadata Mode: rw
- MetaData Operation Health: ok

-------------------------------------------------------------------------
LVM [2.03.11(2)] cache report of given device /dev/mapper/zaphodvg-churn
-------------------------------------------------------------------------
- Cache Usage: 85.2% - Metadata Usage: 24.4%
- Read Hit Rate: 50.1% - Write Hit Rate: 3.1%
- Demotions/Promotions/Dirty: 0/183205/0
- Feature arguments in use: metadata2 writethrough no_discard_passdown
- Core arguments in use : migration_threshold 8192 smq 0
  - Cache Policy: stochastic multiqueue (smq)
- Cache Metadata Mode: rw
- MetaData Operation Health: ok

The math says I should have used 300 GB for my operating system and 700 GB for the cache. Resizing the encrypted root filesystem and juggling everything around felt like too much effort, so I just set up a separate cache on my old Crucial 480 GB SATA SSD.

I call my cached volumes slow and churn

I probably need better names. The slow volume’s name has been grandfathered in, but the churn volume’s name is rather appropriate.

One of those volumes is where I churn through data. I dump 100 GB of video onto that volume at a time, work on it for a few weeks, then I dump another 100 GB of video. This just keeps happening every few weeks. This sounds a bit like churning, doesn’t it?!

The churn volume is an 8 TB slice of my new 12 TB hard drive, and it is cached by the old 480 GB SSD. That SSD is plenty fast enough to handle the 50 and 100 megabit files our Sony ZV-1 and Sony A7S3 cameras create.

The slow volume got its name because it just isn’t fast like the NVMe. This is where my Steam library lives. I have installed just over 2.3 TB of games that are being cached by a 300 GB partition on my 1 TB NVMe.

Is my mid-range NVMe going to survive being a cache?!

I was going to tell you that it depends on who you ask, but both the conservative and pessimistic answers are both positive!

My 1 TB Samsung 980 has a 5-year warranty with a guarantee of 320 TB of written data. I am right around 10 TB of writes after six months. That means Samsung thinks I will make it 10 years past their warranty period.

1
2
3
4
5

Percentage Used:                    1%
Data Units Read:                    22,669,873 [11.6 TB]
Data Units Written:                 19,392,238 [9.92 TB]
Host Read Commands:                 188,764,027
Host Write Commands:                308,581,800

The data in the SMART report says I have only used 1% of my writes. If that’s correct, then this NVMe will outlive me.

I assume that my writes have slowed considerably. I had to drop the lvmcache every time I resized my slow and churn volumes. That means 1 TB of those writes had to happen just to refill the cache.

I’m also no longer passing 300 GB of video files through the NVMe’s cache partition every month. My old Crucial SSD is bearing that weight now.

That little old SSD is doing a good job. It spend eight years being the primary storage device in my desktop computer, and SMART says it has 33% of its life remaining. The data sheet says the Crucial SSD is rated for 72 TB of writes, so it will probably make it through the next couple of years!

The conundrum of two caches

Is it better for me to have two separate caches? Would I be better off with one 800 GB cache instead of a 300 GB cache and a 480 GB cache? It is complicated, and I just can’t make up my mind about how I feel about this.

I do know for certain that I would much rather have both these caches on my NVMe. If they were on the NVMe the proportions would be adjusted.

On one hand, dealing with a single storage volume can be much more convenient. When I installed my 12 TB hard drive, I had to decide how much space I needed for my Steam volume and how much space I would need for video files.

If I made the wrong choices, I will have to shrink one volume and extend the other in a year or two. I will have to disable and recreate both lvmcache caches to make that happen.

On the other hand, having two different caches handling two different kinds of data is a much more effective use of cache space. My Steam games that I play regularly tend to just stay put in the cache, and it doesn’t matter how long the older videos stay in their separate cache, because they won’t be pushing games out!

If I had one unified cache I bet it would probably take a month or more for old videos to get demoted. It wouldn’t surprise me if that means I’d have 300 GB of unnecessary video that I’ve already finished editing clogging up my cache at any given time.

One could argue that I could have sidestepped that problem by buying a 2 TB NVMe and using a bigger cache, but that doesn’t eliminate the issue. It just makes it a lot smaller, right? Besides, the goal was to save money by buying less flash storage!

I’m not running Linux! Is there some way I can do this on Windows?!

Yes. Maybe. Most likely.

My friend Brian Moses has been watching me talk about lvmcache for ages, and he’s been watching me post screenshots full of cache data for just as long on our Discord server. When he built his new gaming PC, he did some research and wound up buying a copy of Primocache.

I don’t think he’s run much in the way of benchmarks, and if he has, he hasn’t posted the results of this tests. I asked Google about primocache for gaming and the first hit is Leonard Putra’s video showing side-by-side footage of a few games loading with and without Primocache.

Primocache seems to be doing the job for Leonard. Three out of four games had a 98% cache hit rate. Just Cause 4 had a slightly lower hit rate, and it didn’t load much faster.

Some games just don’t benefit from faster disks. Most of the games I tested load just quickly from my SATA SSD as they do from my much faster NVMe.

I have no first-hand experience with Primocache, but it certainly looks like it is worth checking out.

Conclusion

This sort of caching is only a Band-Aid. In a five years we will likely have more NVMe storage than we know what do with.

In the mean time, I am excited to have lvmcache available in the mean time. I only have a 1 terabyte NVMe, but I have 2.4 terabytes of games installed. How awesome is that?!

Are you thinking about using a solid-state disk cache in front of a slow disk on your desktop or workstation? Are you already caching your workstation with lvmcache or something similar? How is it working out for you? Let me know in the comments, or stop by the Butter, What?! Discord server to chat with me about it!

Comments