Is lvmcache Effective on a Desktop or Workstation?

| Comments

The answer you should take from me is that I don’t really know yet. At least, I don’t know just how well it is working because lvmcache is quite difficult to benchmark. The truth is that I’m not all that interested in benchmarks. I want to know how my experience of using my computer feels with a big, slow disk behind a fast NVMe cache.

What problem am I trying to solve?

Games load faster from an SSD. Scrubbing around in Davinci Resolve when there are many 4K videos on the timeline is smoother when reading from an SSD. I wish everything could fit on an SSD.

One of the two SSDs in my desktop started failing last month. I only had 1 TB of solid-state storage space, and it was getting really tight. I store most all my video on my NAS, and my NAS has a 100-GB lvmcache. It works great, and editing video over Infiniband with an lvmcache is very much like editing video on a local SSD.

Steam games are getting huge, and I am working toward eliminating the RAID in my NAS. I have a 14 TB hard drive off-site with a copy of all my data. The plan is to eventually stick a 14 TB drive in both my NAS and my desktop PC. I don’t feel the need to do this before the aging drives in my NAS fail, but the idea is on my mind.

How can I store several terabytes of data on my computer without buying a ridiculously expensive NVMe drive while still making sure everything feels like it is stored on an NVMe? I don’t want to feel that 200 IOPS of cheap spinning metal. I want to feel the 100,000 IOPS of a nice NVMe!

This experiment is my life now!

I bought a 1 TB Samsung 980 NVMe. I installed Ubuntu 22.04, left about 300 GB available for lvmcache, then stuck and old 4 TB 7200 RPM hard drive in my machine. This seemed like a good test to verify that lvmcache would be viable on my desktop.

Here’s where I’m at. My Ubuntu install and the data in my home directory are eating up only about 113 GB on the 1 TB Samsung NVMe. So far, I have installed 700 GB of Steam games on the 4 TB drive. I have also copied this year’s video files from my NAS to the 4 TB drive. That’s around 800 GB of video files.

Everything is encrypted, which does limit my throughput on the NVMe quite a bit, but I am still seeing 1.2 to 1.5 gigabytes per second. This is a topic for its own post.

What am I hoping to see?

lvmcache is a hotspot cache. It doesn’t just blindly cache every single read or write operation that happens on the slow disk. Things should only stick in cache if they are used regularly. The trouble is that I have absolutely no idea how lvmcache makes these decisions.

I could micromanage things. I could install one or two 100 GB Steam games on my NVMe, and when I am no longer playing those games, I could move them to the slow storage. I could copy the video files that I’m currently working on to the NVMe, then push them to slow storage when I am done.

I really don’t want to have to spend this much time managing where files live.

My hope is that if I play a game regularly, it will be promoted to the NVMe cache. My other hope is that as I am scrubbing around in the files for this month’s episodes of The Butter, What?! Show, that they will wind up being cached.

The best part is that I don’t need entire games or all of my video files to be cached. There’s a good chance that the early parts of a 150 GB game will start exiting the cache while the later parts of the game enter the cache as I slowly make progress. My video files from January will fade from cache, while this week’s recordings will be accessed quite often.

The cache sure seems to be doing its job!

I noticed the cache doing its job while I was testing to make sure my Nvidia drivers and fsync-enabled kernel were working well together. I kept loading “Sniper: Ghost Warrior Contracts.” The game plays a cut scene every time a level loads, and it lets you skip the rest of the scene when it is finished.

Each time I loaded the game, the message letting me know I could skip the cut scene was happening sooner. I fired up dstat, and I could see the game regularly reading at 70 megabytes per second from the slow drive and 150 megabytes per second from the cache drive!

As far as I was aware, it was letting me skip the cut scene for this level at precisely the same point as it would let me skip on my old Crucial M500 SSD. Could we go any faster?

I copied the game directly to my NVMe and fired it up again. The results were the same.

When a game loads, it isn’t just reading data from disk and plopping it into memory. It is doing work with that data. For this particular game, my CPU and GPU are definitely the bottleneck on load times.

This is definitely a good start.

Watching Davinci Resolve was fascinating!

I still have no idea how lvmcache decides what to keep and what to evict from cache. I do know that it tried to bypass the cache on long sequential writes, and that seemed to be confirmed when I plopped 120 GB of video footage from our last Butter, What?! Show live stream into place. If I were smarter, I would have watched dstat during the copy so I could better confirm this.

I was running dstat the next day when I started editing the footage in Davinci Resolve, and the results were really cool!

As soon as you drop a video on the timeline, Resolve reads through the entire thing to generate an audio waveform. You can see in the screenshot above that it was reading along at around 30 megabytes per second from the slow drive, and at the same time the NVMe cache was being written to. I most definitely didn’t expect this, and it is quite awesome!

Some of my multicam timelines get quite layered and convoluted. This might very well be because I have no idea what I am doing. The important thing is that there aren’t any hiccups or glitches. I don’t seem to ever be waiting for the disk to catch up to me.

I might split this into two caches

This is working well enough so far, but I have some concerns.

My cache is only 300 GB, and it is now normal for me to work with 120 GB to 180 GB of video at a time. It is currently very likely that I will be accidentally pushing games out of my cache once a month.

The lazy answer may just be to use a much bigger cache, and the 400 GB of unused space on my root filesystem suggests that might not be a bad idea. I am not confident that this will accomplish what I am hoping. Video files from two months ago have less value to me than a game I haven’t played in six months.

I’m not going to adjust anything while I’m still using this old 4 TB drive, but when I install a 14 TB drive, I am planning to split the drive and cache into two volumes.

I will have one volume for things that have long-term value to me. That would be the Steam games. The other volume would be for the data that I want cycling itself out of the cache.

I haven’t found another home for my 480 GB Crucial M4 SSD. Maybe I will use that for the video cache. An a7S3’s fanciest video modes only capture video at 150 megabits per second. My old SSD shouldn’t have much trouble streaming 10 of those files at a time, so it ought to be overkill for video editing.

It’d probably be better to steal 300 GB from my root partition, but using the SSD would be simpler.

As long as my NAS hard drives continue to not fail, I will have plenty of time to contemplate this.

How much room will I need for my root and home partition?

I am mostly guessing at everything. I figured 300 GB was a good size for a cache, and I set some aside for a big, useless swap partition. Whatever was left went to Ubuntu.

I am realizing now that big files are always going to wind up eventually being diverted to the slow disk. I most definitely could have set aside more space for cache. I probably should have.

I don’t think it would be a problem for me to use 80% of my 1 TB NVMe as cache, but it is nice to have some wiggle room if something winds up being slower than expected on the big drive.

NVMe vs. SSD

My computer doesn’t feel any different with the NVMe. It does boot significantly faster, but my old Ubuntu install that I’ve been upgrading since 2012 had a weird issue where it was waiting for something for 30 seconds while setting up my Infiniband link. Rebooting only happens every few weeks, so I never got around to troubleshooting that problem. Let’s just say that anything would feel like it boots faster when you eliminate that huge delay!

I’m sure that if you handed me a stopwatch, I could measure that some programs open faster. I know that copying files around will be faster, but it wasn’t exactly slow before.

Upgrading from a 7200 RPM hard disk that averages 150 IOPS to my first SSD that pulled 1,500 IOPS made my computer feel like a totally different machine. Upgrading from my Crucial M4 that could do 5,000 IOPS to the Samsung 980 that can do 100,000 IOPS was nice, but not terribly exciting.

That said, don’t buy a SATA SSD today. If you have an available NVMe slot, fill it with a decent NVMe. NVMe drives that can manage 3 gigabytes per second don’t cost more than SATA SSDs.

If I am only pulling 150 MB/s from the cache, do I need an NVMe?

No. For most use cases on your workstation, you probably don’t. Like I said, though, don’t buy a SATA SSD unless you don’t have a slot available for an NVMe.

Even if you set up identical tests, fire up dstat, and you see that your SATA SSD or NVMe cache read for similar amounts of time, and they both top out at around the same throughput, you still might speed up your workload with the NVMe.

Imagine that your task quickly reads 150 megabytes, does some work for a few seconds, reads another 150 megabytes, does some more work for several seconds, and this repeats over and over again. It’ll take the SATA SSD about 500 ms to pull that data, while your stupidly fast 7-gigabyte-per-second NVMe might be able to do it in 40 ms.

What’s next?

It is a bummer that lvmcache is difficult to benchmark. It is meant to fill up and settle in over longer periods of time. That’s OK by me because I want a cache that will learn what I really need cached over time, and I hope it works out well.

The plan is to sit tight for now. I am just going to do the things I usually do while the cache settles in. Once that happens, I plan to make a small tweak to the fantastic lvmcache-statistics script. It shows cache hit and miss rates since boot.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

-------------------------------------------------------------------------
LVM [2.03.11(2)] cache report of given device /dev/mapper/zaphodvg-slow

Cache hit rate since:  Fri Apr 22 08:12:26 AM CDT 2022
Current time:          Fri Apr 22 08:13:16 AM CDT 2022
-------------------------------------------------------------------------
- Cache Usage: 99.9% - Metadata Usage: 6.6%
- Read Hit Rate: 99.9% - Write Hit Rate: 99.9%
- Demotions/Promotions/Dirty: 16926/18475/0
- Feature arguments in use: metadata2 writeback no_discard_passdown 
- Core arguments in use : migration_threshold 8192 smq 0 
  - Cache Policy: stochastic multiqueue (smq)
- Cache Metadata Mode: rw
- MetaData Operation Health: ok

I am going to tweak it to show hits and misses since starting the script. That will let me drop RAM caches and fire up a game or edit a video and see what the cache is up to since starting the new task.

I will also be able to run a big game, make a note of the lvmcache-statistics output, then do my very best to ruin the cache with hundreds of gigabytes of fresh video files. Then we can see how much of the game managed to stay in the NVMe cache.

Conclusion

I am excited. I am definitely not noticing just how slow that 4 TB drive actually is, which is a huge win as far as I am concerned. This means I will be able to drop a slow 14 TB in here, and I won’t have to worry about managing which files live on the fast or slow storage. That is awesome!

What do you think? Are you using lvmcache on your desktop? Are you using another cache like bcache? How is it working out for you? Let me know in the comments, or stop by the Butter, What?! Discord server to chat with me about it!

Comments