Using lvmcache for Gaming and Video Editing - What I Have Learned So Far

| Comments

I’ve been running and living with my lvmcache experiment on my desktop for more than a month now. Everything is working out about as well as I had hoped, and that means they’re doing even better than I truly expected. In fact, things are sometimes performing better than the measurements would suggest that they should. Doesn’t that seem counterintuitive?

Why use lvmcache on your desktop or workstation?

SSDs and NVMe drives can be expensive. Not only that, but they only get so large. NVMe pricing at up to 2 TB is pretty good, then you wind up paying bit of a premium to get to 4 TB. If you need more storage, it will either cost a lot more, or you need to buy multiple drives. Then you need more m.2 slots.

I decided to put a fast lvmcache in front of a slow hard drive. I bought a 1 TB Samsung 980 NVMe for $100. I set aside 300 GB of that to use as an lvmcache in front of a 4 TB 7200 RPM hard drive I already had on hand.

NOTE: Because so much of my bulky data is stored on the slow disk under the lvmcache, I am only using 142 GB on my root/home volume on the NVMe. Since large files are always going to live in the cache, I don’t expect to use much more space. It would have been safe to set aside 700 GB for caching, and I wish I did that!

The hope has been that my games would load faster, and my video editing experience in Davinci Resolve would still be smooth even when the files are stored on the slow hard drive. I don’t want to have to move large games to a bigger disk. I don’t want to shuffle around games that I am not playing often any longer. I don’t want to have to move my current Resolve project to the NVMe and move it off when I am finished.

I want lvmcache to handle the shuffling for me.

lvmcache is impossible to benchmark

That is probably an exaggeration, but it is definitely difficult to benchmark lvmcache well. Everything you read or write passes through Linux’s in-memory disk cache. That is not how lvmcache works.

Since lvmcache is a hot-spot cache, it tries to capture data that is accessed frequently, and it tends to skip caching sequential writes. That means you might dump 50 gigabytes of video footage from your camera, and it is possible that not a single byte will wind up in cache. You may not even edit that footage for a few days, so it definitely won’t wind up in the cache right away.

How do you benchmark something like this?

I don’t really want a benchmark. I just want to get a glimpse of what the cache is doing for me, and I want to see if I can measure any improvements or see if there are any delays.

I tweaked the awesome lvmcache-statistics script so that it works more like top. That will let me monitor cache hit and miss rates. I’ve been running that alongside dstat to watch how much data is moving around on each of the physical disks. This has definitely been interesting!

Game loading tests

Games are a huge part of why I want to use lvmcache on my desktop. I have so many games that are each between 50 and 150 gigabytes. I don’t hop between games. I am usually playing one single player and one multiplayer game during any given week or month. Most of this can drop out of cache, and it would be fine.

We’ll talk about how this might relate to cache sizing later on.

I already talked about this in the first lvmcache post last month, but I’ll summarize it again here. The first thing I tested was firing up Sniper Ghost Warrior Contracts on my new lvmcache setup. I wasn’t smart enough to use a stopwatch, but the game plays a cut scene while loading, and you can’t skip the cut scene until loading is complete.

The first time I started the game, I got to see way more of the cut scene than I ever did when playing the game on my SATA SSD. I quit the game, dropped my RAM caches, and tried again. This time it loaded faster. I repeated, and it was precisely as fast as loading from my old SSD.

I wasn’t seeing any numbers higher than around 150 megabytes per second on dstat, so I copied the game directly to my NVMe and tried again, and it loaded in precisely the same amount of time. I wanted to make sure lvmcache wasn’t a bottleneck.

The load time was about the same direct from the NVMe.

Before writing this blog today, I figured I should load the same game today. Steam says I haven’t played it in about six weeks. I’ve dumped hundreds of gigabytes of video since then, and I have edited two Create/Invent Podcast interviews and two Butter, What?! Show live streams.

I fired up Sniper Ghost Warrior Contracts and watched the dstat output. I was excited to see that there were rather long stretches of time when only the NVMe was being touched while on the way to the main menu. When I loaded my progress, though, things slowed down.

Yes, data was being pulled from the cache, but not much. It didn’t take all that much longer to load. If I were smarter, I would have used a stopwatch! It was probably an extra 10 or 15 seconds.

Really not that bad sounding, right? And I am confident that it will go faster if I decided to play again tomorrow, because dstat showed plenty of writes to the NVMe during the load, so more data was being cached.

Video editing is both awesome and interesting

We record one live stream and one podcast interview every month. My live streams with Brian Moses generate a large volume of video. We each record ourselves on our Sony cameras, and I record my desktop using OBS Studio. It is easy for us to break 100 GB of footage in 60 to 90 minutes. The Create/Invent Podcast interviews are recorded using Riverside.fm, so the video is more heavily compressed but still of quite good quality.

When I dump video off the cameras, I don’t see much lvmcache action going on. As soon as I add them to a timeline in Davinci Resolve, things start to get interesting. Resolve reads through every video file to generate an audio waveform to display on the timeline. During this part of the process I see reads on the slow disk and a similar amount of writes to the NVMe!

That means it is mostly ignoring the large video files when I dump them to disk. This is either because lvmcache doesn’t think a single write is worth promoting to the cache, or it doesn’t bother caching the writes because the slow disk is plenty fast enough to keep up with my SD card. It is probably some combination of both.

As soon as Davinci Resolve starts scanning those video files, the lvmcache will start getting primed. Isn’t that cool?!

What kind of disk performance do you need to edit video?

The answer to this question will be different for everyone. It will depend on how many videos you have playing simultaneously on your timelines and the bitrate of those videos. The highest bitrate videos that I edit 90% of the time are the 50-megabit files from my Sony ZV-1. Sometimes I edit 100-megabit and 150-megabit files from a Sony a7S3.

You really don’t need much disk performance to playback video files like these. My new 2-in-1 ultrabook laptop can composite three of my Sony ZV-1 files stored on a Samba share over WiFi on the on the same timeline and play them back without dropping any frames. Things get hairy there when you want to scrub through the timeline quickly, but it can handle basic editing tasks this way in a pinch.

The quickest and easiest test I could think of doing was playing back one of my videos faster and faster to watch how much I/O bandwidth would be required. I learned that playing back some footage of myself at 8-times normal speed only eats up 50 or 60 megabytes per second of bandwidth. I could have done math to figure this out, but doing it in real life also showed me that I seem to start dropping frames at 16-times or 32-times playback speed.

NOTE: Knowing this number is handy for me, because it tells me that my spare SATA SSD can outpace my CPU and GPU by nearly an order of magnitude.

My slow hard disk has at least three times more throughput than this. It could almost definitely keep up if there were two of these videos on a timeline, and it might be able to manage with three, but I am not confident of that. Video playback is very sequential disk access, and that is something spinning media is good at, but Resolve is going to be requesting blocks from three different video files. Those seeks will slow things down!

I had a chance to do a more realistic test this week with fresh footage from the Create/Invent Podcast, but disk performance isn’t my problem when editing podcast episodes. I wind up building stacks of timelines on top of timelines when I set up out various split-screen views. When I put those in a multicam bin, the multicam preview eats up CPU and GPU.

The important thing for me is that I had a chance to see first hand that my lvmcache would easily be up for the job.

This really simple playback speed test that I have been doing has been a handy yardstick for understanding how much disk throughput I need to have. I know how fast my 5 GHz 802.11ac WiFi is in my office, so it isn’t much of a surprise that my laptop can playback three video files on the same timeline in Resolve. I also isn’t a surprise that my CPU and GPU are the bottleneck when trying to play video at 32-times speed. As I alluded to earlier, throughput isn’t always enough.

If your video files are stored on a slow network share or a 7200-RPM disk you will certainly notice a slight delay every time you click a new point on your timeline, while on a solid-state drive you will be able to jump around instantly.

Is one cache enough? How big does your cache need to be?

This is another part of the puzzle where I have more questions than answers. Let’s walk through some of what I have been pondering. Maybe my needs will help you compute what is required to work for your own situation.

I have a 300 GB lvmcache volume on my 1 TB NVMe. Hindsight tells me I should have at least doubled that, and I most definitely should have realized that when I set this up.

I process between 100 and 150 gigabytes of video each month. That means I pull the files off of cameras, spend a week or two editing those files in Davinci Resolve, export 10 or 20 GB for YouTube, then I probably won’t touch those files again.

I have games that range in size from 20 gigabytes all the way up to 140 gigabytes. I really only need to fit two or three of those on the SSD cache at any given time.

I don’t have enough cache space to hold much more than one or two games and a month’s worth of video. Sure, the cache doesn’t have to hold every single piece of a 150 gigabyte game to be useful, but I have already seen that editing two live streams and podcasts in six weeks managed to push most of a 50 gigabyte game out of my cache.

Part of the problem here is that I don’t understand how lvmcache makes it decisions, and I don’t have any way to influence them. The way I see it, I have two choices.

I could use a bigger cache. That would give Team Fortess 2 and Sniper Ghost Warrior Contracts more time to show lvmcache that they need to be cached, and lvmcache would have more time to notice that I haven’t touched the video files from March in the last two months.

The other option is to split my data and use two caches. The video files are ephemeral. I am going to work on them for a few weeks and never touch them again. If I do have to revisit those files, they can weasel their way back into the cache. The video storage volume is almost like a scratch drive. I imagine that I only have to make sure the cache on my scratch volume can hold at least one project’s worth of data. Everything else could be devoted to the long-term volume’s cache.

I am leaning towards trying out the second option. I fear that I will outgrow a single large cache and just wind up right back where I am today, and adding a second cache will require a lot less work. I still have my 480 GB Crucial SATA SSD installed. It is more than fast enough to cache the big disk full of video files!

My current lvmcache configuration is still quite good!

Is it really a big deal that video editing will push Borderlands 3 out of my lvmcache? I will just have to sit through a longer loading time once or twice. That seems like a small price to pay to save me from spending time micromanaging which games live on the NVMe and which games live on slow storage.

My setup doesn’t have to be perfect to be fantastic. Neither does yours!

You need to have good backups!

I don’t enjoy writing this section. It is always important to have good backups. When relying on a mechanical drive, it is even more important to have good backups. The statistics from the most recent Backblaze report say that they have an average annual failure rate of 1.22%, while some models of drive fail three or four times more often.

Yes. Solid-state drives can also fail, but they don’t fail as often. Mechanical disks sure seem to be doing much better than they were a decade ago, but they have lots of moving parts.

Seafile will start uploading my huge video files within moments of copying them off my camera. I am pretty safe as long as I don’t immediately wipe the SD card.

Why not store /root and /home on the slow disk?

I would like my computer still be useful when my slow drive inevitably fails. I wouldn’t trust a fresh spinning drive to last, and I most definitely don’t expect the six-year-old 4 TB drive I am using today to last. I wouldn’t be surprised if it fails tomorrow!

I am only using 130 gigabytes on my NVMe. I don’t expect that to grow significantly.

I wish I had more control over lvmcache

I keep thinking about how I would like to be able to tune or give hints to my lvmcache. It would be nice if there was an equivalent to the handy eatmydata tool that ignore the flush command.

It’d be neat to be able to run a command like dontcachethis cp /media/sdcard/video*.mp4 /lvmcached/area/over/here to make sure I don’t pollute the cache with an obviously large and useless write. Maybe there could be a similar tool to inform the cache that what I am about to do is of high value and should be given more priority in the cache.

Then I thought about it some more. Using lvmcache isn’t really a solution. It is a Band-Aid. I am using lvmcache because 8, 16, or 32 terabyte NVMe drives are too expensive or just don’t exist yet. In five years this may be completely irrelevant to almost everyone.

I don’t need better tools for managing my lvmcache. I just need my lvmcache to work until I outgrow it!

Conclusion

I am extremely pleased with my lvmcache setup on my desktop. The games I play load quickly. My video editing workflow is smooth. Best of all, I don’t have to shuffles files around between my NAS, my slow local hard drive, and my local NVMe. For the most part, everything just works!

This is one of the final steps on the road to eliminating my NAS. I am just waiting for the old 4 TB hard disk in my desktop or one of the 4 TB hard disks in my server to fail. Eventually each will have a 14 TB drive of its own, and my RAID and my Infiniband network will go away. I could save $100 or more if my aging disks can survive long enough before I am forced into an upgrade!

What do you think? Are you planning to use lvmcache on your workstation? Are you already using lvmcache to solve the same problem? Or are you using lvmcache on your servers? Let me know in the comments, or stop by the Butter, What?! Discord server to chat with me about it!

Comments