KVM Dirty log ring interface #344

davidkleymann · 2025-09-03T09:33:10Z

Summary of the PR

Add support for dirty page tracking via the Dirty ring interface. Adds KvmDirtyLogRing structure for keeping track of the indices and the base pointer to the shared memory buffer. Implements iterating over dirty pages, thereby harvesting them. Implements reset_dirty_rings on VmFd to trigger recycling of dirty ring buffer elements by the kernel after processing. Adds the dirty_log_ring field to VcpuFd.

This is a draft that needs some review and improvements, I'm hereby asking for suggestions for improving the following remaining weaknesses:

mmap_from_fd will succeed with a size supplied that is smaller than the full size of the shared memory buffer.
Send and Sync are probably not safe on KvmDirtyLogRing because accesses are not stateless, due to the state we need to keep track of in user space (next_dirty)
If any vCPU is running while harvesting dirty pages, the user of KvmDirtyLogRing needs to call reset_dirty_rings before reading contents of dirty pages. This is currently the users responsibility, which allows for calling reset_dirty_rings after all dirty pages have been read by the VMM for cases where all vCPUs are halted, but this may be a confusing interface for other cases.
KvmDirtyLogRing is currently pub, perhaps it can be hidden behind an interface on VcpuFd
No unit tests

More info on the interface:
https://www.kernel.org/doc/html/latest/virt/kvm/api.html#kvm-cap-dirty-log-ring-kvm-cap-dirty-log-ring-acq-rel

Requirements

Before submitting your PR, please make sure you addressed the following
requirements:

All commits in this PR have Signed-Off-By trailers (with
git commit -s), and the commit message has max 60 characters for the
summary and max 75 characters for each description line.
All added/changed functionality has a corresponding unit/integration
test.
All added/changed public-facing functionality has entries in the "Upcoming
Release" section of CHANGELOG.md (if no such section exists, please create one).
Any newly added unsafe code is properly documented.

The capability is used for the KVM dirty ring interface for tracking dirtied pages. Signed-off-by: David Kleymann <[email protected]>

Adds the KVM_RESET_DIRTY_RINGS ioctl and the function reset_dirty_rings in impl VmFd to wrap it. Signed-off-by: David Kleymann <[email protected]>

Adds vCPU functions to mmap the dirty ring and iterate over dirty pages. Also adds return value to reset_dirty_rings and flush_dirty_gfns. More info: https://www.kernel.org/doc/html/latest/virt/kvm/api.html#kvm-cap-dirty-log-ring-kvm-cap-dirty-log-ring-acq-rel Signed-off-by: David Kleymann <[email protected]>

Signed-off-by: David Kleymann <[email protected]>

… of map_dirty_log_ring Signed-off-by: David Kleymann <[email protected]>

Signed-off-by: David Kleymann <[email protected]>

Comments on the safety on the operations used to mmap the shared memory buffer of kvm_dirty_gfn entries. Signed-off-by: David Kleymann <[email protected]>

Signed-off-by: David Kleymann <[email protected]>

roypat

please squash your commits so that we do not have later commits be fixups for previous commits.

roypat · 2025-09-09T06:51:57Z

kvm-ioctls/CHANGELOG.md

@@ -2,6 +2,8 @@

 ## Upcoming Release

+- Plumb through KVM_CAP_DIRTY_LOG_RING as DirtyLogRing cap.


This should go into an ### Added section

roypat · 2025-09-09T06:58:39Z

kvm-ioctls/src/ioctls/mod.rs

+        unsafe {
+            let gfn = self.gfns.add(i as usize).as_mut();
+            if (*gfn).flags & KVM_DIRTY_GFN_F_DIRTY == 0 {
+                // next_dirty stays the same, it will become the next dirty element
+                return None;
+            } else {
+                self.next_dirty += 1;
+                (*gfn).flags ^= KVM_DIRTY_GFN_F_RESET;
+                return Some(((*gfn).slot, (*gfn).offset));
+            }
+        }
+    }


Here we should probably do just a single read_volatile of the struct kvm_dirty_gfn, to avoid data races, and write the updated kvm_dirty_gfn flags via write_volatile (or, on weakly ordered architectures such as arm64, an atomic read with acquire ordering, and write with release ordering. Which means we'll probably also want to support checking KVM_CAP_DIRTY_LOG_RING_ACQ_REL)

roypat · 2025-09-09T07:00:10Z

kvm-ioctls/src/ioctls/vcpu.rs

+    ///     }
+    /// }
+    /// ```
+    pub fn dirty_log_ring_iter(&mut self) -> Option<&mut KvmDirtyLogRing> {


we can hide the actual iterator type here by just returning impl Iterator<Item = blablabla>. Then maybe the entire struct doesnt need to be exported?

roypat · 2025-09-09T07:00:42Z

kvm-ioctls/src/ioctls/vm.rs

@@ -1930,14 +1930,14 @@ impl VmFd {
    /// }
    /// ```
    ///
-    pub fn reset_dirty_rings(&self) -> Result<()> {
+    pub fn reset_dirty_rings(&self) -> Result<c_int> {


this should go into the previous commit

roypat · 2025-09-09T07:01:42Z

kvm-ioctls/src/ioctls/vm.rs

+    /// Resets all vCPU's dirty log rings. This notifies the kernel that pages have been harvested
+    /// from the dirty ring and the corresponding pages can be reprotected.


this should also just go into the commit that introduced the function

roypat · 2025-09-09T07:02:29Z

kvm-ioctls/src/ioctls/vcpu.rs

@@ -2106,8 +2106,7 @@ impl VcpuFd {
        }
    }

-    /// Maps the coalesced MMIO ring page. This allows reading entries from
-    /// the ring via [`coalesced_mmio_read()`](VcpuFd::coalesced_mmio_read).
+    /// Maps the KVM dirty log ring. 


this should go into the commit that introduced the function

roypat · 2025-09-09T07:02:46Z

kvm-ioctls/src/ioctls/mod.rs

and all the changes to this file should go into the commit that introduced struct KvmDirtyLogRing

roypat · 2025-09-09T07:03:35Z

kvm-ioctls/CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Upcoming Release

 - Plumb through KVM_CAP_DIRTY_LOG_RING as DirtyLogRing cap.
+- Added support for dirty log ring interface introducing `VcpuFd::reset_dirty_rings`, `KvmDirtyLogRing`


Just add the changelog entries in the commit that introduces the respective functions.

roypat · 2025-09-09T07:04:16Z

kvm-ioctls/src/ioctls/mod.rs

+        };
+
+        let offset  = page_size * KVM_DIRTY_LOG_PAGE_OFFSET as usize;
+
        if bytes % std::mem::size_of::<kvm_dirty_gfn>() != 0 {
            // Size of dirty ring in bytes must be multiples of slot size
            return Err(errno::Error::new(libc::EINVAL));
        }
        let slots = bytes / std::mem::size_of::<kvm_dirty_gfn>();
        if slots & (slots - 1) != 0 {


slots.is_power_of_two()

roypat · 2025-09-09T07:11:07Z

kvm-ioctls/src/ioctls/vcpu.rs

+    /// # extern crate kvm_ioctls;
+    /// # use kvm_ioctls::{Cap, Kvm};
+    /// let kvm = Kvm::new().unwrap();
+    /// let vm = kvm.create_vm().unwrap();
+    /// let mut vcpu = vm.create_vcpu(0).unwrap();
+    /// if kvm.check_extension(Cap::DirtyLogRing) {
+    ///     vcpu.coalesced_mmio_ring().unwrap();
+    /// }
+    /// ```
+    pub fn map_dirty_log_ring(&mut self, bytes: usize) -> Result<()> {
+        if self.dirty_log_ring.is_none() {
+            let ring = KvmDirtyLogRing::mmap_from_fd(&self.vcpu, bytes)?;
+            self.dirty_log_ring = Some(ring);
+        }
+        Ok(())
+    }


the dirty log ring interface is enabled via KVM_ENABLE_CAP (which we expose as enable_cap() in this crate), which is where the size of the ring buffer is specified as well. So I would propose to make the API here a wrapper on top of enable_cap(), say enable_dirty_log_ring(bytes: usize), which does the VmFd::enable_cap() call and then stores the actual size of the ring buffer directly in the VmFd struct. Then on VcpuFd creation, we check if the dirty ring capability was ever enabled on the VmFd, and if so, just mmap the ring with the size stored in the VmFd at vcpu creation time (maybe disallow calling enable_dirty_log_ring() if vcpus were already created previously, although KVM also already checks this). That way the issue of "what is the correct size to mmap" goes away.

roypat · 2025-09-09T07:14:06Z

Also, do we need to do anything about KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP?

roypat · 2025-09-09T07:31:57Z

Send and Sync are probably not safe on KvmDirtyLogRing because accesses are not stateless, due to the state we need to keep track of in user space (next_dirty)

As for this, next_dirty doesnt imply that it cannot be Send/Sync, because the traits dont mean that multiple threads can access the struct at the same time (thats always unsound in Rust's memory model), just that you can transfer ownership between threads, and that read-only references can be shared between threads. A problem would only arrive if we can also Clone this structure, because then the iterator can cause data races inside the mmap'd area from different Rust threads. But as long as for each vcpu we only allow creating a single instance of the iterator (using safe code that is), there are no problems.

Now, whether Send and Sync will actually be useful is a different matter, because I'm assuming you want them to be able to harvest the dirty ring while the vcpu is still running, but with the current API in this PR, you can only get a &mut reference to the ring buffer structure, which you cannot send to another thread anyway. So in this case, you'd need some API that lets you take ownership of the KvmDirtyRingLog structure (maybe store the one that gets created at vcpu creation time in an option, and then have a function that just wraps .take() on that option, keeping in mind that we must never allow safe code to get two owned versions of it for the same vcpu).

roypat · 2025-09-09T07:32:37Z

Also sorry for the late response, we were all preparing for / traveling to KVM Forum last week!

David Kleymann and others added 10 commits August 26, 2025 14:05

cap: add DirtyLogRing cap

1653881

The capability is used for the KVM dirty ring interface for tracking dirtied pages. Signed-off-by: David Kleymann <[email protected]>

Add basic support for KVM_RESET_DIRTY_RINGS ioctl

c77e3d5

Adds the KVM_RESET_DIRTY_RINGS ioctl and the function reset_dirty_rings in impl VmFd to wrap it. Signed-off-by: David Kleymann <[email protected]>

kvm-ioctls: Clearer comment on reset_dirty_rings

d47609a

Signed-off-by: David Kleymann <[email protected]>

kvm-ioctls: add Send, Sync and Drop to KvmDirtyLogRing, fix docstring…

0b112c0

… of map_dirty_log_ring Signed-off-by: David Kleymann <[email protected]>

kvm-ioctls: Make KvmDirtyLogRing pub

51f99ce

Signed-off-by: David Kleymann <[email protected]>

Merge branch 'rust-vmm:main' into main

6162081

Signed-off-by: David Kleymann <[email protected]>

doc: Document dirty ring changes in changelog

8ddf105

Signed-off-by: David Kleymann <[email protected]>

doc: Safety comments on KvmDirtyLogRing::mmap_from_fd

edb9ea5

Comments on the safety on the operations used to mmap the shared memory buffer of kvm_dirty_gfn entries. Signed-off-by: David Kleymann <[email protected]>

kvm-ioctls: Fix map_dirty_log_ring doctest

f9adaaa

Signed-off-by: David Kleymann <[email protected]>

roypat requested changes Sep 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KVM Dirty log ring interface #344

KVM Dirty log ring interface #344

Uh oh!

davidkleymann commented Sep 3, 2025

Uh oh!

roypat left a comment

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025

Uh oh!

roypat Sep 9, 2025 •

edited

Loading

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

Uh oh!

		@@ -2,6 +2,8 @@

		## Upcoming Release

		- Plumb through KVM_CAP_DIRTY_LOG_RING as DirtyLogRing cap.

		/// Resets all vCPU's dirty log rings. This notifies the kernel that pages have been harvested
		/// from the dirty ring and the corresponding pages can be reprotected.

KVM Dirty log ring interface #344

Are you sure you want to change the base?

KVM Dirty log ring interface #344

Uh oh!

Conversation

davidkleymann commented Sep 3, 2025

Summary of the PR

Requirements

Uh oh!

roypat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roypat Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

roypat commented Sep 9, 2025

Uh oh!

Uh oh!

roypat Sep 9, 2025 •

edited

Loading