fuse: serialize cached IO and invalidation with per-inode range locks - #201
Conversation
Add fuse_range_lock, an interval-tree based byte-range lock embedded in struct fuse_inode. It serializes concurrent cached reads and writes that touch overlapping byte ranges of the same file while letting non-overlapping IO proceed concurrently, and lets attribute and BRL invalidation (fuse_reverse_inval_inode(), truncate in fuse_do_setattr()) block only on in-progress IO that overlaps the range being invalidated rather than serializing against all IO on the inode. Each held range carries a two-state lifecycle. A range is first acquired in INIT state, which reserves it against other local readers/writers but stays invisible to invalidation; this covers the window where a fuse_get_dlm_lock() request to the DLM server may be in flight, so a NOTIFY invalidate never blocks on that unbounded round trip. Once IO is about to touch the page cache the range is promoted to READY, at which point it becomes fully exclusive against overlapping ranges, including other READY holders and invalidation. Callers are responsible for gating on fc->writeback_cache && fc->dlm before acquiring or releasing a range lock, since the range lock only needs to run when the DLM coherency path is active; this keeps the non-DLM read/write paths free of any added locking overhead. dev.c grows a 'complete' callback invoked from fuse_request_end() before the requester is woken, so range locks can be promoted to READY (or released) synchronously with the reply to the DLM lock request rather than requiring the waiter to do it after waking up. Wire the new range lock into the read, write, and writeback paths in file.c, the setattr/truncate path in dir.c, and the invalidate path in inode.c. Signed-off-by Hai Zhong Zhou <hazhou@ddn.com>
|
since I had done this before and we decided against it ... I don't think we need the range lock |
Horst, I think the background is changed now... At that time, byte range lock should be used for read/write IO protection and no "gatekeeper" requirement, now this range lock in my patch is also used as fine-grained "gatekeeper" between invalidation and read/write IO. In addition, I believe at that time we discussed about to introduce the DLM byte range into the kernel to hold daemon side DLM BRL, that is to tight with the DLM... While this range lock is just fuse kernel local range lock, not directly hold the daemon side DLM BRL. Let me explain how this works and why we need this range lock. |
|
Just had a call with Horst, and we will merge the range lock interval tree into the DLM cache interval tree. Let me refresh the patch. |
Horst just create a new PR (#202) which shares the DLM cache interval tree with range lock. So I'll suspend my "merging" interval tree work for this PR. |
2 participants
Add fuse_range_lock, an interval-tree based byte-range lock embedded in struct fuse_inode. It serializes concurrent cached reads and writes that touch overlapping byte ranges of the same file while letting non-overlapping IO proceed concurrently, and lets attribute and BRL invalidation (fuse_reverse_inval_inode(), truncate in fuse_do_setattr()) block only on in-progress IO that overlaps the range being invalidated rather than serializing against all IO on the inode.
Each held range carries a two-state lifecycle. A range is first acquired in INIT state, which reserves it against other local readers/writers but stays invisible to invalidation; this covers the window where a fuse_get_dlm_lock() request to the DLM server may be in flight, so a NOTIFY invalidate never blocks on that unbounded round trip. Once IO is about to touch the page cache the range is promoted to READY, at which point it becomes fully exclusive against overlapping ranges, including other READY holders and invalidation.
Callers are responsible for gating on fc->writeback_cache && fc->dlm before acquiring or releasing a range lock, since the range lock only needs to run when the DLM coherency path is active; this keeps the non-DLM read/write paths free of any added locking overhead.
dev.c grows a 'complete' callback invoked from fuse_request_end() before the requester is woken, so range locks can be promoted to READY (or released) synchronously with the reply to the DLM lock request rather than requiring the waiter to do it after waking up.
Wire the new range lock into the read, write, and writeback paths in file.c, the setattr/truncate path in dir.c, and the invalidate path in inode.c.
Signed-off-by Hai Zhong Zhou hazhou@ddn.com