Fix key performance issues in 6.17 with byte range locks - #202
Open
hbirth wants to merge 31 commits into
Open
Conversation
hbirth
force-pushed
the
redfs-ubuntu-hwe-6.17.0-16.16-24.04.1
branch
from
August 25, 2026 11:27
588473f to
7ffa485
Compare
|
What does this patch do? Do I still need to rebase my PR 201 by merge the range lock interval tree with DLM cache interval? |
hbirth
force-pushed
the
redfs-ubuntu-hwe-6.17.0-16.16-24.04.1
branch
6 times, most recently
from
August 26, 2026 09:52
839c405 to
9fb952d
Compare
Collaborator
Author
|
This is doing pretty well in xfstests |
fuse_dlm_buffered_write() sends the unaligned edges of a write to the server itself so no partly written block is dirtied, and cuts at PAGE_SIZE. Nothing checked that a block is a page: fc->blkbits keeps whatever blksize= asked for, and attr->blksize sets inode->i_blkbits per inode. iomap then goes back for the remainder of an edge block and dirties what the write already sent. Refuse the connection at FUSE_INIT when a writeback cache is negotiated on a block that is not a page, and pin inode->i_blkbits afterwards. st_blksize still reports what the server named. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Readahead fills the page cache past the range fuse_cache_read_iter() locked, so those folios get no revoke when a remote node writes them. Request a read grant over the whole window in fuse_readahead() before any folio is consumed, and skip the window when the request fails. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A FUSE_DLM_WB_LOCK reply and a NOTIFY revoke run on different threads, so a revoke can arrive before the grant is recorded, find nothing, and leave a grant that is never taken back. revoke_gen caught this with a per-inode counter that cannot say which range was hit, so any revoke re-requested every grant in flight. Add enum fuse_dlm_range_state: REQUESTED or REVOKED on the new cache->pending list while in flight, GRANTED in cache->ranges. A revoke marks the pending requests it overlaps, fuse_dlm_request_commit() drops a marked grant, and unlinking the request and recording its grant is one step under the cache lock. Pending requests stay off the interval tree, so no tree walker needs a state filter. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
struct fuse_dlm_range carried a lifecycle state and a mode, the mode stored as FUSE_PCACHE_LK_READ/_WRITE (1 and 2) in a field typed enum fuse_page_lock_mode, whose enumerators are 0 and 1. The two are never independent, and fuse_dlm_range_is_locked() relied on the value ordering to let a write grant cover a read. Replace GRANTED with READ and WRITE and drop the mode field and the FUSE_PCACHE_LK_* values. fuse_dlm_state_satisfies() replaces the ordinal comparison. fuse_dlm_request_begin() no longer takes a mode: the one that reaches the tree is the one passed to the commit. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A granted range does not say whether anything is cached under it, so a revoke assumes the worst and drops with invalidate_inode_pages2_range(), which launders dirty folios and waits for a FUSE_WRITE reply. Add enum fuse_dlm_range_content: nothing cached, data the server has seen, or data it has not. It is an upper bound, raised in fuse_dlm_range_touched() from the two points that authorise cached IO, so it moves before the data lands. Only fuse_dlm_ranges_flushed() lowers it, under i_rwsem held exclusive; a mapped inode is skipped. No caller acts on this yet. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Grants are page aligned, buffered writes need not be, and fuse_dlm_range_touched() marked whole tree nodes: a 4K write into a 1M grant claimed the whole grant, so the record could not say which part of a boundary page this client wrote. Split at both ends of the marked region and mark only what it covers. Both halves keep state and content, so grant coverage is unchanged and fuse_dlm_try_merge() recoalesces. fuse_get_dlm_lock() stops marking on the write path, where its range is the page aligned request; fuse_cache_write_iter() marks the exact range instead. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Writeback sent every dirty folio whole, which is why an unaligned edge cannot be cached: the untouched remainder of a boundary page would go to the server with it. fuse_dlm_dirty_run() walks forward while the recorded content stays the same, so fuse_iomap_writeback_range() sends one written run at a time. An unwritten run is reported as IOMAP_HOLE, which iomap skips, and a folio with no written run has its writeback ended by iomap itself. A range not covered by write grants throughout still goes whole: only a write grant makes this client the one that dirties it. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A data invalidation always dropped the range with invalidate_inode_pages2_range(), which launders dirty folios and waits for a FUSE_WRITE reply, even for a range holding nothing. Ask instead, under the gate so nothing populates or dirties in between. filemap_range_has_page() says whether anything is cached, and fuse_dlm_range_may_be_dirty() whether laundering is needed; when it is not, invalidate_mapping_pages() drops the same folios without waiting for the server. fuse_cache_write_iter() also marks the folio pagecache_isize_extended() dirties, or the record would report it clean. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_unlock_range() removed the ranges it revoked, so a grant taken away looked like a gap. fuse_dlm_dirty_run() calls a gap UNKNOWN and writeback sends the whole range, so a writer that passed fuse_dlm_lock_is_held() before a revoke and dirtied after it had its bytes sent for a range another node now holds. Mark the range FUSE_DLM_RANGE_REVOKED instead; one with nothing cached under it is still removed. It covers nothing, so the IO paths request again. fuse_dlm_dirty_run() returns enum fuse_dlm_run so writeback can tell the two apart and call fuse_dlm_regrant_range() before sending a revoked run. Splitting at the revoke bounds replaces the trim and punch-hole arithmetic, so fuse_dlm_punch_hole() goes. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
wb_inval_rwsem fenced cached IO out for the whole of a NOTIFY invalidate, so a writer could not dirty the page cache under a grant being revoked. The record does that now: a write records its bytes before dirtying them, a revoke marks the range instead of forgetting it, and writeback takes the range again before sending anything marked that way. A read never needed the fence. Gone with it: the re-validation and retry loops that existed only because the gate had to be dropped around a FUSE_DLM_WB_LOCK round trip, fuse_cache_wr_dlm_lock()'s unrecorded flag, and the percpu_rw_semaphore. setattr and O_TRUNC lose it too, holding i_rwsem exclusive. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_do_readfolio() asks for page_zeroing, and fuse_copy_folio() answers that by zeroing the whole folio whenever the request covers less than all of it. That is right for a whole folio read, where the zeroing fills a short reply, and wrong for anything smaller: fuse_iomap_read_folio_range() already asks for one block of a large folio at a time, so a read-modify-write of one block wipes every other block of the folio while their iomap uptodate bits still call them valid. Add fuse_read_folio_range(), which asks without page_zeroing and zeroes exactly what the reply left short, and use it for the partial reads. fuse_do_readfolio() keeps the whole folio callers. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_dirty_run() walked the ranges overlapping the query and gave up on the first gap, returning FUSE_DLM_RUN_UNKNOWN without saying how far that answer held, so a run classified up to the gap was thrown away and the caller sent the whole query range whole. Treat a gap as a classification of its own and report how far it reaches, the same as any other run. A caller now always learns the extent of the answer it got, and a record that covers part of the query is used for that part instead of being discarded. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A cached write covering part of a folio makes iomap read the rest in so the folio can be called valid. Under DLM that read buys nothing: writeback already declines to send the server's own bytes back, an expanding write pays a round trip for a range that holds nothing, and a write-only handle has the READ refused. Skip it. fuse_iomap_read_folio_range() leaves the range alone, fuse_iomap_put_folio() clears the uptodate flag under the folio lock and records what the copy actually reached, and fuse_read_folio() fills the record's gaps on the next read. Only for folios of a single block: clearing the flag on a larger one leaves iomap's per block bits set, which a read consults through ->is_partially_uptodate. Such a folio can be dirty and invalid at once, so fuse_read_folio() waits out writeback first. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The record describes the inode, not one incarnation of a folio. A folio left invalid by a partial write is clean once writeback has sent what it held, and reclaim may take it; the record still names those bytes, so fuse_read_folio_merge() would decline to fetch them for the next folio at that index. Lower the record as writeback hands a run over, for an invalid folio only. The folio lock orders it against a rewrite. fuse_dlm_split_at() can no longer fail: it now decides which bytes exist. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_request_commit() reports -EAGAIN when a revoke overlapped a request while it was on the wire, and __fuse_get_dlm_lock() went round again with nothing stopping it. A remote node revoking as fast as the grants are handed out keeps that going for as long as it likes, and writeback asks for a grant with a folio locked, so the loop is not merely slow, it holds a folio hostage and the task is unkillable while it does. Give it a count and a signal check. Both are generous: every pass is a whole round trip, so reaching either means the range is genuinely being fought over and the caller is better told than left spinning. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_do_setattr() revokes the grants past the new size and discards the page cache there, but never tells the record. A revoked range is kept because it describes page cache dirtied before the grant went, so every truncate leaves ranges describing folios that no longer exist. Nothing frees them before the inode, and until then the inode reports unwritten data and every later invalidate over that region launders. Call fuse_dlm_ranges_dropped() from the first whole page above the new size; the page holding the new end of the file survives, so its record has to. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The handler starts writeback so the bytes go while the grant still covers them, then drops the range. When the record says the range may be dirty the drop launders, and laundering waits out the folios just put under writeback: the handler waits for FUSE_WRITE replies it queued itself. Wait here instead, before the revoke, so nothing classifies as revoked and re-requests the range from inside the handler. A server revoking from a thread it also answers FUSE_WRITE on still deadlocks; the comment now says so. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The forced-direct-IO latch is checked once, so a buffered write that passed the check just before a notify set the latch dirties the page cache after the notify dropped the mapping. The coherency gate used to fence that. What is left behind is invisible to the direct path: a direct read misses the dirty folio, and a direct write lands underneath it, after which the invalidate launders rather than drops and puts the stale folio on the server on top. Write the range back before re-routing. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The kernel-doc of the DLM record says page offset throughout, but every one of those arguments is a byte offset that happens to be page aligned, and the same file computes real page indices a few lines from some of them. The code is right; the wording invites passing one for the other. Say byte offset, and note on struct fuse_dlm_range that grants arrive page aligned while the content bound is split to the exact bytes an unaligned write covers, so the two are not the same thing. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_lock_range_locked() put every range overlapping the grant on the upgrade list whole, without splitting at the grant bounds the way fuse_dlm_unlock_range() does. A range extending past the grant then had its uncovered part upgraded with it: holding a read grant on [0, 8191] and being granted write on [4096, 8191] recorded the whole node as held for write, so fuse_dlm_lock_is_held() reported [0, 4095] covered, a cached write there never sent FUSE_DLM_WB_LOCK, and the server went on thinking this client held only a read there -- cluster exclusion broken without any error to see. The revoked case is worse. Writeback re-grants only the run it is about to send, but the commit flipped the whole revoked range back to held, so revoked-dirty bytes outside the re-granted run reclassified as FUSE_DLM_RUN_DIRTY and were sent without taking the range again -- exactly the lost update FUSE_DLM_RANGE_REVOKED exists to prevent. Split at both grant bounds before walking, so upgrades apply only inside the grant. Both halves of a split keep state and content, and fuse_dlm_try_merge() at the end recoalesces whatever stayed equal. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
fuse_do_setattr() frees the record above the new size, but the invalidate_inode_pages2() it runs empties the mapping below it too, and the record there was left alone. A dirty folio is laundered by that invalidate, which puts its bytes on the server, but laundering lowers the record only for a folio the page cache does not consider valid (fuse_dlm_range_sent()), so an ordinary uptodate dirty folio leaves a DIRTY range describing page cache that no longer exists. The next unaligned write to such a page allocates a fresh folio, leaves it unfilled, and records only its own bytes. The stale DIRTY run over the rest of the page then makes fuse_read_folio_merge() keep folio bytes nobody wrote, and writeback classifies and sends them -- bad data both to a local reader and to the server. fsx reaches this with any truncate-then-partial-write sequence. Drop the record below the new size as well, and only when the drop really emptied it: a busy folio that survives the invalidate still needs its record, so ask filemap_range_has_page() first, the same rule the NOTIFY invalidate path applies before its drop. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
The atomic O_TRUNC branch of fuse_do_setattr() releases every DLM range before dropping the cache, but the far more common path -- fuse_open() with fc->atomic_o_trunc -- called truncate_pagecache() and left the whole record standing. Ranges recorded DIRTY then described folios that no longer existed, and the next unaligned write to such a page left a fresh folio unfilled while the stale run made fuse_read_folio_merge() keep its unwritten bytes and writeback send them: bad data from something as plain as 'echo x > file' followed by a partial write. Release the record before the truncate, exactly as fuse_do_setattr() does; i_rwsem is held exclusive here (is_wb_truncate), so no cached write is between recording and dirtying. The FOPEN_KEEP_CACHE-less open one line below drops the mapping with invalidate_inode_pages2() and has the same problem. Drop the record there too, but only when the invalidate really emptied the mapping, under the same filemap_range_has_page() rule the NOTIFY path applies: a busy folio that survived keeps its record. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
FALLOC_FL_PUNCH_HOLE and FALLOC_FL_ZERO_RANGE flush the span, punch it on the server, and drop it from the page cache with truncate_pagecache_range() -- and told the record nothing. The flush lowers the record only for folios the page cache does not consider valid, so an uptodate dirty folio inside the hole left a DIRTY range describing page cache that no longer exists. The next unaligned write into the hole allocates a fresh folio, leaves it unfilled, and records its own bytes; the stale DIRTY run over the rest of the page makes fuse_read_folio_merge() keep folio bytes nobody wrote and writeback send them. fsx exercises punch hole against partial writes constantly, which is where the bad-data failures on this branch come from. Free the record over the whole pages inside the hole, under the same filemap_range_has_page() rule the NOTIFY path applies before its drop. The partial pages at the edges survive the truncate with their punched part zeroed -- those zeroes are real bytes the folio really holds -- so their record has to survive with them. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
__fuse_copy_file_range() flushes the destination span, lets the server copy, and then drops the copied pages with truncate_inode_pages_range() as stale -- without telling the record. As on the truncate and punch hole paths, an uptodate dirty folio flushed and then dropped leaves a DIRTY range describing page cache that no longer exists, and the next partial write there keeps and flushes folio bytes nobody wrote. Free the record over the dropped span, gated on filemap_range_has_page() like the other drops: the copy runs under i_rwsem but faults do not, and a folio a fault put back after the truncate keeps its record. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
fuse_attr_cache_mask() trusted the local i_size over the server's only while a recorded write grant covered [attr->size, i_size). Two paths leave locally-extended data with no such record: a fault dirties pages under a page-mkwrite lock that is never recorded, and a local truncate revokes its own tail grants, after which cached writes extend the file again. On the next attribute refresh the check failed, the server's smaller size was applied, and truncate_pagecache() discarded dirty pages past it -- cached data destroyed by a GETATTR. The invalidate that follows then dropped the rest of the mapping with the record left standing, arming the stale-DIRTY bad-data path on every page of the file. generic/075 hits this within a few hundred fsx operations; the failure point moves with the attribute timeout, which is what made the runs look nondeterministic. Keep STATX_SIZE cached also while anything in [attr->size, i_size) is dirty or under writeback: those bytes exist only here, and the server cannot have a newer opinion about a size it has never seen. A remote truncate still lands, exactly as the design intends: its revoke launders and drops the tail first, so nothing is dirty there by the time the smaller size arrives. The no-grant early return learns the same rule, since a mapping dirtied only through page-mkwrite has no recorded grant at all. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
When an attribute reply is applied with the size not served from the cache, fuse_change_attributes_i() truncates the page cache to the server's size and, when the data may be stale, drops the whole mapping with invalidate_inode_pages2() -- and told the record nothing. This is the same hole just closed on the truncate, O_TRUNC open, punch hole and copy_file_range paths, reached from every GETATTR, and it is the drop behind the generic/075 corruption: the server-side trace shows a folio whose bytes were flushed correctly once, dropped here with its record left DIRTY, repopulated by a later partial write that leaves the folio unfilled, and then written back with the stale record naming the unfilled head -- zeroes sent over data the server already had. Free the record over what really went: unconditionally above the new size, where truncate_pagecache() leaves nothing behind, and over the rest only when the invalidate emptied the mapping, under the same filemap_range_has_page() rule as the other drops, so a folio that survived or was faulted back keeps its record. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
The interior stage of fuse_dlm_buffered_write() returns @ToTal on error. For a page aligned start the head chunk is empty, so @ToTal is zero and an interior failure returns zero: no error and no bytes. fuse_cache_write_iter() then falls back to @err, which still holds the positive count from generic_write_checks(), and a write that failed outright reports every byte as written. Return the error while nothing has landed, a short write once something has, in every chunk. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_iomap_read_folio_range() skipped the fill under DLM and left the folio invalid, so a write covering part of a block need not read the remainder back from a server that may hold nothing there. A DLM buffered write is cut at the block size now, so its interior covers whole blocks and the unaligned edges never reach iomap. Nothing reaches the skip. It goes, with the invalid folio, its zero fill, fuse_iomap_put_folio() and the cursor between fuse_iomap_begin() and fuse_iomap_end(). Writeback's copy of the assumption goes too, and that one is a bug on its own: iomap leaves a large folio partly valid whenever a write covers some of its blocks and not others, so with no DLM such a folio has its dirty blocks reported as a hole and dropped. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Writeback asked the record what each dirty run was, because a boundary folio could hold this client's bytes next to the server's. Nothing partly written is dirtied any more, so every run iomap reports dirty was written whole by this client. Call fuse_dlm_regrant_range() for the whole run instead and let it early-exit on a grant still held; fuse_dlm_dirty_run() and fuse_dlm_range_sent() go with the classification. fuse_read_folio_merge() asked the same question because reading a dirty folio whole would lose what a write left in it. iomap answers it through ->is_partially_uptodate, which fuse already publishes. That also fixes a mount with no DLM, where the merge was skipped for want of a record and the folio read whole. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
struct fuse_dlm_range carried an upper bound on what the page cache under it held. Writeback no longer consults it, and the three remaining users have exact page cache equivalents: fuse_dlm_range_may_be_dirty() is filemap_range_needs_writeback(), the keep-or-free decision for a revoked range is filemap_range_has_page(), and the revoked-and-dirty arm of fuse_dlm_write_grant_exists() is covered by its caller's PAGECACHE_TAG tests. Remove enum fuse_dlm_range_content and fuse_dlm_range::content, and with them fuse_dlm_range_touched(), fuse_dlm_range_written(), fuse_dlm_ranges_flushed(), fuse_dlm_raise_dirty_shared(), fuse_dlm_mark_populated(), fuse_dlm_touched_level() and fuse_dlm_range_may_be_dirty(). fuse_iomap_end() has nothing left to do. The record is written only on a grant and on a revoke now, so a cached write does not take the per inode cache lock at all. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
hbirth
force-pushed
the
redfs-ubuntu-hwe-6.17.0-16.16-24.04.1
branch
from
August 26, 2026 19:49
9fb952d to
9bb6233
Compare
With the writeback cache the kernel owns append positioning: a cached O_APPEND write is placed at the local i_size, and writeback later sends FUSE_WRITE requests with explicit offsets. Those offsets are only honoured if the server does not re-apply O_APPEND itself -- on Linux, pwrite(2) on a descriptor opened O_APPEND appends regardless of the offset argument. libfuse's passthrough examples know this and strip the flag, but only on FUSE_OPEN; a file created with FUSE_CREATE kept an O_APPEND backing descriptor, and every writeback run landed at the server's EOF instead of its offset. That is not just a theoretical hazard. Writeback legitimately covers the same bytes twice -- a folio whose record still names sent bytes is re-sent from its page start -- and with offsets honoured that re-send is idempotent. Appended instead, each flush's overlap with the previous one is duplicated at EOF and the file grows: generic/069 fails with exactly this shape (an appended file 230356 bytes too long, every byte of the excess a repeated flush overlap; server-side IO traces show each pwrite landing at EOF, sum of pwrites equal to the final file size). Strip O_APPEND from the flags sent in FUSE_OPEN, FUSE_CREATE and the compound open, the same way O_TRUNC is suppressed without atomic_o_trunc. The server cannot use the flag correctly under writeback caching anyway: appending server-side would order writeback runs by arrival, not by offset. Non-writeback mounts are unchanged, since there the server really does own append positioning. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.