Skip to content

OpenVINO Backend: Upgrade graph cache to use buffer pointers instead of names#261

Draft
mostafafaheem wants to merge 1 commit into
ravi9:dev_backend_openvinofrom
mostafafaheem:improve_graph_key
Draft

OpenVINO Backend: Upgrade graph cache to use buffer pointers instead of names#261
mostafafaheem wants to merge 1 commit into
ravi9:dev_backend_openvinofrom
mostafafaheem:improve_graph_key

Conversation

@mostafafaheem

Copy link
Copy Markdown

Overview

This PR replaces using names for graph_key struct with buffer pointers+offsets. This was done due to the previous approach sometimes causing false negatives, where the first decode graph would be deemed a cache miss, even though it is the same as the prefill graph.

Additional information

Example:

diff --git a/../prefill_graph.txt b/../decode_graph.txt
index 18ed618be..b6822c19c 100644
--- a/../prefill_graph.txt
+++ b/../decode_graph.txt
@@ -15,14 +15,14 @@ Graph input: 6:1:node_5#5502
 Graph input: 7:0:node_6#5594
 Graph input: 8:0:leaf_7#905
 Graph input: 9:0:node_7#5617
-Graph input: 9:1: (view)#5686
+Graph input: 9:1:leaf_7 (view)#5686^M
 Graph input: 10:0:leaf_8
 Graph input: 10:1:node_5#5502
 Graph input: 11:0:node_10#5640
 Graph input: 11:1:leaf_9
 Graph input: 12:0:leaf_10#928
 Graph input: 13:0:node_11#5663
-Graph input: 13:1: (view)#5709
+Graph input: 13:1:leaf_10 (view)#5709^M
 Graph input: 14:0:leaf_17
 Graph input: 14:1:node_5#5502
 Graph input: 15:0:node_14#5525

The views in the first decode graph accessed the leaf_7 and leaf_10 tensors after they were named, whereas they were still unnamed when the prefill graph accessed them. This is purely a name difference, the underlying graph is the exact same.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, AI was used to explain relevant code relating to constructing graph keys, and help implementing an alternative using buffer pointers.

@mostafafaheem
mostafafaheem marked this pull request as ready for review July 22, 2026 19:12
@mostafafaheem
mostafafaheem marked this pull request as draft July 23, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant