Don't wait for the GPU when releasing resources from finalizers by maleadt · Pull Request #3318 · JuliaGPU/CUDA.jl

maleadt · 2026-10-01T11:24:22Z

Julia runs finalizers on whatever thread happens to trigger a garbage collection, at whatever point that thread allocates or releases a lock. CUDA.jl uses finalizers to free GPU memory and destroy other objects, but several of the driver and library calls it made there wait for all running kernels to finish, on any stream. Worse, while one thread is inside such a call, kernel launches from every other thread are blocked too. Measured with a 1 s kernel running on another stream, on an RTX 5080 (the Jetson Nano, Xavier and Orin behave the same for the memory calls):

call made from a finalizer	blocks the collecting thread	blocks a launch on another thread
`cuMemFree` (unified memory, or `JULIA_CUDA_MEMORY_POOL=none`)	906 ms	805 ms
`cuMemFreeHost` (`HostMemory` arrays)	906 ms	805 ms
`cuMemHostUnregister` (`pin`, `unsafe_wrap` of host memory)	906 ms	805 ms
`cuArrayDestroy` (texture arrays)	906 ms	805 ms
`cudnnDestroy`, `curandDestroyGenerator`, some cuSPARSE info objects	899 ms	799 ms
`cuModuleUnload`, cuBLAS/cuFFT/cuSOLVER handle destruction	899 ms	–
`cuMemFreeAsync` (the default stream-ordered pool)	–	–

So a GC at the wrong moment stalls a thread, and every other thread's launches, for as long as unrelated kernels run. That includes the GC that synchronize may trigger right before it starts waiting. And if the running GPU work depends on a task on that thread making progress (e.g. a kernel waiting for a flag the host sets), it deadlocks. Moving the calls to a background thread doesn't help, because of the global launch stall: they have to be avoided while kernels run.

With this PR, finalizers never call into CUDA. They only push the object onto a lock-free list, and regular tasks release it later: when allocating, around synchronization, and otherwise within a second. What happens then depends on whether releasing the resource can wait for the GPU:

Releases that don't wait happen right away. Device memory is freed in stream order with cuMemFreeAsync, which now also covers JULIA_CUDA_MEMORY_POOL=none. Pinned host and unified memory is allocated from stream-ordered pools where the driver supports them (CUDA 13+), and freed the same way. Where it doesn't (e.g. on Jetson devices), freed host and unified allocations are cached and reused once the work using them has finished. Events, graphs, texture objects and library objects known not to wait are destroyed, and library handles go back to their cache.
Releases that may wait are held until memory is reclaimed, i.e. on an out-of-memory error or when calling CUDA.reclaim(): unregistering pinned or wrapped host memory (the owner is kept alive until then), emptying the host and unified memory caches, freeing memory without stream-ordered support (very old drivers), unloading modules, destroying texture arrays, and library objects whose destruction may wait (cuRAND generators, cuSPARSE analysis infos, cuTENSOR plans, ...). The background task that trims an idle pool doesn't reclaim anymore.

Memory is often released after the stream it was last used on has been collected, so a destroyed stream records a final event, and memory released afterwards waits for it on a separate non-blocking stream per context.

Every finalizer that CUDA.jl registers goes through a single function, resource_finalizer(f, obj; blocking=true, ctx=context()), which is now public API (CUDA.resource_finalizer). The finalizer only queues the object; f runs later on a regular task, in the context captured at registration: at the next drain if blocking=false, or when memory is reclaimed (the default). Packages wrapping CUDA library objects can use it for the same purpose.

No lock is held while collecting garbage (a task holding a lock doesn't run finalizers, so reclaim() wouldn't free what just became unreachable) or while waiting for the GPU (the work being waited for may depend on a task that needs that lock). Nothing is released while a graph is being captured, as that could invalidate the capture; starting a capture waits for routine releases to finish, and fails instead of waiting while reclaim() synchronizes the device or runs destructors that may wait for the GPU.

User-visible changes in behavior: collected memory isn't released by GC.gc() itself anymore, but by the next CUDA.jl operation that allocates or synchronizes, or within a second (CUDA.pool_status() releases it first). Resources whose release may wait for the GPU are released at the latest when CUDA.jl runs out of memory, or when calling CUDA.reclaim(), which can itself wait for the GPU. That includes the host and unified memory caches on devices without memory pools, which aren't trimmed in the background, as that would block. JULIA_CUDA_MEMORY_POOL=none now also disables the host and unified memory pools.

The design follows what other GPU stacks do: PyTorch, CuPy, XLA, RMM and CCCL don't release storage to the driver when the language object dies either, but recycle it in stream order, and only release it at explicit points like out-of-memory handling or empty_cache().

Not waiting for the GPU also makes allocating host and unified memory considerably faster, as pinning and unpinning, or allocating unified memory, is now mostly avoided. Allocating, touching and freeing an array in a loop, on the RTX 5080 (µs per iteration):

	before	after
`DeviceMemory`, 4 KiB	0.80	0.81
`DeviceMemory`, 4 KiB, `JULIA_CUDA_MEMORY_POOL=none`	32.7	1.7
`HostMemory`, 4 KiB	306	7.8
`HostMemory`, 64 MiB	13822	2988 (mostly the PCIe transfer)
`UnifiedMemory`, 4 KiB	29.2	1.7
`UnifiedMemory`, 64 MiB	7432	2407

The PR also contains a few independent fixes it depends on: kernel output is flushed before reporting a kernel exception (it used to be flushed as a side effect of blocking frees), a failed registration isn't recorded as a pin anymore, an event that synchronize found done by polling is synchronized once more (compute-sanitizer doesn't treat a successful cuEventQuery as synchronization, #3346), and a multi-GPU test restores the device it switched to.

The new tests keep the GPU busy with a kernel that waits for another task on the same thread, and then collect arrays of every memory type, pinned and wrapped arrays, texture arrays, modules and streams. Without this PR, these operations block until the kernel gives up after its 60 s timeout; with it, they pass in seconds. The resource, driver, array, pool and exception tests, and the library tests touched here, pass on an RTX 5080 with both memory allocators.

github-actions · 2026-10-01T15:00:27Z

CUDA.jl Benchmarks

Details

Benchmark suite	Current: `0151118`	Previous: `ae637e5`	Ratio
`array/accumulate/Float32/1d`	`105377` ns	`105001` ns	`1.00`
`array/accumulate/Float32/dims=1`	`77354` ns	`77262` ns	`1.00`
`array/accumulate/Float32/dims=1L`	`1684900` ns	`1684940` ns	`1.00`
`array/accumulate/Float32/dims=2`	`148019` ns	`148043` ns	`1.00`
`array/accumulate/Float32/dims=2L`	`685568` ns	`684338` ns	`1.00`
`array/accumulate/Int64/1d`	`123735` ns	`123503` ns	`1.00`
`array/accumulate/Int64/dims=1`	`81765` ns	`81902` ns	`1.00`
`array/accumulate/Int64/dims=1L`	`1803800` ns	`1803636` ns	`1.00`
`array/accumulate/Int64/dims=2`	`160529` ns	`163105` ns	`0.98`
`array/accumulate/Int64/dims=2L`	`988269` ns	`988275` ns	`1.00`
`array/broadcast`	`16892` ns	`16981` ns	`0.99`
`array/broadcast launch`	`7585.25` ns	`7497.25` ns	`1.01`
`array/construct`	`885` ns	`886.76` ns	`1.00`
`array/copy`	`16290` ns	`16325` ns	`1.00`
`array/copyto!/cpu_to_gpu`	`207432` ns	`206249` ns	`1.01`
`array/copyto!/gpu_to_cpu`	`240458` ns	`240560` ns	`1.00`
`array/copyto!/gpu_to_gpu`	`9066` ns	`8967` ns	`1.01`
`array/iteration/findall/bool`	`134924` ns	`135815` ns	`0.99`
`array/iteration/findall/int`	`146950` ns	`145400` ns	`1.01`
`array/iteration/findfirst/bool`	`70946` ns	`71077` ns	`1.00`
`array/iteration/findfirst/int`	`73102` ns	`73309` ns	`1.00`
`array/iteration/findmin/1d`	`59078` ns	`59845` ns	`0.99`
`array/iteration/findmin/2d`	`102967` ns	`103127` ns	`1.00`
`array/iteration/logical`	`187196` ns	`187405` ns	`1.00`
`array/iteration/scalar`	`58628` ns	`59115` ns	`0.99`
`array/permutedims/2d`	`49024` ns	`49364` ns	`0.99`
`array/permutedims/3d`	`50405` ns	`50795` ns	`0.99`
`array/permutedims/4d`	`51904` ns	`52271` ns	`0.99`
`array/random/rand/Float32`	`10969` ns	`11109` ns	`0.99`
`array/random/rand/Int64`	`17968` ns	`18411` ns	`0.98`
`array/random/rand!/Float32`	`8206.666666666666` ns	`8183` ns	`1.00`
`array/random/rand!/Int64`	`15683` ns	`15765` ns	`0.99`
`array/random/randn/Float32`	`34589` ns	`34501` ns	`1.00`
`array/random/randn!/Float32`	`24586` ns	`24403` ns	`1.01`
`array/reductions/mapreduce/Float32/1d`	`30567` ns	`30573` ns	`1.00`
`array/reductions/mapreduce/Float32/dims=1`	`39832` ns	`40213` ns	`0.99`
`array/reductions/mapreduce/Float32/dims=1L`	`53215` ns	`53356` ns	`1.00`
`array/reductions/mapreduce/Float32/dims=2`	`59367` ns	`59358` ns	`1.00`
`array/reductions/mapreduce/Float32/dims=2L`	`71189` ns	`71210` ns	`1.00`
`array/reductions/mapreduce/Int64/1d`	`37303` ns	`37469` ns	`1.00`
`array/reductions/mapreduce/Int64/dims=1`	`42996` ns	`42915` ns	`1.00`
`array/reductions/mapreduce/Int64/dims=1L`	`89190` ns	`89249` ns	`1.00`
`array/reductions/mapreduce/Int64/dims=2`	`61500` ns	`61370` ns	`1.00`
`array/reductions/mapreduce/Int64/dims=2L`	`87651` ns	`87543` ns	`1.00`
`array/reductions/reduce/Float32/1d`	`30748` ns	`30615` ns	`1.00`
`array/reductions/reduce/Float32/dims=1`	`40177` ns	`40264` ns	`1.00`
`array/reductions/reduce/Float32/dims=1L`	`52686` ns	`52907` ns	`1.00`
`array/reductions/reduce/Float32/dims=2`	`59159` ns	`59462` ns	`0.99`
`array/reductions/reduce/Float32/dims=2L`	`70896` ns	`71728` ns	`0.99`
`array/reductions/reduce/Int64/1d`	`37722` ns	`37570` ns	`1.00`
`array/reductions/reduce/Int64/dims=1`	`43023` ns	`42862` ns	`1.00`
`array/reductions/reduce/Int64/dims=1L`	`89076` ns	`89319` ns	`1.00`
`array/reductions/reduce/Int64/dims=2`	`61441` ns	`61437` ns	`1.00`
`array/reductions/reduce/Int64/dims=2L`	`87669` ns	`86880` ns	`1.01`
`array/reverse/1d`	`15892` ns	`16128` ns	`0.99`
`array/reverse/1dL`	`70093` ns	`70115` ns	`1.00`
`array/reverse/1dL_inplace`	`67575` ns	`67602` ns	`1.00`
`array/reverse/1d_inplace`	`9285` ns	`9293` ns	`1.00`
`array/reverse/2d`	`21299` ns	`21118` ns	`1.01`
`array/reverse/2dL`	`74434` ns	`74528` ns	`1.00`
`array/reverse/2dL_inplace`	`67577` ns	`67778` ns	`1.00`
`array/reverse/2d_inplace`	`10637` ns	`10676` ns	`1.00`
`array/sorting/1d`	`2840009` ns	`2846981` ns	`1.00`
`array/sorting/2d`	`1086677` ns	`1088944` ns	`1.00`
`array/sorting/by`	`3393201` ns	`3400388` ns	`1.00`
`cuda/synchronization/context/auto`	`7823.25` ns	`6766.6` ns	`1.16`
`cuda/synchronization/context/blocking`	`892.9130434782609` ns	`793.7692307692307` ns	`1.12`
`cuda/synchronization/context/nonblocking`	`7865.25` ns	`6805.4` ns	`1.16`
`cuda/synchronization/stream/auto`	`710.0851063829788` ns	`691.0939597315436` ns	`1.03`
`cuda/synchronization/stream/blocking`	`935.2608695652174` ns	`868.875` ns	`1.08`
`cuda/synchronization/stream/nonblocking`	`8297.666666666666` ns	`7338.5` ns	`1.13`
`integration/byval/reference`	`148047` ns	`147882` ns	`1.00`
`integration/byval/slices=1`	`149055` ns	`149033` ns	`1.00`
`integration/byval/slices=2`	`291170` ns	`291277` ns	`1.00`
`integration/byval/slices=3`	`433787` ns	`434160` ns	`1.00`
`integration/cudadevrt`	`105308` ns	`105116` ns	`1.00`
`integration/volumerhs`	`9208418` ns	`9206594` ns	`1.00`
`kernel/indexing`	`11818` ns	`11855` ns	`1.00`
`kernel/indexing_checked`	`12323` ns	`12209` ns	`1.01`
`kernel/launch`	`2463.222222222222` ns	`2433` ns	`1.01`
`kernel/occupancy`	`926.6551724137931` ns	`1008.8235294117648` ns	`0.92`
`kernel/rand`	`15009` ns	`14899` ns	`1.01`
`latency/import`	`4302229477` ns	`4305717656` ns	`1.00`
`latency/precompile`	`5080104774` ns	`5095420673` ns	`1.00`
`latency/ttfp`	`4784129499` ns	`4793974717` ns	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

codecov · 2026-10-03T21:21:21Z

Codecov Report

❌ Patch coverage is 89.70223% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.93%. Comparing base (ae637e5) to head (0151118).

Files with missing lines	Patch %	Lines
CUDACore/src/resources/pools.jl	34.78%	30 Missing ⚠️
CUDACore/src/resources/backends.jl	90.05%	19 Missing ⚠️
CUDACore/src/resources/pins.jl	89.00%	11 Missing ⚠️
CUDACore/src/resources/lifecycle.jl	94.96%	7 Missing ⚠️
CUDACore/src/resources/streams.jl	93.33%	4 Missing ⚠️
CUDACore/src/memory.jl	75.00%	3 Missing ⚠️
CUDACore/src/resources/managed.jl	94.00%	3 Missing ⚠️
lib/cusparse/src/helpers.jl	91.30%	2 Missing ⚠️
CUDACore/lib/cudadrv/graph.jl	83.33%	1 Missing ⚠️
CUDACore/lib/cudadrv/memory.jl	92.30%	1 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3318       +/-   ##
===========================================
+ Coverage   40.06%   84.93%   +44.86%     
===========================================
  Files         187      193        +6     
  Lines       18785    19126      +341     
===========================================
+ Hits         7527    16245     +8718     
+ Misses      11258     2881     -8377

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

compute-sanitizer doesn't treat a successful cuEventQuery as synchronization (#3346), so it reports use-after-free errors for memory that is released after waiting for an event, e.g., when releasing memory last used on another stream. Synchronizing an event that has completed doesn't block, unless it has been recorded again in the meantime, and takes about 70 ns.

Separates the code that manages the lifetime of memory, i.e., the memory pools, the Managed wrapper, pool_alloc and pool_free, reclaiming, and pinning, from the rest of memory.jl. No functional changes.

The test left its worker on another device, which affected the tests that ran on that worker later on.

Synchronizing in a non-blocking way polls for completion, which doesn't flush the device's printf buffer. So the diagnostic output of a kernel that threw an exception was only printed by a later blocking call, e.g. when freeing memory or at exit, after the exception had been reported. Synchronize an event that was never recorded, which flushes the buffer without waiting for any work, also when another stream is still busy. Relax the capture mode for that call, so that reporting an exception from an earlier kernel doesn't invalidate a capture.

When registering failed, e.g. because the memory was registered already, the pin count had been incremented and the object recorded, so pinning the object again returned early without registering it. Also forget the pin count when unpinning, and only register a finalizer the first time an object is pinned: re-pinning a resized array registered another one, which unpinned the memory a second time.

Finalizers run on whatever thread triggers a collection, and several of the driver calls CUDA.jl made there (cuMemFree without a pool, cuMemFreeHost, cuMemHostUnregister, cuArrayDestroy, cuModuleUnload) wait for all running kernels to finish, also blocking kernel launches from other threads meanwhile. If the running work depends on a task on the collecting thread, that deadlocks. Finalizers now only push the resource onto a lock-free list, and regular tasks release it: when allocating (a bounded number of them), around synchronization, and once a second. Device memory is freed in stream order, also when it wasn't allocated from a pool. Releases that may wait for the GPU (freeing host and unified memory, unregistering host memory, freeing memory without stream-ordered support, unloading modules, destroying texture arrays) are held until memory is reclaimed, i.e., when running out of memory or calling reclaim(), so the task that trims an idle pool doesn't reclaim anymore. Owners of wrapped host memory are kept alive until the GPU has finished using it. Releases that fail keep their resource alive instead of being retried, as they may have partially succeeded. Memory is often released after the stream it was last used on has been collected, so streams record a final event before being destroyed, and memory released afterwards waits for it on a non-blocking stream that every context gets when it is first used. Memory last used on the per-thread default stream, which can't be waited for from another thread, is only released when reclaiming memory, after synchronizing the context it was used in (which Managed now records). Every finalizer in CUDACore is registered through resource_finalizer, which is public so that packages wrapping library objects can do the same. Handles returned to a HandleCache stay reusable until memory is reclaimed, as destroying them may wait for the GPU. No lock is held while collecting garbage, which keeps finalizers from running, or while waiting for the GPU, as that work may depend on a task that needs the lock. Releasing resources and starting a graph capture exclude each other using short critical sections; a capture fails instead of waiting while reclaim() synchronizes the device or releases resources that may wait for the GPU.

Freeing host and unified memory waits for all running kernels to finish, so it is only done when memory is reclaimed, which made allocating such memory slow and its usage grow until running out. Where supported (CUDA 13+), allocate it from stream-ordered memory pools instead, which are freed in stream order like device memory. Otherwise, e.g. on Jetson devices, cache freed allocations, rounded up to a size class, and reuse them once an event recorded after their last use has completed. Cached memory is released when memory is reclaimed. Memory imported using unsafe_wrap(; own=true) is never reused, and is freed with cuMemFreeAsync if it was allocated from a pool. JULIA_CUDA_MEMORY_POOL=none also disables these pools. Copies between pointers now use cuMemcpyAsync, as cuMemcpyDtoDAsync reads memory from a host pool when the copy is enqueued instead of in stream order. Memory from a host pool reports the device of the pool's location, so only use a peer copy between actual device memory.

Register the finalizers of library handles, plans, descriptors and helper objects with resource_finalizer, so they don't call into the libraries from a finalizer. Destructors known not to wait for running kernels (returning handles to their cache, cuDNN tensor descriptors, cuSPARSE matrix and vector descriptors) run when retired resources are released; the others (e.g. cuRAND generators, cuSOLVER Mg handles, cuSPARSE analysis infos, cuTENSOR plans) when memory is reclaimed, in the context they were created in. cuTENSOR plans are now destroyed before releasing their workspace, and reclaiming memory destroys the cuRAND generators in the cache it purges instead of leaving them for a later collection. Drop a redundant finalizer on the pointer array of the batched cuBLAS getrf wrapper.

Since finalizers only retire resources (#3318), a resource can be released after its context has been destroyed, e.g., when leaving a `CuContext(dev) do ... end` block. The release then activated the dead context, which crashes inside the driver. Destroying a context already released its resources, and their handles may since have been reused, so drop them instead, as CUDA.jl did before `skip_destroyed` was removed. Host-side bookkeeping (memory accounting, leases, pin counts) is still updated. Some resources outlive the context they were created in: graphs, executable graphs, and allocations from a memory pool. Those are released in another, live context. Allocations made by graphs outlive it too, but the device isn't known when freeing them, so those still leak. Fixes #3403

maleadt added this pull request to stack #3319 October 1, 2026 11:24

maleadt mentioned this pull request Oct 1, 2026

Use GPUToolbox's cooperative_wait for synchronization #3315

Merged

Base automatically changed from tb/coopsync to main October 2, 2026 05:43

maleadt mentioned this pull request Oct 3, 2026

Make CUDA graphs usable #3326

Merged

maleadt force-pushed the tb/deferred-release branch from 11a8eed to 55fdcc1 Compare October 3, 2026 14:16

maleadt removed this pull request from stack #3319 October 3, 2026 15:49

maleadt added this pull request to stack #3327 October 3, 2026 15:49

maleadt mentioned this pull request Oct 3, 2026

Recycle the streams of finished tasks #3317

Merged

maleadt mentioned this pull request Oct 4, 2026

Hand off arrays between streams on the GPU instead of the host #3329

Merged

maleadt force-pushed the tb/deferred-release branch from 55fdcc1 to 5e2e801 Compare October 4, 2026 15:07

maleadt changed the title ~~Don't wait for the GPU when freeing memory from finalizers~~ Don't wait for the GPU when releasing resources from finalizers Oct 4, 2026

maleadt force-pushed the tb/deferred-release branch 2 times, most recently from 9e797a7 to 3fdb466 Compare October 5, 2026 00:17

maleadt removed this pull request from stack #3327 October 5, 2026 08:33

maleadt added this pull request to stack #3338 October 5, 2026 08:33

maleadt force-pushed the tb/deferred-release branch 3 times, most recently from 1115dac to 7b3fbb6 Compare October 6, 2026 13:55

maleadt added 8 commits October 7, 2026 13:46

Move memory pool and managed memory code to resources/

b751777

Separates the code that manages the lifetime of memory, i.e., the memory pools, the Managed wrapper, pool_alloc and pool_free, reclaiming, and pinning, from the rest of memory.jl. No functional changes.

Restore the device after testing copies between devices

5efa9f2

The test left its worker on another device, which affected the tests that ran on that worker later on.

maleadt force-pushed the tb/deferred-release branch from 7b3fbb6 to 0151118 Compare October 7, 2026 12:20

maleadt merged commit 57d5520 into main Oct 8, 2026
17 checks passed

maleadt deleted the tb/deferred-release branch October 8, 2026 05:22

This was referenced Oct 8, 2026

Grant all devices access to the unified memory pool #3367

Merged

Don't wake the log task after its condition is finalized #3385

Merged

Poll an event capturing the context's work when synchronizing the device #3396

Merged

This was referenced Oct 16, 2026

Deferred stream release activates a destroyed context #3403

Closed

Don't touch destroyed contexts when releasing their resources #3404

Merged

This was referenced Oct 19, 2026

Freeing memory from a finalizer waits for the GPU JuliaGPU/oneAPI.jl#717

Open

Some finalizers wait for all GPU work JuliaGPU/AMDGPU.jl#1167

Open

Finalizers wait for the GPU, and can deadlock JuliaGPU/OpenCL.jl#569

Open

Navigation Menu

Repository navigation

Don't wait for the GPU when releasing resources from finalizers - #3318

Don't wait for the GPU when releasing resources from finalizers#3318
maleadt merged 8 commits into
mainfrom
tb/deferred-release

maleadt commented Oct 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

maleadt commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CUDA.jl Benchmarks

Uh oh!

codecov Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

maleadt commented Oct 1, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading

codecov Bot commented Oct 3, 2026 •

edited

Loading