Core implementations to add: practical GP advances

[bwengals]

A list of implementations that would make ptgp more useful as a general-purpose GP toolkit. The goal is broadly useful GP advances, not things hyper-targeted to specific problems. ptgp should also work well as a substrate for people building specialized models on top, but the core should cover what practitioners should reach for more often.

Objectives

• LOO-CV (Leave-One-Out Cross-Validation) — analytic for exact GPs with Gaussian noise. Penalizes overfitting more aggressively than MLL. Useful for kernel model selection without a validation set.
• Predictive log likelihood (PLL) for SVGP — Jankowiak et al. (2020). Uses log E[p(y|f)] instead of E[log p(y|f)]. Tends to produce better-calibrated predictive variances than the standard ELBO.

Sparse / structured approximations

• Spherical harmonic features — Dutordoir et al. (2020). Inter-domain inducing variables on the unit hypersphere. Diagonal Kuu, so solves are elementwise division. Multi-dimensional counterpart to VFF.
• Iterative solvers (CG, Lanczos) — the GPyTorch approach: use conjugate gradients for solves and stochastic trace estimators for log determinants instead of Cholesky. Scales to much larger N than exact methods. How this should work in ptgp is an open question: should PyTensor rewrites automatically choose between Cholesky and CG based on problem size, or should this be a user-facing choice? See Speed and memory optimizations: rewrites, inplace ops, and gradient graphs #9 for related discussion on rewrites.

What else?

What other broadly useful GP methods or features are missing?

[jessegrabowski]

I usually associate iterative solvers like CG with sparse systems, is that actually the case here?