Transfer of Observations to CES (particular focus on high-dimensional covariances)

[odunbar]

Purpose

• Closes Extend new data processing tools to handle the high-dimensional output covariance structures from EKP #366

To-do

• should we return decoded structure matrices as matrices? (currently done), or leave them as LinearMap objects

Content

For dimension reduction

• Add encoder_kwargs_from(...) utility to easily get the obs_noise_cov,observation, prior_cov etc. for dimension reduction in the correct forms (extensible for the likelihood informed quantities)
• Conversion of all StructureMatrix objects into LinearMap objects.
• Change svd into truncated/iterative svd operations for application to Linear Maps. Implement new total-variance estimation to truncate with retain_var in situations where not all s.v.s are computed
• Ensure that StructureMatrices continue to be LinearMaps under repeated application
• Added partial tests that show framework can be applied to (lower-rank) 10^6 dimensional covariances on observations.
• Adds TSVD, LowRankApprox, LinearMaps to project svds for larger matrices
• When interfacing with an emulator for regularlization the encoded structure matrices are converted to matrices again
• When decoding, (e.g. in predict(...,transform_to_real=true)) the encoded structure matrices are decoded and provided back as full matrices

For MCMC

• Allow passing of Observations, and ObservationSeries objects to MCMC
• Unit tests

---

• I have read and checked the items on the review checklist.

[codecov]

Codecov Report

❌ Patch coverage is 94.62810% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.87%. Comparing base (5fdb164) to head (a3779ff).

Files with missing lines	Patch %	Lines
src/Utilities.jl	91.37%	10 Missing ⚠️
src/Utilities/decorrelator.jl	96.10%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #384      +/-   ##
==========================================
- Coverage   94.15%   93.87%   -0.28%     
==========================================
  Files          10       10              
  Lines        1624     1812     +188     
==========================================
+ Hits         1529     1701     +172     
- Misses         95      111      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.