Improving RCT-Based Treatment Effect Estimation Under Covariate Mismatch via Calibrated Alignment

Technical Deep Dive: Improving RCT-Based Treatment Effect Estimation Under Covariate Mismatch via Calibrated Alignment

Overview

• This work proposes a novel framework called "Calm" for improving conditional average treatment effect (CATE) estimation in randomized controlled trials (RCTs) when there is covariate mismatch between the RCT and a larger observational study (OS).
• The key idea is to learn encoders that map the heterogeneous feature spaces of the RCT and OS into a shared low-dimensional representation space, rather than attempting to impute the missing covariates.
• This embedding-based approach is shown to outperform imputation-based baselines in settings with nonlinear CATEs and low-dimensional outcome-relevant information.

Problem & Context

• RCTs are the gold standard for unbiased CATE estimation, but are often underpowered for detecting effect heterogeneity.
• Observational studies (OS) can supplement RCTs, but a key barrier is covariate mismatch - the RCT and OS measure different, only partially overlapping covariates.
• Prior work has addressed this using imputation to fill in the missing covariates, but this can be an unnecessarily difficult problem.

Methodology

Key Innovations

1. Embedding Alignment Framework: Calm replaces imputation with learned embeddings that map the heterogeneous feature spaces of the RCT and OS into a shared low-dimensional representation space.
2. Finite-Sample Risk Bounds: Calm's risk bounds decompose the error into alignment, sufficiency, complexity, and calibration terms, making explicit when embedding alignment outperforms imputation.
3. Theoretical and Empirical Validation: Calm is validated across 51 simulation settings and an IHDP semi-synthetic benchmark, confirming the predictions of the risk bounds.

Calm Algorithm

Calm has four stages:

1. Learn OS outcome models in the OS embedding space.
2. Learn the RCT encoder and calibrate the OS outcome models to the RCT embedding space.
3. Construct calibrated pseudo-outcomes using the RCT embeddings.
4. Estimate a CATE correction using the calibrated pseudo-outcomes.

Data & Experimental Setup

• Simulated data with varied factors: imputation difficulty, intrinsic dimension, RCT sample size, outcome nonlinearity, outcome shift, and shared covariate proportion.
• Also evaluated on an IHDP semi-synthetic benchmark.
• Compared to baselines: Naive, Racer, SR-Oscar, MR-Oscar, Calm-Lin, Calm-NN, HTCE-T, HTCE-DR.

Results

Linear CATE Regime

• In the linear CATE regime, the four calibration-based methods (Racer, SR-Oscar, MR-Oscar, Calm-Lin) perform equivalently, with negligible differences in mean RMSE.
• The identity of the best calibration-based method shifts as the factors are varied, but the gaps are small (< 10^-3 RMSE).

Nonlinear CATE Regime

• In the nonlinear CATE regime, Calm-NN outperforms all other methods, including the calibration-based baselines.
• Calm-NN maintains low RMSE (< 0.8) even at small RCT sample sizes (n^r = 100), where the calibration-based methods degrade sharply.
• Calm-NN's advantage persists across different CATE functional forms and is robust to the strength of the shared covariate signal.

Interpretation

• The results confirm the theoretical predictions: when the outcome-relevant information lies on a low-dimensional manifold and the CATE is nonlinear, learned embeddings can outperform imputation-based approaches.
• The calibration-based methods are effectively equivalent in the linear CATE regime, validating the importance of the calibration mechanism rather than the specific OS-borrowing strategy.
• Calm-NN's advantage in the nonlinear regime stems from its ability to model complex CATEs, not from exploiting information in the unobserved covariates that is inaccessible to imputation.

Limitations & Uncertainties

• The alignment quality r_φ^2 is not directly observable in practice, so it must be controlled indirectly through the alignment objective.
• The analysis relies on Lipschitz continuity assumptions, which may not always hold in practice.
• Calm-NN's negative-transfer protection is not as complete as the linear variant, Calm-Lin, under severe distributional shift.

What Comes Next

• Extensions to multi-source settings, formal inference via debiased ML, and real-world clinical applications.
• Investigating alternative alignment objectives and techniques to further improve the robustness of the nonlinear embedding approach.