Benchmark Multiple Variable Selection Methods

Performs comprehensive benchmarking of variable selection methods including gSuSiE, SuSiE, LASSO, and Elastic Net across multiple simulated datasets. Compares coefficient estimation, variable selection performance, and credible set coverage.

Usage

benchmark(
  settings,
  n_sims = 1000,
  n = 500,
  true_theta = c(1, 0),
  intercept = 0,
  family = gaussian(),
  coverage = 0.95,
  L = 10,
  standardize = TRUE,
  decompose = TRUE,
  shrinkage = TRUE,
  tol = 0.05,
  max_iter = 100,
  ties = "efron",
  lambda = 0,
  tau = 0.5,
  cor_threshold = 0.5,
  include_lasso = TRUE,
  include_enet = TRUE,
  enet_alpha = 0.5,
  nfolds = 10,
  seed = 42,
  verbose = FALSE,
  parallel = FALSE,
  ...
)

Arguments

settings: Data generation settings passed to the generate function.
n_sims: Number of simulation replicates. Default 1000.
n: Sample size for each simulation. Default 500.
true_theta: Vector of true regression coefficients.
intercept: True intercept value. Default 0.
family: Response family. Can be gaussian(), binomial(), poisson(), Gamma(), or "cox" for Cox regression.
coverage: Nominal coverage level for credible sets. Default 0.95.
L: Maximum number of single effects for gSuSiE and SuSiE. Default 10.
standardize: Logical. Should variables be standardized? Default TRUE.
decompose: Logical. Use matrix decomposition in gSuSiE? Default TRUE.
shrinkage: Logical. Apply shrinkage in gSuSiE? Default TRUE.
tol: Convergence tolerance for gSuSiE. Default 5e-2.
max_iter: Maximum iterations for gSuSiE. Default 100.
ties: Method for handling ties in Cox regression. Default "efron".
lambda: Ridge penalty parameter for gSuSiE. Default 0.0.
tau: Threshold parameter for gSuSiE. Default 0.5.
cor_threshold: Correlation threshold for credible sets. Default 0.5.
include_lasso: Logical. Include LASSO in benchmark? Default TRUE.
include_enet: Logical. Include Elastic Net in benchmark? Default TRUE.
enet_alpha: Elastic net mixing parameter. Default 0.5.
nfolds: Number of cross-validation folds for glmnet methods. Default 10.
seed: Random seed for reproducibility. Default 42.
verbose: Logical. Show progress messages? Default TRUE.
parallel: Logical. Use parallel processing? Default FALSE.
...: Additional arguments passed to the generate function.

Value

A list containing benchmark results for each method:

glmsusie: List with coef_summary, pmp_summary, and cs_summary
susie: List with coef_summary, pip_summary, and cs_summary (Gaussian only)
lasso: List with coef_summary, pip_summary, and cs_summary (if included)
enet: List with coef_summary, pip_summary, and cs_summary (if included)

Each summary contains:

coef_summary: Coefficient estimation performance metrics
pmp_summary/pip_summary: Inclusion probability performance metrics
cs_summary: Credible set coverage and size statistics

Details

This function performs a comprehensive comparison of variable selection methods by running multiple simulations with the same data generation process. For each simulation, it:

Generates data using the specified settings
Fits all requested methods (gSuSiE, SuSiE, LASSO, Elastic Net)
Collects coefficient estimates, inclusion probabilities, and credible sets
Summarizes performance across all simulations

The function automatically handles different response families and excludes SuSiE for non-Gaussian families since it only supports linear regression.

When parallel = TRUE, simulations are run in parallel using the future framework. Progress is tracked with a progress bar when verbose = TRUE and parallel processing is disabled.

Examples

if (FALSE) { # \dontrun{
# Simple benchmark with correlated predictors
settings <- list(
  rho = 0.8,
  block_size = 5
)

# Quick benchmark with few simulations
results <- benchmark(
  settings = settings,
  n_sims = 100,
  n = 200,
  true_theta = c(1, -0.5, rep(0, 18)),
  family = gaussian(),
  verbose = TRUE
)

# Print coefficient estimation results
print(results$glmsusie$coef_summary)
print(results$lasso$coef_summary)

# Logistic regression benchmark
logistic_results <- benchmark(
  settings = settings,
  n_sims = 50,
  n = 500,
  true_theta = c(0.5, -0.3, rep(0, 8)),
  family = binomial(),
  include_lasso = TRUE,
  include_enet = FALSE
)
} # }