This experiment evaluates different logistic regression models (including our proposed LIT-LVM variants) on real-world datasets using interaction terms.
Make sure the following dependencies are installed:
pip install torch scikit-learn pandas matplotlibEnsure the following files are available in the repository:
utils.py(should defineGridSearch)dataset.py(should definedata_loader)- Model file:
logreg_LIT_LVM_V1.py
python logreg_experiment.py --dataset <dataset_name> --model <model_name> --d <latent_dim> --lr <learning_rate> --note "<your notes>"--dataset: Name of the dataset file (e.g.,tecator.arff)--model: One of the following:litlvmV1elasticnet: Elastic Net baseline
--d: Latent dimension (default: 2)--lr: Learning rate (default: 0.05)--note: Optional note to describe the experiment setup
python logreg_experiment.py --dataset tecator.arff --model litlvmV2 --d 2 --lr 0.01 --note "Running LIT-LVM v2 with d=2"You can tune the following parameters directly within the script:
| Parameter | Variable Name | Location in Script |
|---|---|---|
| Latent dimension | d |
Set via CLI and used throughout |
| Learning rate | learning_rate |
Set via --lr argument |
| Number of epochs | num_epochs |
Defined near top of script |
| Number of folds | folds |
Defined near top of script |
| Number of experiments | num_experiments |
Defined near top of script |
| Early stopping tol | es_threshold |
Defined near top of script |
| Train/val/test sizes | train_size, val_size, test_size |
Defined near end of script |
| Hyperparam grid | param_grid_1, param_grid_2 |
Defined inside the script |
- Use
param_grid_1for LIT-LVM models (model_1, model_2). - Use
param_grid_2for Elastic Net variants (model_3).
These control the search over alpha, kappa, gamma, etc.
Results are automatically saved in a text file under the specified output folder, with a name like:
<dataset>_model:<model_name>_lr:<learning_rate>_d:<d>_tol:<tol>_folds:<folds>.txt
The output includes the average AUC scores for each repetition and the best hyperparameters selected during tuning.
These experiments evaluate different linear regression models on real datasets with interaction terms.
The setup is similar to the logistic regression experiments, but the script and models are tailored for regression tasks.
Ensure the same dependencies and structure as the logistic regression experiment are available, including:
utils.py(definesGridSearch)dataset.py(definesdata_loader)- Model files:
linreg_LIT_LVM_V1.py
python linreg_experiment.py --dataset <dataset_name> --model <model_name>--dataset: Name of the dataset file (e.g.,tecator.arff)--model: Choose one of the following:litlvmV1elasticnet: Elastic Net baseline
python linreg_experiment.py --dataset tecator.arff --model litlvmV1As with the logistic regression script, the following parameters can be modified within linreg_experiment.py:
| Parameter | Variable Name | Description |
|---|---|---|
| Latent dimension | d |
Dimension of latent embedding |
| Learning rate | learning_rate |
Learning rate |
| Number of epochs | num_epochs |
Training epochs |
| Number of folds | folds |
For cross-validation |
| Number of experiments | num_experiments |
Number of repeated runs |
| Early stopping tol | es_threshold |
Patience for early stopping |
| Train/val/test sizes | train_size, val_size, test_size |
Data splitting ratios |
| Hyperparam grid | param_grid_1, param_grid_2 |
For tuning LIT-LVM or Elastic Net |
- Use
param_grid_1for LIT-LVM models. - Use
param_grid_2for Elastic Net models.
Results will be saved under:
experiments/results/Linear_Regression/<DATE>/<dataset_name>_model:<model_name>_lr:<learning_rate>_threshold:<es_threshold>_tol:<tol>_folds:<folds>.txt
Each file includes R² scores and RMSE across runs, along with best hyperparameters per trial.
To evaluate SparseFM baselines on real data, we use the same scripts as for the LIT-LVM models:
logreg_experiment.pyfor classification taskslinreg_experiment.pyfor regression tasks
The overall workflow—data loading, model training, evaluation—remains unchanged. The only difference is in the hyperparameter configuration.
Hyperparameter tuning is controlled via param_grid_1 in both scripts. For SparseFM, we fix all other hyperparameters and only search over values of gamma, which corresponds to the regularization strength on the latent structure in our paper.
To enforce SparseFM, we use very high values of this regularization strength:
'gamma': [1e5, 1e6, 1e7, 1e8]gammain the code corresponds to the latent structure regularization strength discussed in the paper.- No modifications to the script are needed—only the
param_grid_1setting must be adjusted.
To evaluate HierNet baselines on real data, we use R scripts specifically designed for hierarchical interaction models:
logreg_experiment.Rfor classification taskslinreg_experiment.Rfor regression tasks
These scripts implement the HierNet model which learns hierarchical interactions between features using a structured regularization approach.
We perform simulation experiments to evaluate the performance of LIT-LVM, ElasticNet, and Factorization Machines (FM) under controlled conditions. Each method has its own experiment file:
sim_logreg_experiment_litlvm.py— for LIT-LVMsim_logreg_experiment_elasticnet.py— for ElasticNetsim_logreg_experiment_FMs.py— for Factorization Machines
All experiments use the same simulation class (LogisticRegression_Simulator) and share a similar procedure to the real data experiments. The key difference is that data is synthetically generated based on a known low-rank structure with configurable noise and sparsity.
Each script supports CLI arguments to control the simulation setup:
python sim_logreg_experiment_<model>.py --p <feature_dim> --d <latent_dim> --V_noise <noise_level> --sparsity <0_or_1> --noise <0_or_1> --interaction <0_or_1> --sigma <sparsity_strength>Example for LIT-LVM:
python sim_logreg_experiment_litlvm.py --p 60 --d 2 --V_noise 0.01 --sparsity 1 --noise 1 --interaction 1 --sigma 0.0001 Example for FM:
python sim_logreg_experiment_FMs.py --p 60 --d 5 --V_noise 0.01 --sparsity 0 --noise 1 --interaction 1 --sigma 0.0001 --gamma 1e6Example for ElasticNet:
python sim_logreg_experiment_elasticnet.py --p 60 --d 2 --V_noise 0.01 --sparsity 1 --noise 1 --interaction 1 --sigma 0.0001The following values can be manually changed from within each script (not from command-line arguments):
| Parameter | Variable Name | Description |
|---|---|---|
| Training samples | num_samples |
Number of samples per train/val/test set |
| Learning rate | learning_rate |
Step size for gradient descent |
| Number of epochs | num_epochs |
Training duration |
| Early stopping | es_threshold |
Number of epochs without improvement to stop |
| Number of splits | data_split loop |
Number of independent train/val/test runs (set to 5 by default) |
| Model depth (FM, LIT-LVM) | d |
Latent dimension, also varies across models internally (2, 5, 10) for FM/LIT-LVM |
| Param grid | param_grid |
Hyperparameter grid for model selection, including alpha, kappa, gamma |
Regularization strength (gamma) |
gamma (LIT-LVM) |
Controlled via CLI and used to penalize latent structure |
- In LIT-LVM, we search over values of
gamma, corresponding to latent structure regularization strength. - In ElasticNet,
gammais fixed at 0 (no latent structure penalty). - In FM, the dimension
dis varied manually`.
Each script logs results (AUC, MSE of coefficients) to a file like:
experiments/results/simulations/logistic_regression/<DATE>/<MODEL>_p:<p>_d:<d>_lr:<lr>_Noise:<...>.txt
The logs include:
- AUC on held-out test set
- MSE between true and estimated coefficients (
w) and interaction matrices (V) - Standard errors across runs
# Example output summary in file
result_2: [0.91, 0.89, 0.92, ...], mean: 0.905, SE: 0.012
result_2_w_MSE: [...], result_2_V_MSE: [...]- Use different scripts for each baseline.
- Set model and experiment type via CLI.
- Modify training/control settings inside the script.
- Results are automatically logged with reproducible configurations.
The linear regression simulations follow exactly the same structure as the logistic regression experiments. The only difference is that they use regression-specific simulators and evaluation metrics.
The corresponding scripts are:
sim_linreg_experiment_litlvm.pysim_linreg_experiment_elasticnet.pysim_linreg_experiment_FMs.py