Adding e2e testing for all mia attacks

[henrikfo]

Description

Adding e2e testing for all mia attacks.
Making sure that if any attack breaking feature is introduced, it should be caught.

------ WiP -----
loss_traj and HSJ are failing, but these issues will be dealt with.

• HSJ will in this PR get an updated MIAResults output
• loss_traj breaks because of ShadowModelHandler.construct_balanced_assignments() breaks seqMIA #373 but will be fixed with Seqmia resolved #360

Resolved Issues

• fixes Thorough LeakPro runtime test. #296

How Has This Been Tested?

Ran the tests that is implemented here.

[henrikfo]

@fazelehh When I updated the return statement for HSJ with MIAResult.from_full_scores(...) it complained that len(true_labels) != len(perturbation_distances), so I updated how the true_labels are created.

[fazelehh]

1. PATCHED_ATTACK_BLOCKERS is empty but the meta-test doesn't warn about it

PATCHED_ATTACK_BLOCKERS = {} # empty
assert all(len(reason) > 0 for reason in PATCHED_ATTACK_BLOCKERS.values()) # vacuously True
This is fine logically, but if someone adds an attack to PATCHED_ATTACKS without a reason, the assertion silently passes. A comment here would help future contributors.

2. seqmia special-case sets input_size=5 with a comment about "5 metrics per trajectory step" , but TinyTimeSeriesTargetModel.input_size=2. Worth confirming seqmia uses the image path (not time-series) and that input_size in its AttackConfig is independent of the model input size.

3. result.id assumed to exist
   result_file = output_dir / "results" / result.id / "result.txt"
   This will fail with an AttributeError if MIAResult doesn't have an id field. Worth adding an assertion like assert hasattr(result, 'id') before using it, or confirming MIAResult always has this.

4.(optional) RMIA uses full population for train+test. A comment explaining why RMIA gets a different split would help reviewers.

[henrikfo]

1. PATCHED_ATTACK_BLOCKERS is empty but the meta-test doesn't warn about it

PATCHED_ATTACK_BLOCKERS = {} # empty assert all(len(reason) > 0 for reason in PATCHED_ATTACK_BLOCKERS.values()) # vacuously True This is fine logically, but if someone adds an attack to PATCHED_ATTACKS without a reason, the assertion silently passes. A comment here would help future contributors.

Removed the "Patched" logic. It's unnecessary all implemented attack now runs the real attack. Might Implement something like this for an e2e for minva instead, where there might exist blockers or hard compute constraints than mia.

2. seqmia special-case sets input_size=5 with a comment about "5 metrics per trajectory step" , but TinyTimeSeriesTargetModel.input_size=2. Worth confirming seqmia uses the image path (not time-series) and that input_size in its AttackConfig is independent of the model input size.

seqmia uses TinyImageInputHandler and not TinyTimeSeriesTargetModel since the LSTM model in the attack is evaluating the loss trajectory and not the data itself.

3. result.id assumed to exist
   result_file = output_dir / "results" / result.id / "result.txt"
   This will fail with an AttributeError if MIAResult doesn't have an id field. Worth adding an assertion like assert hasattr(result, 'id') before using it, or confirming MIAResult always has this.

Fixed! Added a few statements that checks this and makes sure that MIAResult.id is a string exists etc.

4.(optional) RMIA uses full population for train+test. A comment explaining why RMIA gets a different split would help reviewers.

Added comments mentioning the fact that it does sample from the full population. But I assume that you meant why RMIA does sample like this and not like the paper or? Because from what I remember, RMIA could not have train + test sizes = 1. Why was this changed, do you know? I have not really followed that change and now the z, the known-out data is sampled from the whole population