Unstoppable gets affected by Default

[nnra6864]

I have this line:
BetterRigidbody.PhysicsLayer = original.BetterRigidbody.PhysicsLayer;
Which copies the physics layer from a prefab with a better rigidbody and it just doesn't seem to work.
I made it so that the Player physics layer shouldn't be affected by the FruitFragment layer and that just doesn't happen.
I made a video displaying the issue, as well as the collision matrix.
https://youtu.be/Aj3-tClchYo

[nnra6864]

Since the video doesn't really show the issue as nicely, let me explain.
Sometimes, when you hit the fruit fragments, instead of the player just ignoring them he gets pushed to the side.
It feels really bad and I am positive it's the fragments that are causing the issue.

[nnra6864]

It might actually be caused by the fruit itself as I fully disabled interactions between fragments and the player, regardless, player still shouldn't be affected as fruit shouldn't have an effect on him.