Add new problem: Mixed Precision Training - #505
Conversation
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
had some comments on the tests
| @@ -0,0 +1,22 @@ | |||
| [ | |||
| { | |||
| "input": "import numpy as np\nmp = MixedPrecision(loss_scale=1024.0)\nweights = np.array([0.5, -0.3], dtype=np.float32)\ninputs = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)\ntargets = np.array([1.0, 0.0], dtype=np.float32)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.4f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")\ngrads = np.array([512.0, -256.0], dtype=np.float32)\nresult = mp.backward(grads)\nprint(f\"Gradients: {result}\")\nprint(f\"Grad dtype: {result.dtype}\")", | |||
There was a problem hiding this comment.
it should be test and expected_output instead of input and output
| }, | ||
| { | ||
| "input": "import numpy as np\nmp = MixedPrecision(loss_scale=512.0)\nweights = np.array([1.0, 0.5], dtype=np.float64)\ninputs = np.array([[2.0, 1.0]], dtype=np.float64)\ntargets = np.array([3.0], dtype=np.float64)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.1f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")\ngrads = np.array([1024.0, 512.0], dtype=np.float16)\nresult = mp.backward(grads)\nprint(f\"Gradients: [{result[0]:.0f} {result[1]:.0f}]\")\nprint(f\"Grad dtype: {result.dtype}\")", | ||
| "output": "Loss: 128.0\nLoss dtype: float\nGradients: [2 1]\nGrad dtype: float32" |
There was a problem hiding this comment.
the test needs to be simpler for the current way the site works, for would we be able to split up the test cases like this
{
"test": "import numpy as np\nmp = MixedPrecision(loss_scale=1024.0)\nweights = np.array([0.5, -0.3], dtype=np.float32)\ninputs = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)\ntargets = np.array([1.0, 0.0], dtype=np.float32)\nloss = mp.forward(weights, inputs, targets)\ngrads = np.array([1024.0, 512.0], dtype=np.float16)\n\nresult = mp.backward(grads)\nprint(result)",
"expected_output": "[1., 0.5]"
},
{
"test": "import numpy as np\nmp = MixedPrecision(loss_scale=1024.0)\nweights = np.array([0.5, -0.3], dtype=np.float32)\ninputs = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)\ntargets = np.array([1.0, 0.0], dtype=np.float32)\nloss = mp.forward(weights, inputs, targets)\ngrads = np.array([1024.0, 512.0], dtype=np.float16)\n\nresult = mp.backward(grads)\nprint(loss)",
"expected_output": "665.0"
}
There was a problem hiding this comment.
Just fixed, the only difference from your example is that:
- I removed the backward pass code for the forward pass test and vice versa for testing separately the backward pass
- I kept checking dtypes in
expected_outputbecause it's the main testing point for this problem. I assume the system supports this, since I think there are some problems on the website that do check dtypes already?
|
Please review @moe18 , ty! |
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
the test look very good, thanks for the changes, there was an issue with some of the tests check out the comments
| [ | ||
| { | ||
| "test": "import numpy as np\nmp = MixedPrecision(loss_scale=1024.0)\nweights = np.array([0.5, -0.3], dtype=np.float32)\ninputs = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32)\ntargets = np.array([1.0, 0.0], dtype=np.float32)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.4f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")", | ||
| "expected_output": "Loss: 665.0000\nLoss dtype: float32" |
There was a problem hiding this comment.
it seems like using type(loss).name gives float not float32
Loss: 665.0000 Loss dtype: float
There was a problem hiding this comment.
Still, it's float32 on my end
| }, | ||
| { | ||
| "test": "import numpy as np\nmp = MixedPrecision(loss_scale=512.0)\nweights = np.array([1.0, 0.5], dtype=np.float64)\ninputs = np.array([[2.0, 1.0]], dtype=np.float64)\ntargets = np.array([3.0], dtype=np.float64)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.1f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")", | ||
| "expected_output": "Loss: 128.0\nLoss dtype: float32" |
There was a problem hiding this comment.
should be float not float32
There was a problem hiding this comment.
It's float32 for me, as expected
| }, | ||
| { | ||
| "test": "import numpy as np\nmp = MixedPrecision(loss_scale=100.0)\nweights = np.array([0.1, 0.2], dtype=np.float32)\ninputs = np.array([[1.0, 1.0]], dtype=np.float32)\ntargets = np.array([0.5], dtype=np.float32)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.1f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")", | ||
| "expected_output": "Loss: 4.0\nLoss dtype: float32" |
| }, | ||
| { | ||
| "test": "import numpy as np\nmp = MixedPrecision(loss_scale=2048.0)\nweights = np.array([0.25], dtype=np.float64)\ninputs = np.array([[4.0]], dtype=np.float64)\ntargets = np.array([2.0], dtype=np.float64)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.1f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")", | ||
| "expected_output": "Loss: 2048.0\nLoss dtype: float32" |
| }, | ||
| { | ||
| "test": "import numpy as np\nmp = MixedPrecision(loss_scale=256.0)\nweights = np.array([1.0], dtype=np.float16)\ninputs = np.array([[2.0]], dtype=np.float16)\ntargets = np.array([3.0], dtype=np.float16)\nloss = mp.forward(weights, inputs, targets)\nprint(f\"Loss: {loss:.1f}\")\nprint(f\"Loss dtype: {type(loss).__name__}\")", | ||
| "expected_output": "Loss: 256.0\nLoss dtype: float32" |
|
Thanks for the review, @moe18. I'll fix this next month when I'll have access to a computer |
…mixed_precision_training
|
Even though for me the test cases were as expected, I still updated them to match your testing env @moe18 |
|
Hey @moe18, please review, ty! |
moe18
left a comment
moe18
left a comment
There was a problem hiding this comment.
Looks great, thanks for the changes and sorry about the late review
2 participants
Resolves: #480