- vanilla gradient descent
- Newton’s method
- damped Newton’s method
- conjugate gradient descent
- momentum
- Nesterov accelerated gradient
- Adagrad
- Adadelta/RMSprop
- Adam
Functions (optimization and visualization) are all implemented in gradientDescent.R . To run the demo, use RStudio to see the visualized graphs and charts, and tweak the parameters.