What is the difference between gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent?

Question

Data Science with Python — Hard

What is the difference between gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent?

Accepted Answer

Gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent differ in the amount of data used per update. While gradient descent uses the full dataset, SGD uses one random sample, and mini-batch uses a subset. SGD converges faster per epoch but with noisy updates, while mini-batch balances noise and computational efficiency.