Open Question 3.8
Explaining compute-optimal batch sizes.
Open Question 3.8: Explaining compute-optimal batch sizes. Why does the batch size prescription of [McCandlish et al. (2018)] based on an assumption of isotropic, quadratic loss nonetheless predict compute-optimal batch size in a variety of realistic tasks?
This is a discussion page for the open question above. Feel free to share ideas, approaches, or relevant research in the comments below.
Discussion