Mastering Batch Size, Gradient Accumulation, and Throughput in LLM Training

Stop Guessing Your Hyperparameters

You fire up your training script. The loss curve looks promising for the first few steps. Then, suddenly, it crashes. Or worse, it finishes in three days instead of three hours because your GPUs are sitting idle, waiting for data that isn't there. If you have ever stared at a CUDA Out-of-Memory (OOM) error while trying to train a large language model, you know the frustration.

The problem is rarely the code itself. It is usually how you handle batch size, gradient accumulation, and throughput. These three concepts are not just settings; they are the engine of your training pipeline. Get them right, and you save thousands of dollars in compute costs. Get them wrong, and you waste weeks of time.

In this guide, we will break down exactly how these pieces fit together. We will look at the math behind effective batch size, why gradient accumulation is your best friend when memory runs out, and how to tune your hardware for maximum tokens per second.

Quick Summary / Key Takeaways

• Effective Batch Size is the total number of samples processed before an optimizer update, calculated as: Micro-batch Size × Gradient Accumulation Steps × World Size.
• Gradient Accumulation allows you to simulate large batches on limited GPU memory by summing gradients over multiple smaller micro-batches before updating weights.
• Throughput should be measured in tokens per second. Aligning your micro-batch size with GPU hardware tiles (e.g., multiples of 16 or 32) can boost utilization from 60% to 90%+.
• Tuning Strategy: First, pick an effective batch size for convergence quality. Then, maximize your micro-batch size until you hit memory limits, using gradient accumulation to bridge the gap.

Decoding the Effective Batch Size Formula

When people talk about "batch size" in LLM training, they often mean different things. This confusion leads to inconsistent results across different teams and frameworks. To fix this, we need to separate the concept into three distinct parts: micro-batch size, gradient accumulation steps, and world size.

Micro-batch size is the actual number of examples loaded onto a single GPU at one time. This is constrained by your GPU's VRAM. If you try to load too many tokens, you get an OOM error.

World size refers to the number of parallel processes or GPUs working together. In a typical setup, this equals the number of GPUs you have available for data parallelism.

Gradient accumulation steps is the multiplier. It tells the system how many micro-batches to process before performing a single weight update.

The magic happens when you combine them. The formula for effective batch size is:

Effective Batch Size = Micro-batch Size × Gradient Accumulation Steps × World Size

For example, if you have 8 GPUs (World Size = 8), a micro-batch size of 4, and 4 accumulation steps, your effective batch size is 128. This means the optimizer updates the model weights after seeing 128 unique samples. This decomposition is standard in frameworks like DeepSpeed and Megatron-LM.

Why Gradient Accumulation Is Essential

You might wonder: "Why not just increase the micro-batch size?" Because memory is finite. Large language models consume massive amounts of VRAM for activations, gradients, and optimizer states. Even on powerful cards like the NVIDIA H100 or A100, fitting a huge batch directly into memory is often impossible.

This is where gradient accumulation shines. Instead of loading all 128 samples at once, you load 4. You calculate the gradients for those 4. You store those gradients in memory. Then you load the next 4, calculate their gradients, and add them to the stored ones. You repeat this four times. Only then do you average the accumulated gradients and update the model weights.

The key insight here is memory efficiency. During the forward and backward passes, your GPU only needs to hold the activations for the small micro-batch (4 samples). The memory footprint remains low, even though the mathematical effect on the model is identical to processing 128 samples at once.

However, there is a trade-off. Storing accumulated gradients does take some extra memory, but it is significantly less than storing activations for a full large batch. Frameworks like Axolotl and DeepSpeed optimize this by averaging gradients locally during each step and performing a single collective communication (all-reduce) at the end. This reduces network traffic between GPUs, which is crucial for multi-node clusters.

Maximizing Throughput: Tokens Per Second

Batch size isn't just about memory; it's about speed. In LLM training, we measure performance as tokens per second. Higher throughput means lower cost per million tokens trained.

Here is a counterintuitive fact: simply increasing batch size doesn't always increase throughput linearly. It depends on your GPU architecture. Modern GPUs, particularly those with Tensor Cores like the NVIDIA H100, process data in specific matrix tile sizes. For FP16 operations, these tiles are often 16x16 matrices.

If your batch size is 127, the GPU has to handle partial tiles, leading to inefficient computation. Research from 2026 shows that on an H100, a batch size of 127 might yield only 61% GPU utilization. But bumping that batch size to 128-a multiple of 16-can jump utilization to 94%. That is a massive difference in speed and cost.

To find your sweet spot, follow this practical tuning method:

1. Start with a micro-batch size of 1.
2. Run a single training step and measure the tokens per second.
3. Double the batch size (1 → 2 → 4 → 8 → 16).
4. Repeat until you either hit an OOM error or see throughput stop increasing.
5. Select the largest batch size that gives you the highest tokens per second without crashing.

Once you have your optimal micro-batch size, use gradient accumulation to reach your desired effective batch size. This approach separates hardware constraints from optimization requirements.

Framework-Specific Nuances: Megatron-LM vs. DeepSpeed

Not all frameworks handle batch size and accumulation the same way. Understanding these differences prevents configuration errors.

Megatron-LM uses pipeline parallelism extensively. In this context, micro-batches are used to keep the pipeline stages busy. When using Megatron-LM through libraries like Hugging Face Accelerate, you often set gradient_accumulation_steps to 1. Why? Because the pipeline parallelism itself handles the batching logic. The framework manages a global_batch_size parameter that distributes work across pipeline stages. Setting accumulation steps incorrectly here can lead to double-counting or misaligned gradients.

DeepSpeed, on the other hand, treats gradient accumulation as an explicit feature. It offers advanced optimizers like LAMB that are designed for large effective batch sizes. DeepSpeed also optimizes communication by delaying the all-reduce operation until after all micro-batches are processed. This makes it highly efficient for multi-node setups where network bandwidth is a bottleneck.

Axolotl simplifies this for fine-tuning workflows. It clearly distinguishes between the memory-bound micro-batch and the quality-driven effective batch. Its documentation emphasizes that for diverse datasets, gradient accumulation does not negatively impact learning dynamics, making it a safe tool for achieving higher throughput.

Small Batch vs. Large Batch: The Optimization Debate

There is ongoing debate in the research community about whether larger batches are always better. Traditional wisdom suggests that large batches provide smoother gradient estimates, leading to more stable convergence. However, recent studies in 2025 and 2026 challenge this view.

Some researchers argue that very large effective batches can actually harm generalization, especially if the learning rate is not scaled correctly. They propose that smaller micro-batches, combined with careful tuning, can achieve similar or better results with higher throughput. The argument is that small batches introduce beneficial noise into the optimization process, helping the model escape local minima.

The practical takeaway? Don't blindly chase the largest possible effective batch size. Start with a range typically seen in successful fine-tuning jobs, such as 16 to 128. Monitor your validation loss. If a smaller effective batch size converges just as well but allows for a larger micro-batch (and thus higher throughput), choose the smaller effective batch. Efficiency matters.

Common Pitfalls and How to Avoid Them

Even experienced engineers make mistakes when configuring these parameters. Here are the most common issues:

• Ignoring Learning Rate Scaling: When you increase the effective batch size via gradient accumulation, you may need to adjust your learning rate. A common rule of thumb is to scale the learning rate linearly with the batch size, but this is not always necessary for fine-tuning. Always monitor loss curves for instability.
• Misconfiguring Pipeline Parallelism: In Megatron-LM, forgetting that micro-batches serve as pipeline units can lead to severe underutilization. Ensure your microbatch_group_size_per_vp_stage is aligned with your pipeline depth.
• Overlooking Hardware Alignment: Using batch sizes that are not multiples of your GPU's tensor core dimensions (like 16 or 32) wastes compute power. Always check your GPU's technical specifications for optimal tile sizes.
• Single-Device Accumulation Overhead: On a single GPU, gradient accumulation adds memory overhead for storing gradients without reducing communication costs. If you are not memory-bound, consider increasing the micro-batch size directly instead of accumulating.

Next Steps for Your Training Pipeline

Optimizing batch size, gradient accumulation, and throughput is an iterative process. Start by profiling your current setup. Measure your tokens per second and GPU utilization. Identify if you are memory-bound or compute-bound.

If you are memory-bound, reduce your micro-batch size and increase gradient accumulation steps. If you are compute-bound, try aligning your micro-batch size with hardware tile sizes. Use tools like NVIDIA Nsight Systems to visualize bottlenecks.

Remember, the goal is not just to finish training, but to do so efficiently. By mastering these three levers, you gain control over both the quality and the cost of your LLM projects.