Deep neural networks are susceptible to adversarial examples, which can lead to incorrect predictions by introducing imperceptible perturbations. Transfer-based attacks create adversarial examples for surrogate models and transfer these examples to victim models deployed in black-box scenarios. Recent studies reveal that adversarial examples in flat loss landscapes can alleviate overfitting on surrogate models and exhibit superior transferability. However, these works ignore the influence of perturbation directions, resulting in limited transferability. To overcome this limitation, this paper proposes a new attack method named Residual Perturbation Attack (ResPA), which employs the residual gradient as the perturbation direction to guide the adversarial examples to search toward the flat regions of the loss function. Specifically, ResPA conducts an exponential moving average operation on the input gradients to obtain the first moment as the referenced gradient, which encompasses the direction information of historical gradients. Moreover, to avoid over-relying on the local flatness, instead of directly using the current gradient as the perturbation direction, ResPA further considers the residual between the current gradient and the referenced gradient, which can capture the changes in the global perturbation direction. Comprehensive experimental comparisons show that ResPA can remarkably enhance adversarial transferability. In addition, ResPA can be naturally combined with existing input transformations to further improve transferability.