Improving the modelling of redshift-space distortions - II. A pairwise velocity model covering large and small scales