Measuring the growth of structure by matching dark matter haloes to galaxies with VIPERS and SDSS