On the dimension of Voisin sets in the moduli space of abelian varieties