On the regularity of a graph related to conjugacy classes of groups: small valencies