Duality invariance in Fayet-Iliopoulos gauged supergravity