Long-Lived Neighbors Determine the Rheological Response of Glasses