Brownian forces in sheared granular matter