I’ve been trying to wrap my head around this part and what it means for cause and effect for a while. I think Feynman said something like a photon is only ever emitted when the source and destination agree to exchange one. Which makes sense if the exchange is instantaneous to the photon. But how can billions of years pass for us in the mean time?

To be pedantic, photons never accelerate. They only ever travel at one speed in one direction