When a wave goes from a faster material to a slower material at an angle, the side that enters the new material first slows down first, and as a result by the time the whole wave is in the new material it is traveling in a different direction.

When the wave leaves the slower material traveling into the faster material the side that leaves first speeds up first, so when the whole wave has exited the slower material it again changes direction.

On the drawing above the sides of the slower material are parallel so the waves as they leave the material return to the original direction. The second refraction, when the waves changed direction the second time, returned them to their original orientation.

If we follow a particular position on the wave crest as it travels, we have traced the path of the photon that would be following that wave crest. See how the path of the photon changes direction?