You are correct here. The performance is relative to the air mass the wing is sensing, not any fixed point like the ground. To get the best performance to something relatively motionless to the air mass, L/D max would work best. However, since the air and ground are diverting with increasing time, time in the air becomes a factor as well. By spending less time in a headwind, the distance the moving air mass traveled away from the ground is reduced.