I often pitch the yeast as soon as I reach the highest temperature that is supported (depending on the yeast it can be around 25°C to 28°C).
Many books and articles mention that the wort is most vulnerable when it is cooling down. The heat no longer protects the wort and other bacteria could contaminate it, so it is better to pitch the yeast at high temperature rather than waiting hours before reaching the perfect temperature. Honestly, I could not really tell the difference between a batch where I pitched at high or at a perfect temperature.
Of course the ideal is to get a wort chiller (or ice bath for smaller batches) so it does not take long to reach the perfect temperature. When you become good at brewing and want to improve, little things like this can make a small difference. Although sanitation and good raw materials are much more important in my opinion.
To answer the question, read the maximum temperature supported on the yeast package. Pitch at that temperature if you want, the off flavor will be minimal. Safale s-04 mentions a maximum of 25°C (77°F), so I would follow the instructions. If you think it would be too long to get the wort from 28°C to 25°C, I would think it is better to pitch at, lets say 27°C (close enough) rather than waiting one or two hours. The off flavors will be minimal and most will not notice the difference.