I know if you are adding the yeast directly to the wort you want to keep the temperature similar to that of your eventual fermentation temp to minimize shock to the yeast. However, I plan to make a starter, flocculate the yeast in the fridge, and then pitch the slurry. This being the case, I'm not worried about shocking the yeast with a temp change. Can I up the temp of my starter given this information to encourage faster yeast growth? Will off flavors get into the beer even if I only pitch the slurry and throw out the beer from the starter? Are there other potential problems, and what would be an ideal temp for the starter?
Pitching the slurry is key to your process here. The higher temp will put some less desirable flavors into the starter wort. So crash chilling and pitching the slurry is the best bet.
I think I remember hearing a podcast with Chris White from White Labs say they propagate most yeast (even lager yeasts) at 80F. Most yeast grows best at temps above normal ferment temps. You are on the money with your process.