Ideally, you want the liquid portion of the starter to be crystal clear, meaning no yeast is left in suspension. In this scenario, you carefully pour the liquid off the sediment, leaving a enough to swirl around, bringing the sediment up into suspension, and then pitch.
If the starter is small (1 or 2 quarts), and the yeast hasn't settled out completely, you can just swirl the whole thing and pitch the entire contents. It will increase your beer's volume slightly, but not enough to notice. If the starter is large (1 or 2 gallons, say), then you're better off waiting until all the yeast has settled out and following the procedure above.
You're doing the right thing keeping the starter at a very low temperature as the yeast will drop out quicker, but be sure to warm it to the same temperature as the wort before pitching. Otherwise, you risk shocking the yeast, which may cause it to go dormant and not initiate the fermentation.