Every batch of homebrew is contaminated. You can't hope to have zero airborne contaminants in your wort especially with it just sitting there mostly open during the chill process. (depending on how you chill).
That said brewers yeast loves wort. Brewers yeast starts to create an undesirable environment for wild yeasts and bacteria. Primarily through lowering the pH, then buy using up nutrients and lastly creating alcohols. These things do not necessarily kill wild contaminants, but they create a static growth environ for them. The use of hops also has a bacteria-static force on the contaminants too.
So your wort is contaminated. And what little contaminants are in there, they do start to multiply early on, just as the yeast are trying to as well. But my pitching the proper amount of yeast that lag is short and the good things listed above start to happen. Brewers yeast simply wins out.
(Wort is acidic also which tends to help as well)