I'll try to present both sides of the story:
If the tree is fully cleaned, and your sanitizer is sufactant-based (such as StarSan) so that kills organisms on contact then maybe (and only maybe) you can get away without sanitizing. That's about as far as you can guess as to the consequences of not sanitizing the brew tree.
If it's not clean, then forget it. Each dust particle harbours enough acetobacter to make a very nice vinegar from your batch of beer.
Whether it works or doesn't will very much depend upon your environment. Some people feverishly clean their equipment before and after use and store in a sealed container, some don't. If someone says it works, do you know their processes, cleaning routines? I wouldn't base any judgement other than "do it, don't take chances" on someone's anecdotal evidence that they successfully brewed a beer without cleaning. And how do you know the beer was up to your standard? I can smell a contaminant almost before I've lifted the class, and wouldn't take the chance at all.)
Personally, a quick spray with starsan is simple enough to do that I choose not to leave it out. Why take the chance of a waste of a day of brewing for a few seconds of additional sanitation?