I've always bottled my beers (haven't invested in a kegging setup yet), and I've never once used a bottle brush. I simply soak my bottles in OxyClean to clean them, give them a shake, then rinse, sanitize and bottle. I've never had infection problems or gunk in the bottles, even with ones that haven't been rinsed out directly after use.
Are bottle brushes really necessary, or are they more of a precautionary, "just-in-case" kind of thing?