Ok, I've become comfortable brewing all grains but I want to start experimenting. Where do I start? Should I start experimenting at the primary fermentation stage and brew a bunch of small different batches? Or should is it better to start when transferring to the secondary? If you'd recommend this approach what do you use for fermenters? Do they make small 1 gallon fermenters? I'd like to have my experiments beside each other for testing.
Although I don't do all grain yet, as you do, I have taken quite a bit of time to start creating my own recipes. The best thing That I can recommend is this:
http://www.northernbrewer.com/shop/1-gallon-jug.html. A word of caution. If you are brewing high gravity beers, make sure to set these up with blow off tubes large enough to fit the bottleneck.
Once you've nailed down what works, make that your de facto recipe and rinse/repeat.
Depending on your knowledge of the different tastes and qualities imparted by the various components of the brew and how scientific/casual you want to be about it, I would recommend starting with different types of yeast. They can produce vastly different flavors in the same brew, and it's one of the easiest things to tweak in a recipe. Try making a solid base recipe like a pale ale, split it in half, then pitch one half with something standard like California Ale Yeast and the other half with a Belgian or German Lager Yeast. Adjust your fermentation temp appropriately for the strains for best results. You'll be surprised by how different the end results will be :)
The same method can apply to many post-boil additives, too. You can try things like taking the whole batch through primary, then bottling half and racking the other half onto some dark Candi sugar and letting it go through secondary. Just make sure to boil the additives to sterilize them (best method is to draw some of the beer into a pot and use that to boil, so you don't dilute it with more water) and make sure to chill it so it doesn't shock the yeast when you put it in the fermenter.
When it comes to grains, you can't go "wrong" very easily. Chances are you will always come out with a good tasty brew, it just won't necessarily fit a particular style guideline. It's like a domestic dog, just because it's not pure-bred doesn't mean it can't be a great pet if you take care of it :)
Edit: From the advice of Tobias Patton - There are some general grain guidelines (for the most part there are just some you shouldn't use too much of, IE: Black Patent), you can find many resources for that at sites like this
I read over the other responses very briefly so sorry if I repeat a few things. I am really excited because I am doing almost the same thing right now! I started by deciding what I wanted to discover; hops, grains, yeast, and so on (I started with hops). One thing I think is difficult but really important is to have a "control" type of recipe, for example (I'm all grain) my first batch was just two row base malt (calculated to a medium to low ABV) and CTZ hops. All the future batches in this experiment will have the same grain bill, same hop addition times, yeast, and fermentation temps... but different hops.
Don't know what you are looking to experiment with, but if it is secondary stuff (dry hop, fruit, wood/oak...) then ya brew a batch and split it up but keep things as "common" as possible.
In doing this experiment I have really understood beer in a new light, the first beer was surprisingly good without a thousand grains and a mash of hops, maybe one of the most simple and fastest poured out of all the beers I have brewed. It is so much easier to pin point each flavor when there are only two ingredients, so I guess my advice is to keep it simple, clean, and as similar as possible. Good luck! Keep us posted (REALLY! REALLY!) and if you have more questions throw them out! CHEERS!