Southern English brown ale, which I like, seems to differ from porter, which I dislike, by only a matter of degree. It just seems to be about the amount of specialty grains. Is there any other difference?
Southern English Brown is mildly sweet and Porter can be fairly dry and and roasty. I think it matters upon the substyle of Porter one is looking at. Robust Porter is the substyle most people are familiar with and I view it very different in flavor and composition that a Southern English Brown.
If I was forced to draw the line to bend my thoughts to your question I'd say the difference is that more often than not you are going to see some black patent in the Porter and probably not see it in the Brown Ale.
I view Porter as being having roasted, coffee and almost burnt flavors. With a mild malt undertone. Brown Ale in general is more chocolate, toasted bread like and mildly fruity (if english) in character.
I think Porters are also more dark, maybe the darkest beers in the style. Also Porters should have more chocolate and roasted flavors then English Brown Ales - they have more caramel and toffee flavors then porters and also they have often medium to medium low body. Porters are more full bodied I think.
There are also different styles of porters - Robust, Baltic, Brown, Imperial, ...