I agree with others; I think your fluctuating temperature is the likely cause. I have never tried the swamp method but I am about to convert a refrigerator into a fermentation chamber.
I have a temperature control unit I picked up on ebay from someone in Hong Kong - the unit name escapes me at the moment. I am going to by-pass the thermostat on the fridge and hook up the temperature control to activate the compressor when too warm and a low wattage light bulb, that I will fix to the bottom of the fridge, when too cool to provide heat.
I am hoping to source a second-hand fridge this week, so I might be able to get started on this at the weekend.
If you Google 'fridge fermentation chamber' you can find quite a few links to blogs and forums on this.
EDIT: I dug out the controller details. Here is a link to Amazon's US page, just for info. All-purpose Temperature Controller STC-1000 With sensor