Why are my pea plants yellow and wilting from the bottom up?

I decided to try to grow some veggies in containers this summer. I picked up some sweet pea plants at Home depot and transplanted 4 plants into a window box. They are starting to flower but the bottom leaves are yellowing and wilting. There are occasional aphids I've removed but only a couple, not really an infestation. They get 6+ hours of direct sunlight. I live in M and we have had a lot of rain. The soil never really dries even though it's got drainage holes in the bottom.