Can Rest Make You Faster?

Can rest make you faster? When I take a long time off from swimming - say, over three days - I am definitely feeling the need to work hard just to do a comfortable speed. If I take a weekend off, I feel great on Monday and can usually go full bore. If I take no time off (go weeks without a break), I can get sore and take almost my whole workout just to get loose. So given all this, I see there is a fine line between taking too much time off and too little. Finding that perfect mix of "efforting" and rest is a delicate balance. I've found five days of swimming and two days of back-to-back rest is about perfect for me. I usually do some sort of physical activity on the two days off, but its usually not swimming (laps). I've heard that this recovery time is good for muscles to allow them to grow.