My son was a walk-on college swimmer but only did it for a year (probably heresy to you, a rugby player) but he was a young 17 and couldn't hack/enjoy the beer culture. Does triathalons now (he's usually the swimming leg, in the Pacific).
My daughter has been on a swim team since she was 5 but I don't think she'll make a college team because it's her "winter" sport at her high school. Coming from California, she certainly contributes to her east coast team but I think anyone who doesn't swim at the club/USS level year round is unlikely to swim at college and she can only rejoin her USS team in the summers. [One of my pet peeves: why should kids at a very young age have to dump all sports in favour of one ... don't get me started!].
Did you do other sports in high school or just swimming or did you do club swimming at the USS level while still participating in other sports at your high school? And if you have already answered this, I apologize and will go troll in the archives. Thanks.