Swimming does work the upper body, but I think it's fine to do it the day after an upper body weight workout. It may even help work out some DOMS. If you were to add more resistance to swimming somehow, it might get into the area of 'working the upper body out two days in a row,' but I don't think you have to worry about regular swimming. It would be like doing step aerobics the day after a lower body exercise: you will feel it in the lower body, but you don't have to avoid it.
YMMV, and some people do need more recovery than others. If it feels okay to you, then I say go ahead and do it. (I see you already did it! So you must not have had too much problem with it.)
(FYI: you should let body parts rest for 48 hours...a day off in between, not 24 hours...working out the next day.)