Honestly, we can almost always not tell them apart either. I mean, I (an American born and raised here) cannot tell a Canadian from an American from the Midwest or New England or the Pacific states, unless the American has a strong accent. If they are more "TV broadcaster" sounding, and no one uses the words "oot" or "aboot", then I am always surprised to learn I have been speaking with or listening to a Canadian.
New Zealand and Australian accents are a lot more distinctive than Canadians and Americans.
And as the original poster, yes, I was just kidding. For me the toughest "English" accent to understand are people from Scotland. HELP!!