Depends at what speed you travel. If you put along on back roads at 40-45 mph or less, that's not unusual. Once you get up into the 50s, that would surprise me.
I am like you, I rarely drive over 40 mph and I have never had a balance problem. I have a 30 Chrysler that I drive up to about 55 and I have to have those tires balanced.