Backfiring is unburned gas exploding when enough of it accumulates in the exhaust system. Anything that interferes with gas burning sufficiently in the combustion chambers will result in backfiring. There is a near infinity of possible causes, including electrical, fuel, and air issues. If this thread runs long enough, you'll hear them all.
If the problem occurs mainly under load, the cause is almost certainly electrical, i.e., inadequate spark. This is because under load, the throttle is wide open and the rpm's are reasonably low, so the engine can breathe deeply, the pressure in the combustion chamber is high making it most difficult for the spark to jump the plug gap. Anything in the in the ignition circuit could be the culprit: Make sure it is tuned up correctly, and there is no shorting-to-ground via tracks or cracks on the distributor cap or body or plug porcelains or wires.
If the backfiring occurs mainly under idle or light load--especially at high rpm--like when roaring down a hill, decelerating, or shifting gears, the problem is probably fuel related. Paradoxically, too lean a mixture can cause cause accumulation of unburned gas in the exhaust system and backfiring. This is because if the mixture is too lean, the sparkplugs can't light it, so out the exhaust it goes.
What's going on when the engine is decelerating is that the throttle is closed and the engine is revving--this is a recipe for too-lean a mixture a mixture. The spark plug is trying to fire on every compression stroke, which are coming fast and furious, but the only gas being supplied is from the idle circuit, which is hopelessly inadequate for the demand. That's not enough gas to support combustion by the spark plug, so off it goes to the muffler. Do what you can to avoid high rpms with the throttle closed: don't back down in low gears and do your gear shifting at lower rpms.
Of course, if the carb is set too lean, or there is an intake vacuum leak letting too much air into the intake manifold, or if there is a fuel restriction of any sort (e.g., clogged fuel filter or jet), those all could cause too-lean mixture under normal running conditions. In all of these cases, a quick-fix straw to grasp at is to open the gas adjustment valve (on choke rod) a turn or two. If that helps, the problem is definitely fuel related, not electrical.
Another source of backfiring is a leak in the exhaust connections. This is normally a problem after shutting down the motor, whereupon fresh air is sucked into the fuel-loaded exhaust system as it cools down. This could be either at the manifold, which you claim to have eliminated, or at the exhaust pipe clamp. But, since your problem occurs when going down the road, when there is always positive pressure in the exhaust system (unless something is really screwed up with the valves), I don't think this is your problem.
In your case, since the problem is intermittent and sometimes at steady highway speeds, I agree with Keith that it's probably electrical. Tighten EVERY single connection and check for possible shorts EVERYWHERE in the ignition primary and secondary circuits. You can try putting a jumper wire around the ignition switch and the ammeter (junction box lugs) to eliminate those contacts.
However, as others have suggested, strange things can happen when rust flakes, etc. are bouncing around the float bowl. If at all possible, swap your carb, jus to be sure that's not the problem.
The problem WILL be found!
Steve