All true gearheads love that backfire sound! Roll the window down and enjoy! (Or am I giving away my age here?!)
This is caused by the presence of both unburned fuel and oxygen in the hot exhaust manifold. The heat or flame from a cylinder's open exhaust valve ignites this unburned fuel-bam! Under the right conditions, she'll really cackle!
If you really want to quiet it down, check that the timing is OK (if you have the original Load-O-Matic vacuum only advance, make sure it is working). Maybe richen the idle mixture a little. May be comparing apples and oranges here, but with my Model A in high school, if I wanted it to backfire more, I would close the mixture rod (GAV) to lean it out and throw the spark lever up to full retard. Result: lots of noise (blown muffler). If I left the timing advanced and kept the mixture knob open, it was much quieter on deceleration.
Finally, add a dashpot to the carb. This slows down the closing of the throttle plates instead of them snapping shut when you back off. They were standard on auto trans cars back then.
For the hard starting, the float bowl may be leaking into the intake. Does it act flooded after sitting a couple of hours? Try looking down the secondaries with the throttle held open. You will get a couple drips from the accelerator pump, but if you see puddles, the carb has issues.