don't know how a broken key would do this. The keyway sure can get chewed up from this though.
either the drum is binding on the backing plate or on the brake shoes.
I have been known to take a 4" right angle grinder and grind a little off the side of the shoes, especially the E-brake band.
If binding on the backing plate, you could grind a smidge off the inside edge of the drum, or they make shims to put between the drum hub and the axle which spaces the hub outboard (outward) more..
In any even, fix it soon since the hubs/drums are supposed to be very tight to the axle else you will have mega wear going on in there. I am surprised it doesn't 'clunk' at least once if you change directions at slow speed, like backing out of a parking space