Turns out my problem was most likely that last post from Terry, OH. I did learn a thing or two about 46-48 backing plates and parking brake arms. My backing plates have the correct brake shoe clips as Ford only made the two tiered clip pre-war. The parking brake lever is also different pre and post war. My fellow Early Ford V8 club member, Keith Smart, schooled me on the difference and sold me a pair of the correct ones.
Even with the correct arm, until I reversed the return spring to come in the back side of the rear shoe, the brake lever arm stood out to far from the shoe web and likely would have come in contact with the drum again.
Hopefully all good now. Here are some pics of the different parking brake levers and the spring orientation called out by Terry, OH.
Parking brake levers, pre-war on the left and post-war on the right.