If its been like that for some time then maybe there is rust on the edge of the hub that you hear rubbing. Sand the rust off. Maybe the hub is actually rubbing against the backing plate requiring the hub/drum to come off and find out why.
Well, I thinking the front hub. If its the rear then it could be going on too far requiring an axle shim.