Could be a few things but when it happened to me it was the gear on third worn out causing the torque to make it twist away from itself. Easiest thing to do first is to pull the shift tower and make sure that is functioning properly and the ball on the end is not worn.
... Fixing half the problem will result in an half fixed solution...
Not only can the ball be worn, but the shifting forks where it rides could be worn as well. Fixing half the problem will result in an half fixed solution.... In this situation, you would have a very large "neutral" area in the shift pattern. Jim