Thanks for all the ideas and questions. Since I was worried that it could be the sound of impending detonation/doom, I swung by my mechanic to listen.
They didn't seem to think it sound like anything, so at least I feel good that I likely have time to find it and fix it. I think part of it is that older folks' hearing is slightly diminished, and it didn't seem as loud to him, but to me its quite loud and driving me bonkers.
After watching him with a mechanics stethoscope, that gave me the idea to stick tubing to my ear and listen for the noise. I spent last night poking around I think I have found it - a particular spark plug boot, on the drivers side closest to the dash which would explain why its so loud in the cabin. When I push on the boot, the sound changes and/or goes away, and I could feel pulsing that matches the timing of the noise.
I have yet to change the wires since i've had the car and knowing what i know about it and the previous owners, i'm guessing the wires are at least 10 years old. This particular boot was once black/grey and is now whiteish so i'm guessing its at the end of its life.
I ordered new wires last night, so hopefully that fixes it and sorry for the wild goose chase.