When it comes to hearing loss, it doesn't matter what you're listening to...music, a podcast, even standing near a loud lawnmower or watching fireworks.
Noise that's too loud damages cells inside the ear, called hair cells. They translate sound into electrical signals that get sent to the brain. Damage to many of these cells and your hearing starts to go.
If you're standing near someone, about an arm's length away, and you can't hear what they're saying because whatever you're listening to is too loud, or noise in the background is too loud -that's how you know when sound is too noisy.