SafeEar is an innovative audio depth detection framework that can identify deep audio without relying on speech content. The framework employs a neural audio codec design to separate semantic and acoustic information from audio samples, using solely acoustic features (such as prosody and timbre) for depth detection, thereby safeguarding the privacy of speech content. SafeEar enhances the detector's capabilities by augmenting the codec in real-world scenarios, enabling it to recognize various forms of deep audio. Extensive experiments conducted on four benchmark datasets demonstrate that SafeEar is highly effective in detecting various deep technologies, achieving a minimum equal error rate (EER) of 2.02%. Moreover, it protects speech content in five languages from deciphering by both machine and human auditory analysis, as validated by our user studies revealing a word error rate (WER) exceeding 93.93%. Additionally, SafeEar has established a benchmark for anti-deep and anti-content recovery assessments, laying the groundwork for future research in audio privacy protection and depth detection.