Warp Records Bleep ~ http://www.warprecords.com/bleep ~ Some of the greatest electronica ever made, and availble as high-quality mp3's - with no DRM! - for a reasonable price.
Magnatune ~ http://www.magnatune.com/ ~ "We're a record label. But we're not evil." No, they're not. Downloads in a variety of genres, including Classical, Electronica, Jazz & Blues, Metal & Punk Rock, New Age, Rock, & World. Available under Creative Commons licenses. 50% of sales go to the artist. Downloads available as high-quality mp3, ogg, wav, or flac!
Telltale Weekly ~ http://www.telltaleweekly.com/ ~ Not music, but spoken word readings of fiction, nonfiction, poetry, and audio theatre. All very cheap - usually less than a dollar - and unencumbered by DRM. They even have ogg! I bought a reading of Walt Whitman's Leaves of Grass, which is just too cool.