It's been a while since I was last in New York, but the main two I remember buying stuff from was the aforementioned Academy Records and BookOff on 45th Street, which I think is mainly used CDs, but there's quite a lot of titles and I snagged some bargains. I think there might have been another shop on Bleeker Street (next door to a nice little coffee shop, where Carole went to chill, when she got bored waiting for me to rummage).
I did find a couple things at Academy Records - at the location mentioned here. There is another one much further East (415E) they have no CDs whatsoever, and only LPs.
BOOK OFF for soundtracks had only mainstream releases - nothing from any specialty labels whatsoever.
The Academy at just south of the Flat Iron building had stuff from all the specialty labels. I'd say that's your best bet.
I did not make it to Brooklyn to explore the Record Store scene there but it looks like a lot of places are focused on LPs.
