That's why I use aweber. A third party (aweber) deals with all of it - mail comes from aweber and you can even set it up to tell them when and where (and what IP) they subscribed from. I've got about 25k people on my aweber lists and I've never had a 'real' issue. Its only 30 or 40 bucks for that many people a month as well. First 10k is 19.99 I believe. Each additional 10k is 9.99 per month on top of that...
Not really a big deal at all.... because if you have 10k people you BETTER be making 19.99 a month mailing them
Pretty hard to say something is spam when it says:
You subscribed and agreed to receive this newsletter on xx date from IP address xx.xxx.xx.xx. Please use the following unsubscribe link if you no longer wish to receive mail from us.