Not that it's a bad idea, probably should've always been part of email, but it's too little too late IMO. Sort of ike trying to put a band-aid on a severed limb.

I recently read a 27 page document from BITS (2007 April) that discussed DKIM as one of four protocols that are being recommended to financial institutions. Those are...

1. TLS - Transport Layer Security
   
2. SIDF - Sender ID Framework
   
3. SPF - Sender Policy Framework
   
4. DKIM - DomainKeys Identified Mail

BITS is recommending that each of the protocols be implemented within 18 months.

I've found myself immersed in email technologies these past couple of years and my brain hurts! SMTP is no longer a viable protocol for the sending of "mission critical" email. Not in its basic form anyway.

For those of you wishing to expand your portfolio of services, becoming a Trusted Email Provider would be a great niche to target. Be prepared for some initial cash outlay to bring your email servers up to snuff. But, the demand will be there for guaranteed email delivery. It is now...

The difficulty will likely be in getting everyone on board. Though some phase in would be great. To start, add the key to all outgoing mail for a year. Incoming email that has the key gets through, without it it's forced through a spam filter. After a year or so move to allow emails with keys only.

I may be missing something though - how does this stop someone from sending spam from their local internet connection (via their ISP). Or from someone registering a domain in a third world country and sending the spam from behind that kind of wall - just like they do now.

When I read this I immediately sent off something to my ISP (rhymes with Black Space) and they said..

"Unfortunatly at this time we do not support DomainKeys. It requires extensive customization of the mail systems and we aren't familiar enough with it to attempt it."

That is going to be the problem.

BZ

Unless I've missed something, the fundamental weakness with the smtp email system is that it is a push technology rather than a pull technology.

If email messages were pulled of the originating server (in response to "come-and-get-me" requests) it would be impossible to fake the origin of the message because it would be collected directly from the originator (by IP address).

If email protocols are going to be radically reworked, surely it makes sense to fix the source of the problem rather than use crytographic bodges.

This would have the added advantage that the originator would know immediately when an email is read - thus eliminating the need for receipts. It would also allow genuine mailing lists to remove users that no longer read their mails.

Kaled.

This really does more to stop phishing than it does to stop spam.

It helps a bit with spam in that it makes it easier to identify the source of spam. (i.e. if there is a valid signature). Once adoption is universal, unsigned mail can just be ignored. It won't prevent signed spam, but at least makes it more likely that the source can be identified (perhaps some lawmaking is needed to make the signer of an email responsible for it's content) and makes it possible for receivers to choose to filter mail based on sender, with an assurance that the sender can't be faked.

But it really doesn't solve the problem.

What's going to happen is that the smaller email users are going to be forced to use a third party service to guarantee delivery of their email. Many are not in the position to begin to understand what is happening here and where email is headed.

In fact, we are in the process now of getting out of the email business for our clients. After reading everything I have and staying up to date with the latest changes, it just isn't worth it for us to make the investment in hardware and software to make it happen. Way too expensive right now for the smaller consumer of email services.

Best option is to sign up with a third party provider, point your MX records to their server and be done with it.

If you think this is going to be tough to implement, talk to the folks at SPF. It has been out since 2004 October. I can't tell you how many bounced emails I've seen due to misconfigured SPF files on the recipient side. Arrrggghhh!

I've been looking for a company that I could outsource my website emails to... Any recommendations please sticky-mail me... I am basically looking to replace my use of local SMTP and use some sort of API from a provider to send my mails.

Does such a service exist? I thought I could do it with Google's mail service for domains. But it doesn't seem possible...

Does such a service exist?

Yes they do. I believe smtp.com would be a good starting point. There are others too. Based on my research, these companies have the hardware and software to provide a Trusted Email Environment and guarantee delivery of your mail. And yes, you send from whatever domain you specify, not a third party domain.

Thanks pageoneresults!

kaled for e-mail czar!

If email messages were pulled of the originating server (in response to "come-and-get-me" requests) it would be impossible to fake the origin of the message because it would be collected directly from the originator (by IP address).

The "problem" with this approach is that queued-to-be-retrieved e-mail messages would sit on the originating server until retrieved, thus making ISPs whose users send vast quantities of e-mail foot the bill for this temporary storage. They'd also have to decide how to time-out messages which were never retrieved. For those reasons, it may meet resistance; They'd all prefer to just "fire-and-forget." However, I think anything that pushes the costs back toward the sender like this is good.

I think it more likely that the ISPs will adopt whatever method they can charge the most for... :(

Jim

The "problem" with this approach is that queued-to-be-retrieved e-mail messages would sit on the originating server until retrieved, thus making ISPs whose users send vast quantities of e-mail foot the bill for this temporary storage.

This is a good thing.

The "problem" with this approach is that queued-to-be-retrieved e-mail messages would sit on the originating server until retrieved, thus making ISPs whose users send vast quantities of e-mail foot the bill for this temporary storage.

I think the problem with Pull technology would be the fact that you'd have to hit every single email server to see if there's a message there waiting for you over and over and over again.

I think the problem with Pull technology would be the fact that you'd have to hit every single email server to see if there's a message there waiting for you over and over and over again.

That was one of the first things I thought of as well. One possible solution is that the originating server sends a "notification" message to the recipient's server notifying it of the queued message. Then the recipient's server knows to grab the message when the recipient checks for new messages.

And yes, it does seem to add some overhead...

But then again, it also alleviates some overhead as servers could maintain blacklists and ignore all notifications from blacklisted servers (so the full message would never be transmitted).

But it really doesn't solve the problem.

No it doesn't. And its unfortunate that solving the problem is a problem in itself. ;)

Changing email protocol would be a monumental task. That's like trying to get people to validate their code, it just won't happen until they are forced to do it, and that isn't going to happen either.

I believe we are stuck with SMTP for at least a few more years. In the mean time, we need to work with the existing protocols and implement the various suggestions to help minimize the impact of UBE, UCE, etc.

Here's the problem in itself; getting people to adopt the protocols.

A little OT, that whole challenge response protocol sure didn't go anywhere, huh? I get those Earthlink notifications and they go right into the bin. Same goes for SpamArrest.

> This is a good thing.

Which is why I quoted the word "problem."

> One possible solution is that the originating server sends a "notification" message to the recipient's server

Which is essentially what kaled proposed above... :)

I think the "overhead" would be more than compensated for by the impact this would have on mal-mail.

Jim

I agree with pageoneresults... it's just getting to be too much for the little guy to keep up with this. I have started moving my clients from our in-house email to Google's domain apps. With the email version of the Sword of Damocles (IP blacklisting), it's just too much of a risk to our servers to handle much email in-house. I limit our servers to only sending automated mails such as admin reports, and even then I keep each one properly setup with SPF's in our dns servers.

Even with domain hosted Gmail, I still have to outsource our (non-commercial) mailing list, because I haven't the time, knowledge or resources to keep all the ISPs happy.

Unless I've missed something, the fundamental weakness with the smtp email system is that it is a push technology rather than a pull technology.

You've missed the basics of spam: if it's convenient and cheap for any user to transmit information (via pull, push, or donkey) to any other user worldwide, then it's spammable.

Example: Devise a brand new, completely unspammable email system. If it ain't easy for you to use it to send email to fred@whatever.com, then nobody will adopt it. If it is easy, then it will likewise be easy for the bot that has infected your machine to use exactly the same steps you do to send email. Got an absolutely foolproof authentication system? Excellent -- that means the spam the bot sends from your machine can be absolutely authenticated as coming from you.

Any system that lets me send mail to anybody else, including someone I've never had any contact with before (part of the key to email being the #1 application in the world), then it's spammable.

Spam is like cancer: it's many problems, not one, and all you can do is reduce it by hitting it from many different directions. So, graylisting+honeypot DB does a good job at eliminating spam not sent by real MTAs, SPF does a good job at eliminating joe-jobbing, DKIM might help with phishing, etc.

The real battle in spam right now is the botnets. When you get an infected machine using the user's account to send email via the user's MTA, it's pretty impossible to distinguish it as spam based on the envelope alone, which forces people to degenerate to awful filter systems that require tuning and too high false positive rates (doesn't take much to be too high).

While the botnets continue to thrive, spam will always have a base level of traffic that cannot be eliminated.

I can see one big advantage to "pull". Once a spammer is identified, it is quite possible that most of the spams would not have yet been delivered to recipients. The sender's ISP could delete most of the spam before it is ever delivered.

The "pull" might be done directly by the end user's computer, or by an intervening server. In either case, it would be useful to accept mail from known senders promptly, but defer retrieval from unknown senders until the user checks their mail, or for some predetermined period. This allows some time for spam sources to be discovered and spam emails deleted prior to delivery.

Based on what everyone has said so far, I know one thing. Whatever changes are implemented, there are going to be a whole group of ISPs jumping the SMTP ship. They won't want to deal with this level of email management. Or, if they do, it will come at a premium.

I've been comparing prices for third party SMTP services and it isn't exactly cheap. I'm also looking at Virtual Exchange Mailboxes at $12.95 per month. Its a cottage industry that is going to be booming here real soon, mark my words.

Kaled explained it well. However I don't think a full "pull" is needed. How about sending out the message with a unique verification id. Receiver verifies that the sender meant to send the message and lets it through. Or is that what domainkeys does?

This shouldn't be that hard a problem to solve. I would think if email protocol was developed with the Internet and spam in mind, people would've come up with a decent one in the first place...

Based on what everyone has said so far, I know one thing. Whatever changes are implemented, there are going to be a whole group of ISPs jumping the SMTP ship. They won't want to deal with this level of email management. Or, if they do, it will come at a premium.

I don't know about that, I think ISP's will be the first ones in line for tighter email control. There would be a lot more usable bandwidth if we were able to kill 90% of email traffic.

Since there was some approval of my suggestion, I shall elaborate, just in case someone with clout is reading this.

1) The originator sends a brief "I-have-mail-for-you" signal to the recipient. This could contain from:, subject:, and date: headers, etc.
2) The recipient then either requests the mail or sends an "unwanted" signal. (The request could include a public encryption key for secure mail.) This request could be sent by servers or by users - there are advantages to both so ideally, the user should be able to choose which system to use, but, on balance, I think user-requests would be preferred (but mail retrieval would be slower).

This would not stop bot-generated spam but it would mean that much of it is never sent since it would mostly be rejected by users after reading the from: and subject: headers. Thus this system would reduce bandwidth requirements considerably.

This system is simple - there are schoolkids out there that could implement it and no cryptography is required. And because it would reduce the load on recipient servers, the tendency to block mail by IP address should be reduced (this blocks many legit mails too!)

Also, if a protocol similar to http was used for mail retrieval, bandwidth requirements for attachments such as photos (currently hex-encoded) would be much reduced.

This would not eliminate spam entirely but it would make it impossible to fake the origin of any spam that is sent. It is simple, requires no cryptography and would reduce the overall bandwidth requirements of email traffic considerably. It opens the door for truly secure email and eliminates the need for receipts. Can any other system come close to these advantages?

Kaled.

To achieve backward compatibility with the existing smtp mail system...

Send the "I-have-mail-for-you" signal using a rigidly-formatted smtp message that includes a "click-here-to-read" link. Provided the originating server can respond to an http request and deliver the mail, users with existing mail clients would still be able to read mail on the new system. How cool is that?

Kaled.

The "I have mail for you" is still spam.

The simplest solution to this mess has always been to add an origin of email tracking system on top of the existing SMTP system.

Concept is a trivial email sender handshake system:

a) you only accept email that originates from the domain in the "FROM" field
b) when an SMTP connection is made, you make a new connection back to the originating SMTP server by domain name the get a list of EMAIL ID's being transmitted
c) all email lacking a valid EMAIL ID from the current connection is rejected.

You could still use pre-existing SMTP sites, optionally over time requiring ONLY connections that support the new EMAIL ID delivery verification system. Before full adoption of this methodology you could tag email that was ID VERIFIED vs NON-VERIFIED which would make it easy to sort out the most likely good from the probably bad with very few false positives.

There would be no way to spoof existing domains using this method so all of the email would have to originate from the source it claims to originate from or bounce, easy enough.

This would mean spammers would have to pay for a new domain for each spam run, as that's about how long it would take to lose the domain. Gets expensive real quick and the pressure would be on the registrar to stop selling spam domains after repeated spam abuse reports. These domains are easy to spot as you can reject email from any domain registered within the last 14 days to make sure they aren't "tasting" domains for free.

Now if you want to make it more costly for spammers to spam, require that each mail server have it's own SSL certificate and only accept email from properly secured servers. Then losing a domain name would also mean losing an SSL cert as well, also allowing pressure to be placed on the SSL providers to stop selling to spammers.

Whatcha think? :)

[edited by: incrediBILL at 2:06 pm (utc) on May 24, 2007]

Now if you want to make it more costly for spammers to spam, require that each mail server have it's own SSL certificate and only accept email from properly secured servers.

I like it! It works with the existing protocol and appears to be a viable solution "on the surface". In theory, it looks like a possible solution.

My ISP is Whizzyispinc. It may be whizzy but all the other users (not me of course) are idiots and their machines get infected and send out spam. Unfortunately, Whizzyispinc gets banned so my legit emails don't get through either.

The system I suggested is not perfect, but it is better by far than the DomainKey system that began this thread - and it's very simple - no special handshaking required - it could be introduced easily and gradually.

Kaled.

If the majority cannot adhere to SPF, do you think DomainKeys really has a ghost of a chance?

If the majority cannot adhere to SPF, do you think DomainKeys really has a ghost of a chance?

I think so. At the time SPF was released (2004 October) the issues we are now faced with were not as prevalent so adoption of SPF was very slow.

At this point in time, I think more and more people are becoming aware of their email issues and are taking corrective measures where possible.

If DKIM is adopted globally, SPF will be too. There will be a snowball effect of those in the know implementing the various methods described to secure their email networks. Of course the early bird gets the worm in this instance. :)

P.S. There are still some "cart before the horse" issues here. For one, there is a whole network of comprimised servers out there that need to be shut down and/or blocked permanently at the root level.