Forum Moderators: phranque

Message Too Old, No Replies

Percentage of legit emails lost

How to find approximate numbers?

         

coachm

8:53 pm on Mar 7, 2010 (gmt 0)

10+ Year Member Top Contributors Of The Month



Doing research for a book and would like to know if there is some estimate of how many legitimate, wanted emails get lost/ never seen or read by recipient.

Been searching Internet but not finding much useful, not sure what keywords to try.

tangor

9:09 pm on Mar 7, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Wide ranging question. I say this as too many out there change their email addresses as most folks change their underwear. A wanted email might be sent to an address that no longer exists because the user has switched ISP!

As to those who do not bounce around there are those whose inbound is so filled with spam they just delete everything for the next day's inbound and never see legitimate emails. Then there are the lazy folks who don't know how to scroll down to see what's there and later wonder why their email clients (Outlook, Thunderbird, Pegasus) crap and die. More fun is those on webmail systems that only keep "x" amount of email and purge it if not collected, read or managed.

I'd narrow the query to "corporate" v "personal" and go with corporate as the book study since most folks I know in the personal category don't have a clue. :)

lammert

9:16 pm on Mar 7, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



legitimate, wanted emails

According to the sender or the receiver?

piatkow

1:42 pm on Mar 8, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Emails are never lost, the recipient's spam filter never has false positives, their mailbox is never full, its always our fault for not responding to customer queries!

As Tangor says, its very wide ranging. Do you include addresses that no longer exist? Are you looking at person to person messages, mail outs, automated response systems?

Hoople

3:37 pm on Mar 8, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Add to the above read and delivery receipts. Many domains suppress sending them back through the internet as it's a known method of harvesting names.

As an e-mail admin (MS Exchange) an 'operating' goal is zero loss due to our own actions. Over the years one or two fellow admins were not so careful (-o. SPAM filters in the enterprise, even configured by best practices are still setup at some 'acceptable' false positive level. There is a wide span of the percentages depending on the product in use. A few get into the mid single digit percentages until trained and configured. A few enterprises do not allow the recipients access to the SPAM quarantine or alert them to a message being sent there.

DNS - Many times a sending domain's DNS is configured poorly or is NOT optimized fully in favor of email. This gets their mail stream tagged as SPAM even when it's not. Response by the recipients domain also can get flagged as SPAM if the DNS isn't perfect (I've seen this while at a Fortune 10). Email administrator using DNS blocking may add too many blocking lists, choose ones known for some/lots of false positives or give them all an equal 'vote' in rejection (rather than a weighting percentage). This sometimes blocks major ISP's like AOL and RoadRunner. Whitelisting can manually work around this but the recipients never know of the mails rejected that the sender never telephoned about to get Whitelisted.

AOL and Yahoo! (and other free email services) occasionaly get short term '100% is SPAM' jail sentences by the major ISP's that last hours to weeks. Some years this happens several times (Comcast.net)

End users (unknowingly) will also willingly discard non-spam messages by setting up add on programs that require their sender to 'authenticate' themselves by filling out a form and sending it back. The users of these programs rarely load their address books into these programs to 'whitelist' their sender to reduce/eliminate senders having to do this. Many senders (>50%) say "fuggeddaboutit" and will NOT use other means (telephone) to alert the recipient to their sending loss.

Believe it or not there are several recipient domains that are not directly reachable but are still 'on the internet'. They get their emails from the next upstream ISP via DIALUP on a schedule. If the schedule fails for some reason the next upstream mail server MAY send a failure message "Domain xyz has been unreachable for too long". Still happens for many 3rd world regions. A rare few sending servers place a delivery 'hop' restriction on mails they send. If the recipient domain is one hop beyond the limit the next upstream IPS refuses it with a "too many hops" error!

One 'trick' Exchange admins sometimes do is to add the email addresses of departed (quit, fired, laid-off, died) employees to a distribution list containing no members. This creates a 'black hole' that deletes the message without a trace. No error is sent back to the sender even if the server is set to allow responses to the internet! (This done to keep the mailbox data online but block external mails).

Lastly, despite what any glossy sales literature tells you about how perfect XYZ's mail server is on a VERY rare occasion a server will loose some emails without a trace. In a rarer subset of those losses a 1 in 50 reboot will suddenly find them and deliver those 'lost' two month old messages!

coachm

4:43 pm on Mar 8, 2010 (gmt 0)

10+ Year Member Top Contributors Of The Month



Thanks. I already have a list of the many ways that emails can go awry (ie. not be read), and I've found some numbers that suggest that ISP's may filter out as much as 25% of even opt in mail (I'm suspicious of the number), but I can't find any really good figures one way or another.

I'm highlighting the fact that, for very important material, people should NOT assume that their email will actually get to where it's supposed to and be read, even if it's opt in, but I can't find numbers.

Was hoping for some specific numbers.

(Hoople, special thanks. You pointed out a few things I hadn't remembers like autentication.)

Hoople

5:53 pm on Mar 8, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Also keep in mind that SMTP is a 'best-effort' protocol. There is no guarantee that a successful delivery will be made even though a network path is available and the recipients MX gateway server is available.

The fact that SMTP delivery may have worked 100% of the time between two hosts in the past does not mean in the future delivery will also be 100%. Delivery historically has been considered the option of the recipient's mailhost. Though the general public assumes that delivery is guaranteed. RFC5321 is clear about how message receipt should be done but older server do not always follow this (or RFC2821 or RFC821 even).

Many anti-spam devices/programs 'bend' the understanding of the RFC's, particularly to reduce the data load of SPAM before it's rejection (why accept a mail with a huge attachment and then reject it?). Seems trivial but many Fortune 50 and larger companies get over 5gb of mail submitted in the first two hours of Monday morning with >95% SPAM!

StoutFiles

6:03 pm on Mar 8, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



how many legitimate, wanted emails get lost/ never seen or read by recipient


That's a tough one...I'd ask Google's support team for their rough estimate.

piatkow

11:51 pm on Mar 8, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I used to keep a bounce log when I ran a mailing list for a community group. A rough tally shows a bounce rate of between 2 and 7 percent. That covered everything from closed accounts to AOL suddenly deciding that it didn't like me. That of course is just what came back to me, I have no idea what ended up unread in spam folders or auto deleted without a bounce or which just vanished.

coachm

3:26 am on Mar 9, 2010 (gmt 0)

10+ Year Member Top Contributors Of The Month



Thanks, hoople and all. THat's basically the point I'm making in the book chapter, which is to help people understand some of the business problems that can result from assuming delivery of email, and not having other methods for followup for business or critical information notifications.

The 2-7 percent sounds reasonable to me, but since I'm interested in functional results (actually having message read) I'd bet a lot more gets lost or ignored on the human side or where there is no bounce notice.

I guess that's close enough, since I can't find better numbers.

Hoople

4:06 pm on Mar 9, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Another factor to consider is SPAMishness of an individual email. Take a given sender and recipient domain. Normally mailflow is a perfect 100% both ways for GENERAL messages.

Now consider an email sent by a fictitious marketing guy Joe 'buzzword' Marketer. In his emails there are several 'signals' that a SPAM filter picks up on, giving his 'normal' messages a less than perfect SPAM 'score'.

Joe's messages might be dropped during the initial SMTP handshake, put in the global SPAM quarantine or if the recipient domain is using Outlook and/or Exchange it gets deposited in the users Junk Mail folder. Depending on the user it may be read immediately, a week later or automatically deleted if the Outlook default for the Junk Mail folder was changed by the user.

Was it delivered or not? No, yes and maybe all apply and all may correct. According to Joe 'buzzword' Marketer it was NOT delivered and I've been screamed at by his kind in this scenario MANY times. Reminding them that delivery is an option of the recipient or suggesting they test their emails with an online SPAM scoring app (to improve their chances of not being SPAM) usually gets the same reaction or the ol manager end run.

lammert

6:06 pm on Mar 10, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The bad thing about current email recipient systems is that they often accept the message, and only after they have accepted and processed the full message decide if it should be sent to a spam folder, or to a regular mailbox. If it is sent to the spam folder it will probably never be seen again, neither by the recipient, nor by the sender. Some receiving email servers send a bounce response when the email is rejected, but that bounce message is sent to the email address mentioned on the message headers, rather than directly to the sending email server.

The correct way would be to reject unwanted emails before they are fully read. In that case the receiving email server can tell the rejection reason directly to the sending email server. Although the recipient won't receive the email, the sender at least knows the message hasn't been delivered, together with a reason.

My experience is that only a small percentage of the receiving email servers are setup with rejection, and those servers setup to bounce messages can be easily abused by spammers. The general approach nowadays seems to automatically delete messages, or send them to unread junk folders.

Hoople

2:46 pm on Mar 11, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Most 'old school' mailservers (Sendmail, Qmail, Exim, etc) on their own can't do spam checks until after the message envelope (from + to) has been accepted. There are some newer products, both open source and commercial that do drop the message after the envelope has been sent by the sending server (but before it's acknowledged). The SMTP protocol says acknowledging the envelope before asking for the body part (DATA) is the normal course. Thus not acknowledging the envelope before accepting the body 'should' mark it not delivered at the sending end per the SMTP protocol. Not doing so is a failure of RFC compliance of the sender's server! Dropping the connection before the body is sent saves a huge amount of bandwidth, a BIG consideration for those buying bandwidth by the month.

The 'correct way' survived until the outgoing mail queues on big firms became clogged with SPAM related NDR's (delays going over 10's of minutes and in some cases hours). After undergoing several of these attacks many companies (after much hand wringing) decided to stop external NDR's. Myself and several colleagues billed many hours moving SPAM to holding queues so business mail could flow. Google 'joe job' and 'email backscatter' to see what are the major causes. After the worm attacks become huge dropping the body part became a 'no-brainer' for most.