Avoiding Spam Filters: What Really Happens to Your Email?

Who Are “Spam Filters”, how to avoid them and what does it mean to my email delivery?

While spam filters is a pretty general term most fall into three categories. I do do not plan or attempt to cover any end user home pc anti spam programs. What we are interested in is the big picture filtering picture here.

  • Content filters like Spam Assassin.
  • Blacklists online like Spam Cop and Spamhaus which is the main Yahoo email spam filtering engine.
  • ISPs like Comcast and AOL and Mail Service Providers like Gmail, Yahoo! and Hotmail.
  • Here is a quick look at all three.

    Content Filters

    Content filters are the most prolific and common spam filters on the Internet. The most widely used anti-spam application Spam Assassin™ spam identification tool. Depending on how your newsletter is written, or how an application like this is configured, your opt-in email may be inadvertently tagged as spam.

    Spam checker can help you combat these “false positives” by checking your message before you send it. Run your sample message through the Content Checker, and you’ll receive a detailed evaluation. The report will tell you if your message triggered any red flags that might block it. You can then make recommended changes, and mail with the confidence that your message meets industry standards.

    Online Blacklists

    Online blacklists are very powerful user generated, spam reporting services, sending reports of spam to both ISPs and Web Hosts of both possible spammers and actual known sources of spam. You can easily land on a blacklist by a single spam complaint from anyone receiving your email or newsletter.

    Spam Cop is by far the most powerful blocklist and is now owned by Ironport Systems. If Spam Cop lists you as a spammer the blocking can last a few days or longer depending on the number of complaints by Spam Cop users.

    Here is a great graphic illustration of how a blacklist works and you can check to see if you are blacklisted here.

    ISP Filters

    ISP and Mail Service Providers use multiple combinations of all the filter technology available. Remember that their number one concern is bandwidth and server load. The more spam they can filter out the less money they need to dedicate to administrators, hardware and bandwidth.

    These consist of “this is spam” button complaints more that any other single source.

    Part Two: Anatomy of a Spam Filter

    This is the process that your emails face after they are received on ISPs and Mail Service Provider’s email servers.

    Step #1 Is The Sender Whitelisted

    The first hurdle your email faces is whether or not you have already been whitelisted by the user. If your domain, email address or email server IP address are found on the recipient’s whitelist you go directly to the inbox.

    You can generate user whitelisting instructions here at my site.

    If the sender is not whitelisted then on to step two.

    Step #2 By IP Address Cross Referenced to Black Lists

    There are two kinds of black lists and being listed on either will result in a lot of your email being blocked.

    ISP Black Lists

    All the ISPs have their own rules for how they prefer email to be sent to their servers, the most particular and largest being AOL, Yahoo! And Hotmail. They all maintain their own blacklists based on these preferences and the spam complaints of their users.

    These black list criteria almost always include:

    • The “this is spam button” at the user level
    • The number of bounced emails sent to non-existent addresses

    Unfortunately ISPs do not look at your email in detail. They only look at the IP address from where the email is sent. If you are sending from a broadcast or autoresponder service or if you are on a shared server and using a script to send your emails then all emails sent from that location will be judged by the actions of everyone in that shared environment.

    On a shared hosted server that can be as many as 1000 other sites! If any of these other sites are blacklisted for sending spam, real, imagined or falsely accused you will be blacklisted right along with them.

    If the autoresponder or email management service you are using is accused of spam and blacklisted then all email sent via this ASP (application service provider) will be blocked as spam as well.

    Online Black Lists

    Online blacklists compile their own blacklists based on the input of anyone who cares to complain that you spammed them. This may be the United States where you are supposedly innocent till blah, blah, blah, but this is not how these list operators work.

    Just one complaint will get you listed as a spammer and may result in lots of blocked email. Spam blacklist sites compile lists based on IP addresses, so you will be guilty for all actions based on your shared server and all actions taken by users of a email broadcast service.

    Black lists make money by selling results of their lists to corporate email users and ISPs, providing filtering based on their lists and black list access to other filters.

    Spam Cop is the best known of the private online blacklist agencies and is highly respected. They allow reporting of spam from users and then supply their list to ISPs, spam filters and you, if you care to check. Spam Assassin definitely uses Spam Cop for black list references.

    The problem with any blacklist starts here: Email FROM and TO headers are so easily forged that black list operators go by the IP address that the supposedly offending email was sent from. Just like ISP blacklists everyone in a shared environment gets blacklisted.

    Not On Blacklists? On to Step 3.

    Step #3 Finger Printing

    Spammers usually write poor HTML or use software that does. Many use tags that are used for other purposes than what was intended by HTML standards, like IMG tags that resolve to file extensions that are not .gif or .jpg. (tracking links) and .exe (viruses). Also FrontPage generated HTML emails are usually filtered too.

    Finger printing filters use an algorithm to assign each character an email a value. The total of the values create a fingerprint of the email in question. The fingerprint is compared to the fingerprints of known spam.

    Using good email practices is the best way to avoid this filter. Do not send HTML emails generated by cheap software. They are not rendered in most email clients these days unless you are whitelisted. Do not use huge graphics and never use Front Page to build HTML emails.

    Step #4 Content / Heuristic Filters

    Content based filters are the most common of all spam filters and almost every spam filtering solution depends on a content based filter at some stage of the filtering process. The most highly used content filter is Spam Assassin. Your emails will face Spam Assassin at almost every level, starting with the ISP, both outgoing and incoming, and at the recipient’s ISP or web host.

    Most spam content filters work off of a numerical scoring system that adds to your score each time an offensive word, phase or email coding item in their database is found in your email. Once a score threshold is reached, your email is deemed spam and deleted, quarantined for admin review or sent to a junk folder.

    Tests include spam words and phrases, forged return addresses and the sending domain, words all in caps (YELLING), extremely large image files, MS FrontPage (or other crappy software) used to create the HTML email, certain attachment types and highly used spam words and phrases (we will go in to this in detail) used in known spam emails.

    Step #5 Learning / Bayesian Filterers / End User Level

    Learning or Bayesian filters are recipient end user filters installed in the email client. As the user begins training the filter they vote each email as spam or not spam. The learning filter then assigns each element of the email a value. Things like images, HTML tags, symbols and words all carry a value determined whether or not they were used in previous spam.

    The filter then acts on incoming mail by filtering what it considers to be spam or passing real email on to the inbox.

    Step #5.5 Collaborative Filters / End User Level

    Collaborative filters work on the input of the end user community. Each email user can vote a particular email sender as safe or as a Spammer. This is usually found in the corporate environment and on Gmail, AOL, Yahoo! and Hotmail. Yes, this is the dreaded “This is Spam” button.

    While your newsletter is gold to most of your subscribers it only takes one idiot who thinks blocking or clicking the “This is Spam” button is a way to unsubscribe. The recipient’s network or ISP keeps track of how many “This Is Spam” complaints you get and over time can lead to blocking.

    Your recipients can usually unblock your emails by recovering them from their spam folders. While you and I may do this as informed Webmasters and marketers how many average email users do you think would do this? Also look at your “Junk Mail” folder, there is a ton on spam to filter through. Once again the best solution is to inform your subscriber upon sign up how to white list your sending domain.

    Solution

    Checking the content of your emails with the Lyris Spam Assassin content checker is a must before sending any broadcast email or
    adding any content to your autoresponder. You can use the content checker to run your emails past Spam Assassin’s tests.

    New recipients must recover your very first email from the junk mail folder and mark it as “Not Spam” should it land there. The largest major filter you face is the bulk mail folder at the major ESPs (email service providers), like AOL, Hotmail and Yahoo!

    Whitelisting of your email address 99% of the time overrides any filtering solution in place. In the score based solutions it applies a negative number so high that it overrides all positives for spam scoring numbers and passes your email.

    You can use my email whitelist generator here…

    You can get the full lowdown on how to get you email delivered every time at Email Delivery Jedi…

    2 Trackbacks

    1. […] Anatomy of Spam Filters […]

    2. […] Anatomy of Spam Filters […]