We send and process millions of emails from WordPress each month. There are times when there may be 10s of thousands of emails in the send queue at once. These include emails like:
- new site or user welcome emails
- password reset requests
- new comment notifications
- subscription emails when new posts are published
- moderation notifications for reviews
- and notifications from contact forms. So many contact forms.
Y’all, it turns out, sending system emails from WordPress is hard.
For example, send a few too many email notifications when spammers fill out your contact form and your host or email service provider may shut your emails off.
Or no matter how careful you are, you may find that your important and valid systems emails are ending up in spam folders or not arriving at all.
The struggle is real, and recently for us, it got really real.
To our customers that have experienced the recent problems, thank you for bearing with us over the past month or two as we’ve worked through this. We know that a handful of you have noticed issues with some emails not getting through due to unexpected side effects of what we describe below. We are sorry for the troubles that this has caused and are continuing to work tirelessly to provide the best and most reliable email service possible.
We’d like to share the lessons learned from all of this here. For anyone running any WordPress site, jump to the bottom of this post for tips and tricks that you can immediately implement to ease your WordPress email sending pain.
For over a decade, we’ve quietly and successfully managed email sending from our own private mail servers that we’ve configured and maintained. Sure, there have been bumps in the road when an IP would get blacklisted or the send queue would get stuck. This has worked (mostly) well forever until it didn’t. Our datacenter stopped allowing us to use the mail servers in the way that we had, giving us limited time to come up with a new plan.
The good news is that we do our best to have multiple irons in the fire with all that we do. Meaning that while 90% or so of our customers have been on our trusty mail servers, we also have customers that have been sending mail for several years with both AWS SES and Mailgun. We knew they worked. And we knew that both cost us more than the managed mail servers.
We immediately ruled out AWS SES (even though just about everything else we do we use AWS services for) because they have already burned us in the past. Support is slow at best when there are problems beyond our control and they will shut off email sending at the drop of a hat. We couldn’t risk it.
During what is already our busiest time of the year with the majority of our customers beginning new school years, we quickly went to work enabling mail sending via Mailgun to replace our old servers.
For the most part, this was seamless, required nothing of customers, and worked like a charm. But this is where the real challenging part of sending mass emails becomes so hard.
Bots and Spam
Some of our customers send more mail than others. And some seem to randomly send a higher percentage of emails due to spam too.
Not all ‘spam’ is really ‘spam’. In education, there’s a good amount of mobility where educators retire or change schools, so their email addresses are no longer valid and they bounce. Or, parent provided email addresses, especially when handwritten on those first-day-of-school packets we all get, can be much more frequent than ‘acceptable’ bounce rates allow.
We had one customer where a ton of emails was sent while the entire school was on vacation – almost all accounts sent back a ‘out of office’ auto-responder that was inaccurately picked up as a bounce, and sending for that domain was blocked. That’s what they get for going on vacation while we were working! 🙂
Some sites really do send a lot of actual spam. This is mostly from two sources:
1) Comment notifications – bots that leave spam comments and an email notification is sent to the blog/site owner with spammy links and content included in the body of the emails.
2) Contact form spam – same as above, bots are loving contact forms more and more.
The contact forms may also be quizzes, polls, surveys, etc – bot traffic doesn’t care.
Hit From Both Ends
Almost immediately on moving to Mailgun, with a handful of customers, we started being thrown roadblocks from two sides.
Mailgun monitors for bounce rates and spam and will automatically shut off sending for a domain if its artificial intelligence suspects abuse.
Customers, especially larger universities, also monitor for incoming email and will block our IPs or sending domains when they suspect abuse. Even when it is their own domains and sites sending the emails!
We completely understand and support what both Mailgun and our customers are doing – IP and domain reputation is something that is harder and harder to keep clean. And with phishing campaigns and attacks being more sophisticated than ever, we all need to be on guard.
Plus, please don’t get us started on the additional fronts impacting email delivery, with Internet Service Providers and email services like GMail with their own spam detection and filtering. When an email is sent, there are so many different touchpoints and places that it all could go wrong.
What We Are Doing
For starters, we’ve written and implemented our own mail verification service that checks the format of all email addresses and does its best to ensure that the email address exists before we send the email. This has greatly minimized bounce rates.
We are also working closely with customers to whitelist where we can and to minimize the potential for emails with spam content being sent.
This means that we’ve also had to minimize the use of our Subscribe By Email service that sends emails to a subscription list when new posts are published. We’ve had to implement a limit to subscribers and all new subscribers need to be double-opt-in, meaning, unfortunately, that teachers can’t simply upload a list of emails for students or parents. Those subscribers will need to opt-in before receiving notifications.
What We All Can Do
Anyone with a WordPress site will benefit from working through the following advice on keeping your email reputation clean and your sending of emails flowing.
Check Comment Settings – if you don’t need comments, turn them off. Or, check Settings > Discussion in the WordPress admin and look for the option to turn comments off after a set number of days once a post is published.
Send Mail via SMTP – There are many plugins and services out there for sending mail via a valid SMTP account. For those on CampusPress, reach out to us to ask about options for using your own SMTP services so that mail can be sent from a valid school/university domain.
The ‘no-reply’ Address – a quick google search makes clear that most ‘experts’ think that using a firstname.lastname@example.org email address is a bad idea – mostly for building trust and making it easy for those receiving an email to contact you back. However, there are valid technical reasons to use a no-reply address. In our case, we are almost always sending emails on behalf of domains that we don’t own or manage, so using a no-reply address tells spam-fighting tools to let our emails in the door. Some hosts and tools require the use of a no-reply address unless you configure SMTP as described above.
The ‘reply-to’ Address – many plugins and form tools allow you to set a different ‘reply-to’ address than the address that the email is sent ‘from’. If the ‘send from’ address has proper DKIM/SPF DNS records configured, this should be fine. But if you are finding emails ending up in spam, setting the ‘send from’ and the ‘reply to’ addresses to be the same, may help improve deliverability.
Use Specialized Email Tools – it is tempting to use WordPress plugins for sending newsletters or mass email subscriptions, but the truth is, WordPress is a web publishing platform – not an email service. Your experience will be better using tried and true services like MailChimp, Mailgun, etc.
Follow Anti-Spam Best Practices – If you are using WordPress to send emails of any kind to a list, or you are bulk subscribing email addresses somehow, make sure you have permission to send those emails to the recipients, and even better if you have them opt-in. Having an ‘unsubscribe’ link and your contact information in the footer not only helps you comply with laws and regulations, but it can also help keep the spam-fighting tools from falsely putting you in email jail.
Other Ideas – we haven’t gone this far, yet, but it may help to minimize the content that is sent in the actual email. For example, you can just send a simple one-line email when a form is filled out that lets you know that there is something new and a link to the full content. That way, spam links or content aren’t actually ever sent so they won’t be picked up by spam-fighting tools. Most contact form plugins make it easy to edit and configure the actual content of any email notifications. Sadly, this would make the email notifications way less informative or useful.
We will do our best to ensure that it is smooth sailing from here on.
In the short term, we are actively working on a way to automatically add a captcha on all contact forms that we host for logged out traffic if the user hasn’t already added one. We’ve long done the same thing for comments for logged out users. We will roll this out carefully after going through our extensive testing process.
We will also work closely with Mailgun and any customers impacted by delivery problems.
It’d be nice if bots and spam were a thing of the past, but the reality is that it will probably continue to get worse before it gets better.