Securing the email interconnect
Email is everyone's responsibility

In my frustration with getting basic bufferbloat fixes from a fractional percent of deployment to something fuller, I decided to take a break from it and tackle something else for a bit. Email interchange between providers is, nowadays, largely secured by STARTTLS - submissions, always!

All the major email providers - google, yahoo, verizon, comcast, facebook - use starttls universally. Nearly all of my smaller providers - usually using pretty current code, did also. By traffic, starttls for email interchange is over 60%! What would it take to cross the chasm to 100%?

I decided to take a hard look at email security issues for a variety of other reasons:

• I was annoyed at CISA passing as a budget rider, after being roundly defeated in prior attempts. My faith that companies would not just fall over when key records (such as email and chat records) were demanded by authorities, never high to begin with, collapsed. What I had long wanted for email, in particular, was that once it crossed the door into your home, for it to be protected by all the same age-old protections that protect your home - warrants, and so on. Running my own email servers, inside my home, I have that. Outsourcing it to the big providers of the world, I don’t.

• The logjam vulnerability fix had not got anywhere near the amount of traction heartbleed did, and is easier to fix. It affects far more than just web traffic - imap, ssh, and email are all affected!

• I was also annoyed at the EFF’s let’s encrypt effort - securing web sites uselessly while not bothering to tackle email and chat, bothered me. The foundations of the internet were built on mail, and chat - the web came later! Why was fixing email not on their TODO list, or well supported by their tools?

• I reasoned that I was losing a huge percentage of email, silently, via anti-spam measures that far dwarfed what enabling STARTTLS universally would do.

• In laying out the mandates as we did to the FCC a few months back, we’d kind of missed as to who was responsible for everything… when interchanged.

• I wanted to know by who, why, and where starttls was not being used. Was it a government thing? Laziness? Malice? What?

So…

I setup a test server with encryption required in both directions (on taht.net) and briefly flipped on the same for my main mail server. After sending out a few emails that exercised all my subscribers… I gathered up the email addresses of the 38 out of 521 people I exchanged email with regularly - that my logs showed weren’t using starttls - then turned off mandatory starttls support for outbound email - and wrote:

Subject: Your email provider does not support starttls encryption

In an effort to improve email transport security (and among other things, lower my administrative overhead), I have flipped the bufferbloat.net servers to use starttls encryption always on submissions to our lists.

This means that your attempts to participate on these lists will ultimately be rejected with an error message (3 days, usually), saying TLS could not be negotiated.

(Email to you from these lists will continue to work)

I know this move will be unpopular[1], and I urge you to pester your email server administrators to get starttls working right (notably closing the logjam vulnerability), or to move your email subscription to my lists to a provider doing email more right.

[1] If you want to yell at me about why i did it, see here: https://plus.google.com/u/0/107942175615993706558/posts/GGMuEZ9Dk4s

And then turned mandatory starttls back on.

Some were grateful

“Thanks for the note. I had opportunistic TLS support enabled for my server-side but had forgotten for client side. Should be fixed now with any luck.” - L

Some were annoyed

And asked me to turn off starttls.

Boy, did I get some irate emails. I made ‘em all the more irate by immediately bouncing all non-SSL attempts back to them. Some went to extraordinary lengths - contacting me via linked in, or via phone! to tell me my server was bouncing their messages. Next time I do this I am for sure going to point people at the bofh page so they lightened up first….

Some said “Get offa my lawn!”

Running your own email server is hard, and adding even one more requirement for a successful interchange a burden. New techniques like SPF, dkip, and DANE are one step beyond merely enabling starttls - although it looks like SPF has pretty wide uptake and is as well supported by modern dns servers and users. (more data is needed)

While channelling my inner bofh, I did not intend to anger friends so much as to turn them into enemies. Sadly, I did.

Some were pessimistic

the busted-ass mail servers of the world have first mover advantage– they are at the top of the mountain. i don’t like it either, but i am no more willing to go ssl-only than that other guy is. you, sir, have got to be realistic. - PV

Don Quixote is my hero. And in this case what I’m advocating people try is turning starttls mandatory on input, not on output - reversing Postel’s dictum to - “Be strict about what you receive, and liberal about what you send” - and asking your users to fix it for themselves.

The idea has the benefit of having not been tried. Again - how do we cross the chasm?

Turn on mandatory starttls both ways, gather up who fails, send them an email, then make it optional again - the interruption in service won’t be noticable. The improvement, potentially dramatic. Read on…

(I don’t believe the busted-ass mail servers of the world are at the top of the mountain, either)

Some said their providers wouldn’t change

And asked me to turn off starttls. I figure that if enough people make starttls mandatory, that those providers will change.

Some said the cost of enabling starttls was not low enough

You come across as barging in on our business, telling us it’s easy to do this and ignoring us when we tell you, no, it’s not!

This is VERY true. How others run their mail servers is their business, not mine. For older exchangers, the essential upgrades tend towards hard, and the problems induced by more-constant maintenence and upkeep when there are other things to do, are painful.

The thing is, how I chose to run my mail server and communicate with users of yours, is mine. Emails to me fail with a clear error message when starttls is not present. What more can you want? The world will shift one way, or another. Perhaps I’ll tired of using a separate email account to talk to the remaining 6%, and flip starttls off, in a week or three.

And some just went ahead and did it!

With my argument: it’s nobody’s business what mailing lists you were subscribed to, freelists.org decided to turn starttls on optionally for all their subscribers. WIN! My arocket subscription started working again.

Wow!

In a few weeks of trying, in my spare time, I’d gone 11 for 38, I’d boosted the amount of encrypted email by 10s or 100s of thousands per day, and I felt pretty good about it. It did look like writing some tools, talking to more people, getting better error messages would help in crossing the rest of the chasm… and there were still a few adamant hold-outs - one that fixed his starttls just so he could keep arguing with me about the uselessness of starttls!

With a mixed response like that I decided to dig in, and try a variety of techniques to learn what barriers remained to a full, if basic, starttls deployment for exchanging email between servers, gathering each objection, and I resolved to patiently wearing down my corresondents - I’m not the “nagger-in-chief” of the bufferbloat.net effort for nothing!

My techniques included “public embarrassment”, also. More on that in my next post. Anyway, while attempting to convince multiple correspondents to at least enable starttls optionally, I hit multiple objections, which I’ll document here:

Enabling STARTTLS is a PITA

No, in most cases, if you are already running a submit port, sasl instance, imapd, etc, you are two command lines away from enabling it optionally.

In the time people spent arguing with me, several realized it was faster to just go ahead and fix it.

TLS breaks on many exchangers

The facebook email data set from March, 2014, showed TLS misconfigured on about 1.6% of all sites. My own data was different - only on three exchangers was TLS enabled, but misconfigured. In all cases, it turned out older versions of sendmail use a too small for modern times diffie-helman key vulnerable to the logjam attack. A fix for that is easy, you can add a “dhparameters 1024” to your sendmail.mc configuration file, or better, generate a 2048 bit one. I contacted all three with what the fix was…

Betcha didn’t know your email submission process was vulnerable, huh?

Go fix it.

STARTTLS is insufficient

While debating this, one person pointed me at a page that detailed everything that was wrong with the starttls concept- nearly every problem with which had been fixed in the decade since publication. My conclusion, (not his) was that starttls is everywhere - in imaps, in email submissions - and that the only thing that is still wrong with starttls is that it is vulnerable to active MITM and/or disablement.

Which, as one concern of mine was merely in making passive worldwide survailance harder, didn’t bother me. Learning who was “interfering with the mail” was more interesting. I can always send or get an email via a coffee shop or another path.

Even with only passive worldwide survailence, fixing the traffic analysis problem remains hard, but reducing things down to who was talking to who, without knowing about what, seemed worthwhile. Now, were I a big email exchanger, the problem with cutting off 8% of my users or more would probably get me in hot water with someone, but I decided to not care for my own small site.

It won’t stop spam

Of all the points I ever made, the spam question evoked the most emotional response. Everyone - including me - is scarred about the antispam battle. That’s why this particular bullet item is WAY down on this list, because stopping spam was not at all my intent - my intent was to annoy “the man”! I am more annoyed at losing mail silently due to a random spam block than I am at all these tools that exist for survailing it - and my trust of things like real time block lists has been fading yearly. There was a time when I trusted that rbl’s were working in my best interest - no longer!

I operate a few spam sumps (never mind where!) - requiring STARTTLS on inbound email by default, cut the number of successful spams to those accounts by over 90%!

Even if it stops spam now, it won’t later

The “spam community” has just as many ill-maintained, badly written or abandoned spambots as we have working servers - or more! Taking the next step - of not allowing unencrypted email transport - instantly obsoletes all those bots and reduces the adversary to those actively pursuing their grand game.

There will always be this game - nobody’s come up with a way to outshout the spammers for long.

The sysadmins at clueless companies won’t listen

I targetted users - not sysadmins - with my email. My correspondents are free to nag their sysadmins on their own, or switch to another email provider. There are so many - including all the top 10 - that thinning out the herd of underconfigured/undermaintained email email servers in this way seems worthwhile. Also, unlike anti-spam stuff, where emails “just vanish”, sometimes for no good reason, everyone failing to get a message to me or my lists will get back an informative error message in 3 days. They can choose if they want to keep communicating or not. I’ve made my choice.

And… well… the companies and orgs that are NOT accepting starttls email turned out interesting. More in my next post.

PS: I am not going through the bother of running my own comment server - please use

https://plus.google.com/u/0/107942175615993706558/posts/GGMuEZ9Dk4s to comment on this blog entry.

January 25, 2016
2079 words