Sad tail of misplaced trust

Another infrequent post not related to family; this is in the weeds of geekdom (but perhaps important).

September first Sarah said she was having trouble connecting to the HUD VPN and that their IT support folks were saying it was an issue only impacting Xfinity / Comcast customers. They recommended turning off the Security Edge service.

I spent a bit of time when I got home troubleshooting and had almost convinced myself it was an issue on HUD’s end because there was no IP address returned for hudvpn1.hud.gov for any query I made. Just to be sure I was smarter than their IT folks, we popped Sarah’s laptop onto a Sprint hotspot — that worked fine, no issues connecting at all. Clearly, I was wrong.

Back at the drawing board, I dug a bit deeper and found several references to this sort of issue. And then I found the smoking gun test:

Query Google’s open DNS server normally (plain text, anyone on the network can see the query) and get no address back. But encrypt the query (again to the same Google open DNS server) and get back the correct address.

The implication is Comcast / Xfinity was sniffing the DNS queries sent everywhere (not just to their DNS servers) and dropping responses for hosts they didn’t like. That’s a form of censorship (just one of the ways China’s Great Firewall works to censor their citizen’s Internet access). Not to mention privacy invasion; they seem to be reading deep into packets crossing their network.

That cast a new light on issues I ran into a couple years ago (see this blog post). I had to change the way my DNS servers work when they started failing. After changing our service and being explicit that I wanted no Security Edge any more (having to reduce our bandwidth to avoid paying more for removing a service — which would have been too unpalatable), I reset the DNS servers to be “normal” rather than to forward queries they can’t answer. That worked for a couple weeks, but starting failing again. When talking with Comcast tech support, they pointed out they can only temporarily turn off Security Edge, to really remove it, you need to contact the billing department and request a new bundle that does not include it. It’s not listed on my bill now, but my trust in Comcast has been very shaken so I’m not really sure it’s no longer in our network path.

If there is ever an option to find another ISP for us, I will very likely jump at the chance. I’m convinced Comcast is interfering with my DNS queries still and that’s not the sort of business I want to support.

Why would they do this? My theory, no proof, just thinking out loud: they are both an ISP and a cable company and streaming services are in competition with their cable service. And one option for getting around regional blocks on streaming services (blocks that serve to force you to subscribe to your local cable service) is to route your traffic through a VPN in another region.

The same day Sarah had trouble connecting to her office’s (a federal government agency) VPN, USPTO posted a notice about a similar problem. Both have been resolved, I’m sure it was a mistaken configuration change on the part of Comcast. But it shows how deeply they inspect the Internet traffic on their network and their ability and willingness to cause havoc with a core building block of the Internet.

You would think I would know better, and generally, I do. I was surprised to see this so blatantly exposed. I really enjoy Cory Doctorow’s writing and he’s been warning about this for years. Little Brother has a great scene of a GPG signing party in a cave outside San Francisco as the wiley kids wise up to snooping and work to encrypt their communication. And he touches briefly on this sort of issue in his more recent How to Destroy Surveillance Capitalism (but that is more focused on the issue of monopolies and the fact that we really have only one broadband option — one I’m very much not a fan of).

If you’ve read this far, here are the details of my smoking gun test:

dig hudvpn1.hud.gov @8.8.8.8 timed out
(Comcast appears to have dropped that query from the wire)

kdig -d @8.8.8.8 +tls-ca +tls-host=dns.google.com hudvpn1.hud.gov
provided the correct IP address (encrypted between me and Google so Comcast couldn’t sniff and drop the request)

(kdig comes from the knot-utils package for my flavor of Linux.)

Here’s a similar complaint on Comcast’s forum.

And an option I may look into to longer term (presuming I can’t find a Comcast alternative).

Consider this one more vote in favor of municipal fiber for Alexandria.