If you are using an Amazon Application Load Balancer, and forwarding traffic to internal services using HTTPS, the ALB will not validate the certificate presented by the internal service before forwarding the traffic.
So we're clear here, let's say you are running a web server on Amazon ECS. The webserver is configured to present TLS certificates to incoming requests, receive encrypted TLS traffic. The web server is part of a ELB v2 Target Group. There are two hops in this flow:
-
Customer iPhone/laptop/whatever connects to Amazon ALB. You can upload a certificate to Amazon to present to the customer. The customer's iPhone/browser/whatever will (hopefully) verify that certificate before sending requests to the ALB.
-
The ALB forwards the request to your webserver. ALB will look up the right Listener for the request, and then forward it to a ELB v2 Target Group. You can configure the Target Group to receive requests over HTTP or HTTPS.
-
If you choose HTTPS, the ALB will establish a connection and request a certificate from a random host in the Target Group. It will not validate that certificate; it will just send the traffic.
Here is the configuration for a Target Group. There's no check box for "validate HTTPS traffic from internal service."
The entire point of HTTPS is to encrypt traffic. Otherwise, a random person snooping on the network could present a weak certificate and send back whatever data it wants. We learned in 2014 that the NSA was doing this, between and inside data centers and at key points in the US.
It's unacceptable for a major Internet service in 2018 to blindly accept certificates presented by an internal service without validating them.
For the moment, I suggest using the Network Load Balancer type, which forwards the raw TCP traffic to your machine. You don't get any of the nice features of an ALB, but at least you will have the ability to reject raw traffic. If you know of other providers that offer load balancers with TLS certificate validation, please send me an email.
Update: Please read the reply from AWS.
Liked what you read? I am available for hire.
FYI, Classic ELB supports backend auth (manually adding a public key to the ELB)
https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-create-https-ssl-load-balancer.html#config-backend-auth
Hi Kevin!
I’m the lead engineer for Amazon ELB and I’m also the main author of Amazon s2n, our Open Source implementation of TLS/SSL, and a contributor to the TLS/SSL standards. Hopefully I’m qualified to chime in!
You’re right that ALB does not validate the certificates on targets, but it’s important to understand the context that ALBs run in to see why this is still a pending item on our roadmap, rather than something we’ve shipped already as a “must have”.
The role that server certificates play in TLS is to authenticate the server, so that it can’t be impersonated or MITM. ALBs run exclusively on our Amazon VPC network, a Software Defined Network where we encapsulate and authenticate traffic at the packet level. We believe that this protection is far stronger than certificate authentication. Every single packet is being checked for correctness, by both the sender and the recipient, even in Amazon-designed hardware if you’re using an Enhanced Networking interface. We think it’s better than the ecosystem where any CA can issue a certificate at any time, with still limited audit controls (though certificate transparency is promising!).
The short of it is that traffic simply can’t be man-in-the-middled or spoofed on the VPC network, it’s one of our core security guarantees. Instances, containers, lambda functions, and Elastic Network Interfaces can only be given IPs via the secure and audit-able EC2 APIs. In our security threat model, all of this API and packet level security is what plugs in the role performed by server certificates.
This contrasts with the older EC2 classic network, a big shared network, which is why classic load balancers do support backend authentication.
We actually find that many customers actually load their targets and backends with “invalid” certificates that are self-signed or expired, because it’s so operationally hard to stay up-to-date and it’s hard to automate, even with projects like LetsEncrypt, when your instances are inherently unreachable on the internet.
All that said, we’ll be adding support for certificate validation, probably including pinning and private CAs! Used well with good operational controls it can be a measure of defense in depth, and it’s important for cases such as targets hosted on less secure private networks such as on-premesis data-centers.
Thanks Colm for the excellent reply. I’ve pulled this out into a full post so it gets the attention it deserves, and will link to it from this post shortly. https://kev.inburke.com/kevin/aws-alb-validation-tls-reply/
Hello Colm!
I’m wondering if the same considerations you’ve laid down for the traffic handling within the VPC are also fully relevant in a scenario where the ALB is a private one (behind a VPC Link and an API Gateway).
In this configuration the only component that is publicly accessible is the API Gateway. Can it be said that the communications between the API GW and the ALB (though PrivateLink) also occur only over a Software Defined Network, thus the network-level man-in-the-middle concern being irrelevant? If so, may it be said that the entire area of communications downstream from the API GW are not really needing HTTPS, in this scenario?
Or there are other considerations at play?