NOTE: For this post, you should be familiar with the basic concepts of WebSockets and ELB.
Recently, a project required us to implement WebSockets. This posed quite a challenge, because:
- AWS ELB doesn’t support WebSockets on HTTP/HTTPS (Layer 7).
- Switching ELB protocols to TCP/SSL does the trick, but we do not receive X-Forwarded-For header from ELB anymore, i.e. client IP information is not obtained.
The solution we developed addresses both points above and covers the following points:
- WebSockets should work on ELB with TCP/SSL protocols enabled, while client IP information should be retained.
- All the TCP traffic should be redirected to SSL, allowing only secure communication to take place.
Fixing #1: Enable Proxy Protocol on ELB
AWS introduced support for proxy protocol on ELB. After enabling proxy protocol on ELB the X-Forwarded-For
can be received even at a TCP/SSL level. Once proxy protocol is enabled, ELB prepends a human-readable header to the request header, which contains connection specific information. For nginx, running on instances behind ELB, to understand this additional header configuration change is essentially needed. We recommend performing the configuration change prior to enabling proxy protocol on ELB, otherwise all requests will return a 400 - Bad Request error.
This post was quite helpful in having required nginx configuration:
Using Proxy Protocol With Nginx
You should be very careful about following configuration changes:
- Define custom log format: Add this log format before including any virtual host configuration.
log_format elb_log '$proxy_protocol_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent"';
- Define default virtual host:
server { listen 80 proxy_protocol; server_name _; #Default virtual host }
You might have already noticed that the default virtual host listens to proxy_protocol
. A rule of thumb for virtual host definitions is to have one default virtual host listening to proxy_protocol
. The first virtual host that listens to proxy_protocol
returns 400 - Bad Request
to all requests received; all subsequent virtual hosts work properly. We’ve filed a bug with nginx for this issue.
- Virtual hosts for application:
#Virtual host accepting TCP traffic server { listen 80 proxy_protocol; server_name mysite.com; access_log logs/mysite.access.log elb_log; location / { root html; rewrite ^(.*)$ https://$server_name$1 permanent; index index.html index.htm; } } #Virtual host accepting SSL traffic server { listen 81 proxy_protocol; server_name mysite.com; access_log logs/mysite.access.log elb_log; …… }
The first virtual host accepts TCP traffic and all requests are redirected to SSL. Both virtual hosts listen to proxy_protocol and a custom log format is associated with corresponding access log files.
Currently, this feature can only be enabled through API’s and AWS CLI. Considering the ease of use, we’ll enable proxy protocol on ELB from AWS CLI:
- Create ELB Policy for proxy protocol:
aws elb create-load-balancer-policy --load-balancer-name my-loadbalancer --policy-name EnableProxyProtocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=True
- Set ELB Policy for Backend Servers:For Port 80:
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-loadbalancer --instance-port 80 --policy-names EnableProxyProtocol
For Port 81:aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-loadbalancer --instance-port 81 --policy-names EnableProxyProtocol
In above commands, place your load balancer name instead of my-loadbalancer
.
Fixing #2: Redirect TCP traffic to SSL
The nginx configuration done in step one, above, takes care of this issue partially. The ELB listeners need to be changed, this can be done through AWS Management Console. Change your settings to the following:
The ELB should be all set to support WebSockets!
Like what you read? Join our community to get more technical information, chances to win prizes, and more: built.io/community