Raw socket checkers

A collection of check scripts that uses RAW sockets to force the destination
MAC address, suitable to be used for load balancers (LB) in direct routing
mode (LVS-DR) to ensure that the real server is indeed answering to packets
that have as destination IP the Virtual IP (VIP).

The problem

Load balancers in direct routing mode (LVS-DR) usually works with this setup:

• The load balancer has the VIP assigned to one interface as an alias.
• The incoming traffic into the LB to the VIP is modified at Layer 2 in the
  ethernet frame to change the destination MAC address from the one of the
  VIP interface on the LB itself to the one of the real server’s interface
  to which the LB decided to route the traffic.
• The real servers have the VIP assigned to their loopback interface as an
  alias in order to answer to traffic that has the VIP as destination IP.
• The real servers are configured to not announce and reply to ARP requests the
  IPs in their loopback interface.

IF one of the real server is missing the configuration of the VIP as an alias
of the lo interface, usually this is what happens:

• All the incoming traffic to that real server is rejected as not recognized by
  the Kernel as valid.
• The checks of LB management softwares like ldirectord or keepalived
  will not fail because their checks are performed using the real server IP as
  destination address.
• The statistics on the LB don’t show any anomaly because from the LB‘s
  perspective the traffic was properly routed to the real server and because
  there is no NAT in place the responses from the real servers go directly to
  the client without any knowledge of it by the LB.
• Any monitor on the VIP service will not clearly detect the issue and at
  most there will be some flapping checks when the request is routed to the
  failing real server.
• The LB will continue to route the traffic to the real server until a manual
  intervention will fix the situation.
• The whole check is following a path that is different from the real traffic.

Available alternatives

Configuration management tools

While the enforcement of the VIP configuration on the loopback of the real
servers could be done with any configuration management tool, it will usually
not ensure that the interface is actually up at all time.

Alarming tools

While an alarming tool could obviously check the status of the loopback
interface on the real server at all time, it will still just raise an alarm and
a manual or automatic intervention would be needed. Also in the case of an
automatic intervention the usual delay between checks in the alarming tools
will not avoid that some traffic got lost.

The proposed solution

The proposed solution is to check the real servers using the same kind of
packets that characterize the normal routed traffic from the LB.
To achieve that a RAW socket is used to bypass both the ARP and the Kernel
networking stack that would not allow to use the VIP as the destination IP
in the checks given that the VIP is configured and announced on the LB
itself and the packets would not exit the LB at all.
The real server MAC address is resolved through ARP (for IPv4) and will be
used as the destination MAC address for all the packets. The destination IP
instead will be set to the VIP. RAW sockets are used in order to be able to
set all those values manually, by-passing the Kernel TCP stack.

Caveat

In case of a cluster of LBs, usually the checks on the real servers are
performed by all the LBs, not only by the master one. Because the LBs in
stand-by/backup mode don’t have the VIP configured, the Kernel would reject
the traffic in response to the checks performed using the RAW sockets. In
this scenario the possible solutions are:

1. Block with iptables the outgoing reset packets sent by the Kernel.
2. Add the VIP address to the loopback interface to all the LBs in the
   cluster.
3. Fallback to standard checks using the real server IP on stand-by/backup
   LBs

The first two solutions requires a manual configuration on the LBs, while the
third one requires that the check is aware of the LB role. In order to make
the checks aware of the role of the LB to use the third solution, an option
is available on all checks (-r) that allow to specify the path of a file
where the LB role is saved. The software that is in charge of changing the
LB role will need to update this file accordingly.

List of available checks

Because of the usage of RAW sockets, all checks need to be run as root
or have the CAP_NET_RAW capability.

• TCP check
• HTTP GET check (BETA)
• HTTP check (BETA)

Dependencies

• GNU C compiler (i.e. gcc package).
• GNU C Library: Development Libraries and Header Files (i.e. install
  libc6-dev package (or equivalent) on Debian based distros, glibc-devel on
  RedHat based ones).
• [HTTP checks only] OpenSSL development files (i.e. install libssl-dev
  package on Debian based distros, openssl-devel on RedHat based ones).

Compilation

To compile all checks just run:

make

• The binaries are created in the bin/ directory.
• All the other build artifacts are created in the build/ directory.
• Run make clean to clean all compiled artifacts.
• The checks can be compiled individually, using make tcp, make http_get
  or make http.
• All checks were tested on Debian 8 (jessie) and Fedora 23 Server.

TODO

• Add Tests.
• Extract common parameters and options to a common parser.
• Support fragmented responses in HTTP checks.
• Support packet loss in HTTP checks.
• Accept also HTTP responses with LF only.
• Add IPv6 versions of the checks.
• Evaluate options to eventually support SSL.

Useful resources

• C Language Examples of IPv4 and IPv6 Raw Sockets for Linux by P. David Buchan. A very
  useful resource about RAW sockets in C with examples for all major
  network protocols.