Which section in an Ethernet frame contains the data from higher layers such as Internet Protocol?

Dissecting Packets

With some math out of the way, let’s return to the packet shown in Figure 13.2 and break it down by each individual protocol. If you have an understanding of how packets are built, you know that a packet is built starting with the application layer data, and headers from protocols operating on lower layers are added as the packet is being built, moving from top to bottom. This means that the last protocol header that is added is at the Data Link layer, which means that we should encounter this header first. The most common data link layer protocol is Ethernet, but let’s verify that this is what’s being used here.

In order to verify that we are indeed seeing Ethernet traffic, we can compare what we know an Ethernet header should look like to what we have at the beginning of this packet. The Ethernet header format can be found in Appendix 3, but we’ve included it here in Figure 13.13 for convenience.

Looking at the Ethernet header format, you will see that the first 6 bytes of the packet are reserved for the destination MAC address, and the second six bytes, starting at 0x6, are reserved for the source MAC address. Figure 13.14 shows that these bytes do correspond to the MAC addresses of the two hosts in our example. The only other field that is included in the Ethernet header is the two-byte Type field at 0x12, which is used to tell us what protocol to expect after the Ethernet header. In this case, the type field has a hex value of 08 00, which means that the next embedded protocol that should be expected is IP. The length of the Ethernet header is static at 14 bytes, so we know that 00 is the last byte of the header.

From the Trenches

While I’ve included the Ethernet header in this example, the data link layer header is not printed by tcpdump by default. Because all of the examples in this book use Ethernet, the examples moving forward won’t show this header, and will instead begin from the network layer protocol instead.

Since the Ethernet header was kind enough to tell us that we should expect an IP header next, we can apply what we know about the structure of the IP header to the next portion of the packet. We are attempting to break this packet down by individual protocol, so we aren’t concerned about every single value in this header, but there are a few values we will have to evaluate in order to determine the length of the IP header and what protocol to expect next.

First, we need to determine what version of IP is being used here. As we learned earlier, the IP version is identified by the higher order nibble of byte 0x0 in the IP header. In this case, we are dealing with IPv4.

The IP header is variable in length depending on a set of options it can support, so the next thing we need to ascertain is the length of the IP header. Earlier, we learned that the IP header length field is contained in the lower order nibble of byte 0×0 in the IP header, which has a value of 4. This is a computed field however, so we must multiply this field by 5 to arrive at the IP header length, which is 20 bytes. This means that the last two bytes of the IP header are 02 1e.

As our last stop in the IP header, we need to determine what protocol should be expected next in the packet. The IP header gives us this information with the Protocol field at 0x9. Here, this value is 06, which is the value assigned to the TCP protocol (Figure 13.15).

Now that we’ve made our way to the TCP protocol, we must determine whether or not any application layer data is present. To do this, we must determine the length of the TCP header (Figure 13.16), which like the IP header, is variable depending on the options that are used.

This is achieved by examining the TCP data offset field at the higher order nibble of 0×12. The value for this field is 5, but again, this is a computed field and must be multiplied by four to arrive at the real value. This means that the TCP header length is really 20 bytes.

If you count off 20 bytes from the beginning of the TCP header, you will find that there is still data after the end of the header. This is application layer data. Unfortunately, TCP doesn’t have any sort of field that will tell us what application layer protocol to expect in the application, but something we can do is take a look at the destination port field (assuming that this is client to server traffic, otherwise we would look at the source port field) at 0×2:2 in the TCP header. This field has a value of 00 50, which converts to 80 in decimal. Since port 80 is typically used by the HTTP protocol, it might be the case that the data that follows is HTTP data. You could verify this by comparing the hex data with a protocol map of the HTTP protocol, or by just taking that data, from the end of the TCP header to the end of the packet, and converting it to ASCII text (Figure 13.17).

Caution

Just because you find data on a port that is typically associated with a particular service, such as port 80 and HTTP or port 22 and SSH, you shouldn’t always make the assumption that these services are explicitly responsible for the traffic you’re seeing. The fact of the matter is that any service can be configured to run on any port, and attackers will often use this tactic. For instance, it is very common for attackers to run custom protocols used for command and control over port 80. This provides many benefits to the attacker, including the ability to get traffic out of the network since port 80 is almost always allowed out of egress firewalls, and the ability to hide amongst traffic that is erratic and unpredictable because of user-driven HTTP traffic.

The protocol level break down of the packet we’ve just dissected is now shown in Figure 13.18.

Now, let’s talk about some tools that you can use to display and interact with packets.

Sending ARP requests

This is something we may need to do when we want to find out the hardware MAC address of a remote machine (assuming we know its IP address). In other words, when we need to send an ARP request ourselves. We create a transmit packet by assembling an ARP request packet in a locally defined structure S_ARPPKT and a MAC header packet in a locally defined structure S_ETHHDR. We then send both to the nic module transmit function. The data in both structures need to be filled as follows.

The 14 bytes of the Ethernet header (in structure S_ETHHDR):

This is followed by the 28 bytes of the ARP frame (in structure S_ARPPKT):

We transmit the packet, and then we wait for an ARP reply, the reply will contain the remote's hardware address. Helper function mac_ReqArp() has been included, which performs all these actions under one entry point. Note how the protocol constants are specified in network compatible big-endian fashion. This is to avoid an unnecessary format conversion call. As mentioned, care should be taken when porting this part of the program to other compilers or machines. The listing also shows a local helper mac_FillHdr(), which is used to assemble an Ethernet frame for transmission. This function is also called by other modules when preparing their own frames for transmission.

11.6.3 Packet formats

RoCE tunnels most of the IB packet into an Ethernet packet [13]. The Ethernet header provides similar functionality to the IB LRH. It allows Ethernet nodes to communicate with each other in a given subnet. So the RoCE packet does not include the LRH in the tunneled Ethernet packet. LRH fields are mapped into equivalent Ethernet header fields.

Since the Ethernet packet is covered with Frame Check Sequence (FCS), VCRC from the IB packet is not required in RoCE packets. The remaining fields in an IB packet are carried intact in an RoCE packet.

Support for MSS Replacement

PPPoE implementations may have optional support for MSS (Maximum Segment Size) replacement. The PPPoE header adds 8 bytes of data to each Ethernet packet. As a result, the effective MTU becomes 1492 instead of then normal Ethernet MTU of 1500.

When PPPoE is used in a gateway, clients on the network will have no knowledge of this fact. When establishing TCP connections, hosts will advertise a MSS of 1500 rather than the correct value of 1492. This can result in the oversized segments getting dropped by a gateway enabled with PPPoE. This is solved by dynamically replacing the MSS in TCP packets with the correct value.

20.2.2 Coding the traffic

In the following hands-on examples, we consider only the encoding of UDP communications. Specifically, we only consider the payload of the UDP packet. The Ethernet header, the IP header, and the UDP header are stripped from the original packet and stored for later usage. After encoding the payload, coding coefficients and metadata (such as the generation number) are assembled with the original header as a new UDP packet with a coded payload. The structure of the newly generated UDP packet is illustrated in Fig. 20.4. After encoding and assembling, the packet is mirrored back from the coding instance. Recoding is implemented in a similar fashion, whereby the payload is recoded while the headers are stripped. For decoding, the payload of the coded UDP packet is decoded and reattached to the original UDP header.

13.4.1 Ingress Packet Processing

When an IP packet arrives from the network, it first enters the network interface. For the sake of discussion, let us assume that the packet is received on an Ethernet port. The network interface interprets the Ethernet header, detects frame boundaries, and identifies the starting points of the payload and the IP packet in the frame. The L2 processing logic in the card removes the L2 header and constructs a packet context. A packet context is a data structure that essentially serves as a scratch pad for carrying information between different stages of packet processing inside the router. The L2 processing logic appends to the packet context information about L2 headers, for instance, in the case of Ethernet, this would be the source and destination MAC address. In addition to L2 information, the packet context can carry additional information which are shown in Figure 13.4. Use of other fields in the packet context will be revealed later in the discussion.

Now the L2 processing logic peels off the payload, which is an IP packet, and along with the packet context sends it to the L3 processing logic. The L3 processing logic locates the IP header and checks its validity. It extracts the relevant IP header information and stores it in the packet context. The header information includes the destination address, source address, protocol type, DSCP bits (for differentiated services) and if the IP packet is carrying TCP or UDP payload, the destination and the source ports as well.

At this point, the packet context contains enough information for route lookup and classification of the packet. Next, the entire packet context is sent to the forwarding engine in the line card. The forwarding engine searches a table (the forwarding table) to determine the next-hop. The next-hop information contains the egress line card and the outgoing port the packet needs to be transferred. This information is populated in the packet context.

While the forwarding engine is determining the next-hop using the packet context, the L3 processing logic sends the IP packet to be temporarily stored in the buffer memory. When the forwarding engine completes its part, the packet context is appended with the address of the packet in memory and it is sent to the backplane interface.

From the packet context, the backplane interface knows to which line card the packet needs to be transferred. It then schedules the packet for transmission along with the packet context over the backplane. Note that the priority of the packet is taken into account while transmitting on the backplane: higher priority packets need to be scheduled ahead of lower priority packets.

16.3.1.2 Virtual switches

Virtual switches are the key networking components in VMware. A virtual switch is built at runtime from a collection of small key functional units that are listed in the following:

The core Layer 2 forwarding engine; it only processes Layer 2 Ethernet headers.

VLAN tagging, stripping, and filtering units.

Layer 2 security, checksum, and segmentation offload units.

When the virtual switch is built at runtime, ESX Server loads only those components it needs. It installs and runs only what is actually needed to support the specific physical and virtual Ethernet adapter types used in the configuration.

A virtual switch; works in much the same way as a modern Ethernet switch.

It maintains a MAC: port forwarding table, and supports VLAN segmentation at the port level (with access to single and multiple VLANS).

VMware Infrastructure enforces a single-tier networking topology. There is no way to interconnect multiple virtual switches, thus the network cannot be configured to introduce loops. As a result, the Spanning Tree Protocol is not needed and is not present.

Virtual switches make private copies of any frame data used to make forwarding or filtering decisions. This is a critical feature of the virtual switch and is unique to virtual switches. Virtual switches have no dynamic trunking support.

9.2 Carrier Ethernet Service

Carrier Ethernet is a combination of point-to-point and multipoint services for the carriage of Ethernet frames across an MPLS network. In the case of multipoint-based services, forwarding is based on Mac learning and loop prevention is based on split horizon. Service delineation in Carrier Ethernet can be classified by any part of the Ethernet header, such as the VLAN ID or Type of Service (IEEE 802.1p) field in the IEEE 802.1q header. However, services are commonly delineated by a customer’s vlan (C-VLAN), a service provider’s VLAN (S-VLAN), a combination of both, or a combination of service provider VLAN and ranges of customer VLANs. For example, in a simple scenario, each VLAN on an IEEE802.1q-enabled port could be considered a discrete service. Table 9.3 shows a set of possible service examples.

Table 9.3. Possible Service Examples

FCoE

Many data centers include dedicated FC SANs in addition to their Ethernet data networks. For large cloud data centers, there is a strong financial desire to converge these two into a single network. This was one of the driving forces behind the publishing of the Fibre Channel over Ethernet (FCoE) standard (called T11 FC-BB-5) in 2009 by the International Committee for Information Technology Standards. This standard depends on the IEEE DCB standards for efficient operation, which we described in Chapter 5.

When using FCoE, the FC frame is encapsulated with both an FCoE header and an Ethernet header as shown in Figure 8.10. Keep in mind that the FC frame itself is a transport for SCSI commands much like iSCSI is a transport for SCSI, but instead using a FC transport protocol. The Ethernet header contains an EtherType of 0x8906 to identify the frame as FCoE. A 4-bit version number is used in the FCoE header along with start of frame (SoF) and end of frame (EoF) indicators. Reserve bytes are added to maintain the minimum size Ethernet frame when encapsulating smaller FC command frames.

When the FCoE BB-5 standard was issued, it required all FCoE frames to be forwarded through switching devices called Fibre Channel Forwarders (FCFs) as shown in the left side of Figure 8.11. These devices also act as a bridge to traditional FC SANs by encapsulating and de-encapsulating FC frames and providing both FC and Ethernet ports. In the servers, converged network adapters (CNAs) are used to connect to the Ethernet network and can provide the functionality of both a traditional NIC as well as an FCoE HBA by generating and receiving FCoE frames. Special Ethernet switches called Fibre Channel Initiation Protocol (FIP) Snooping Bridges (FSBs) are used which must be connected to FCFs in order for the network to function in a secure and robust manner. The main purpose of FIP snooping is to make sure that only servers that have logged in to the FC network can have access to that network. Snooping is performed by examining certain FC header fields and filtering traffic that is not allowed access.

One of the main industry complaints about the FC-BB-5 standard is that it increases the cost of the network by requiring too many FCFs which are only available from a handful of network equipment vendors. This is because all data must be routed through an FCF and all FSBs must be connected to an FCF. For example, an FCoE storage target connected to the same FSB must have its data routed through the FCF as shown in the left side of Figure 8.11. This increases network congestion and limits the usefulness of FCoE targets. In fact, most FCoE networks have been implemented using FCFs and FSBs with storage targets within the FC SAN.

To improve the situation, several leading original equipment manufacturers including IBM, EMC, and HP are promoting a new FC-BB-6 standard that introduces a new type of Ethernet switch called a Fibre Channel Data Forwarder (FDF). With this standard, the FCF still provides FC services in addition to address assignment and FIP processing, but can delegate forwarding to other DCB enabled switches in the network that have FDF capability.

The FDF capable switches provide FCoE forwarding and zoning based on information provided by the FCF. In addition, the FDFs don’t need to be directly connected to a FCF, providing for the use of many more lower cost FDF switches in the network compared to the number of FCF switches. In the example on the right side of Figure 8.11, the server can connect through the FDF directly to the FCoE target without the need to be routed through the FCF. This new standard should increase the adoption of FCoE in large data center networks.

Networking basics

This book is not meant to provide a deep understanding of network protocols and standards, but instead provides a thorough overview of the technology inside of cloud data center networks. In order to better understand some of the subject presented in this book, it is good to go over some basic networking principals. If you are familiar with networking basics, you may want to skip this section.

The network stack

Almost every textbook on networking includes information on the seven-layer Open Systems Interconnect (OSI) networking stack. This model was originally developed in the 1970s as part of the OSI project that had a goal of providing a common network standard with multivendor interoperability. OSI never gained acceptance and instead Transmission Control Protocol/Internet Protocol (TCP/IP) became the dominant internet communication standard but the OSI stack lives on in many technical papers and textbooks today.

Although the networking industry still refers to the OSI model, most of the protocols in use today use fewer than seven layers. In data center networks, we refer to Ethernet as a layer 2 protocol even though it contains layer 1 and layer 2 components. We also generally refer to TCP/IP as a layer 3 protocol even though it has layer 3 and layer 4 components. Layers 5-7 are generally referred to in the industry as application layers. In this book, we will refer to layer 2 as switching (i.e., Ethernet) and layer 3 as routing (i.e., TCP/IP). Anything above that, we will refer to as the application layer. Figure 1.1 shows an example of this simplified model including a simple data center transaction.

Figure 1.1. Example of a simple data center transaction.

In this simplified example, the sender application program presents data to the TCP/IP layer (sometimes simply referred to as layer 3). The data is segmented into frames (packets) and a TCP/IP header is added to each frame before presenting the frames to the Ethernet layer (sometimes simply referred to as layer 2). Next, an Ethernet header is added and the data frames are transmitted to the receiving device. On the receive side, the Ethernet layer removes the Ethernet header and then the TCP/IP layer removes the TCP/IP header before the received frames are reassembled into data that is presented to the application layer. This is a very simplified explanation, but it gives you some background when we provide more details about layer 2 and layer 3 protocols later in this book.

As an analogy, think about sending a package from your corporate mail room. You act as the application layer and tell your mail room that the gizmo you are holding in your hand must be shipped to a given mail station within your corporation that happens to be in another city. The mail room acts as layer 3 by placing the gizmo in a box, looking up and attaching an address based on the destination mail station number, and then presenting the package to the shipping company. Once the shipping company has the package, it may look up the destination address and then add its own special bar code label (layer 2) to get it to the destination distribution center. While in transit, the shipping company only looks at this layer 2 label. At the destination distribution center, the local address (layer 3) is inspected again to determine the final destination. This layered approach simplifies the task of the layer 2 shipping company.

Packets and frames

Almost all cloud data center networks transport data using variable length frames which are also referred to as packets. We will use both terms in this book. Large data files are segmented into frames before being sent through the network. An example frame format is shown in Figure 1.2.

Figure 1.2. Example frame format.

The data is first encapsulated using a layer 3 header such as TCP/IP and then encapsulated using a layer 2 header such as Ethernet as described as part of the example in the last section. The headers typically contain source and destination address information along with other information such as frame type, frame priority, etc. In many cases, checksums are used at the end of the frame to verify data integrity of the entire frame. The payload size of the data being transported and the frame size depend on the protocol. Standard Ethernet frames range in size from 64 to 1522 bytes. In some cases jumbo frames are also supported with frame sizes over 16K bytes.