Which type of network traffic cannot be managed using congestion avoidance tools?

What Is a Microburst?

A microburst occurs when a large amount of burst data is received in milliseconds. Typical microbursts last for 1 to 100 milliseconds, so that the instantaneous burst data rate is tens or hundreds times the average data rate or even exceeds the port bandwidth.

The NMS or network performance monitoring software calculates the real-time network bandwidth at an interval of seconds to minutes. In such an interval, the network traffic seems to be stable, as shown in Figure 1-1, and no network exception occurs. However, one second is a long period of time for a port that sends and receives data packets at a high speed. At lower levels of granularity (for example, milliseconds), more traffic bursts can be observed, and the traffic rate exhibits a sawtooth pattern, as shown in Figure 1-2. If the sawtooth is sharp, microbursts occur.

Figure 1-1 Traffic statistics at a higher level of granularity

Figure 1-2 Traffic statistics at a lower level of granularity

Causes of a Microburst

Microbursts occur on a network due to the following causes:

1. Service traffic fluctuates. In many service models, user requests and server responses are discrete, making service traffic intermittent and unstable. In addition, services that are sensitive to latency and bandwidth require data to be transmitted quickly, which intensifies service bursts.

   Figure 1-3 Service traffic fluctuation
   

2. The total bandwidth of ingress ports exceeds that of egress ports. For example, a high-bandwidth port sends traffic to a low-bandwidth egress port, or multiple ingress ports working at the same rate send traffic to one egress port.

   Figure 1-4 A high-bandwidth port sending traffic to a low-bandwidth egress port
   

   Figure 1-5 Multiple ingress ports working at the same rate sending traffic to one egress port
   

3. TCP uses the slow start and congestion avoidance mechanisms to send data packets out as soon as possible. Slow start prevents the transmission rate from increasing rapidly. When the throughput reaches the upper limit, the size of TCP sliding window decreases by half, and the data transmission rate declines rapidly. As a result, session traffic exhibits a sawtooth pattern and is bursty. TCP always expects the data in the send window to be sent out as soon as possible. Therefore, after the TCP packet arrival acknowledgment (ACK) arrives, TCP continues to send data by using the sliding window mechanism. Consequently, the packet sending rate becomes unstable and bursts may occur.

   Figure 1-6 Traditional TCP session traffic curve in a sawtooth pattern in the congestion avoidance mechanism
   

Impact and Generation Process of a Microburst

When a microburst exceeds the forwarding capability of a switch, the switch buffers the burst data for transmission later. If the switch does not have sufficient buffer space, the excess data is discarded, causing congestion and packet loss.

The following figure shows a typical millisecond-level microburst scenario. Assume that Port1 and Port2 respectively send 5 MB data to Port3 at a line rate of 10 Gbps. The total transmission rate is 20 Gbps. Port3 supports only a rate of 10 Gbps, which is a half of the total transmission rate. It sends only 5 MB data out and buffers the other 5 MB data for transmission later. However, the switch has only 1 MB buffer space. Therefore, 4 MB data is discarded due to insufficient buffer space. Without considering overhead data such as the inter-frame gap, preamble, frame checksum, and packet header, the microburst duration is 4 ms (5 MB/10 Gbps).

On an actual network, the number of ingress ports is greater than that of egress ports. Therefore, more buffer space is consumed and more packets are discarded due to congestion when microbursts occur.

How to Evaluate the Anti-Burst Capability of a Switch?

RFC 4445 defines the delay factor (DF), which is a key indicator for measuring the transmission quality of data flows. DF indicates the delay and jitter of service traffic. A greater DF value indicates a higher jitter of the service traffic. When a microburst occurs on the network, the DF can be measured in milliseconds.

For switches, the DF can be converted into the required buffer. The conversion formula is as follows:

Where:

T: buffer required by a switch

DF: delay and jitter of service traffic

W: bandwidth of the egress port

U: bandwidth usage of the egress port

A: sum of physical bandwidths of server ports where bursts occur

B: rate of the egress port

C: sum of rates of server ports where no burst occurs

Based on the preceding formula, a larger buffer size of a switch indicates a lower buffer requirement and stronger anti-burst capability of the switch.

Why Cannot a Microburst Be Monitored Through an NMS or the Observing Port of a Device?

Customers are accustomed to using an NMS or observing port to monitor port traffic or the maximum rate of outgoing packets. However, the NMS or observing port cannot monitor microbursts.

Monitoring Port Traffic Through the NMS

The NMS monitors port traffic around the clock and determines the traffic trend on the network based on traffic curves of devices. The traffic curves seem to be smooth on the NMS although microbursts (packet loss) have occurred. The NMS obtains device data in SNMP get mode, which has the following disadvantages:

• Limited management scale: The pull mode is used to obtain monitoring data of devices, and cannot monitor a large number of network nodes.

• Second-level monitoring interval: Generally, an NMS monitors traffic at an interval of 30s. The minimum interval can be set to five seconds and even one second on some third-party NMSs. Therefore, the traffic monitoring accuracy is second-level. In addition, the accuracy of traffic monitored by the NMS depends on the accuracy of the data reported by managed devices. Even if the NMS can calculate the traffic rate at the millisecond level, the accuracy of the traffic graph is still second-level because the data reported by devices is generally second-level.

Why cannot devices support millisecond-level traffic statistics? This is because switches have a large number of ports. When a switch collects packet statistics on ports, the CPU traverses all ports, lowering the performance of obtaining port packet statistics. As a result, the CPU cannot traverse all ports to obtain traffic information during microbursts. In addition, millisecond-level polling on all ports greatly consumes CPU resources, which may affect normal services of the switch. Therefore, the switch cannot capture the microburst information.

Monitoring the Maximum Rate of Outgoing Packets on the Observing Port

When microbursts (packet loss) occur on the network, the maximum rate of outgoing packets on the observing port of a switch is far lower than the port rate, and the port usage is even lower than 10%.

The forwarding chip of the switch records only the total number of received packets. The rate of any packet on the switch can only be calculated by dividing the total number of packets in a period of time by the measurement period. Therefore, the measurement accuracy depends on the length of the measurement period. By default, Huawei CloudEngine series switches calculate the average values of the peak and instantaneous traffic rates based on a period of 300s. The measurement period used for calculating the peak rate is configurable, and the minimum value is 10s. That is, the statistics accuracy is 10s.

How to Detect a Microburst?

A microburst occurs when any of the following conditions is met:

• The number of discarded packets on an egress port of a switch is not 0.
• The following alarm is generated on a port: QOS_1.3.6.1.4.1.2011.5.25.32.4.1.11.51 hwXQoSPacketsDropInterfaceAlarm.
• The forwarding engine detects the following packet loss log: QOS/6/QOS_PACKET_DROP.

Currently, you can detect microbursts by using Telemetry, a third-party packet capture and analysis tool, or the discarded packet capture function.

• Telemetry determines whether a microburst occurs on a port based on the buffer usage and the millisecond-level packet rate statistics of the port. Although the minimum accuracy of the interface rate is only second-level, the collection of buffer usage of interface queues can be accurate to 100 ms. Therefore, the microbursts lasting for 1 ms or even 10 ms cannot be monitored. Telemetry is recommended for routine monitoring of microbursts.
• A third-party packet capture and analysis tool can monitor only packets whose rate is within 10 Gbps Packets cannot be monitored when the packet rate exceeds 10 Gbps. If this method is used to monitor microbursts for a long time, the performance and capacity of the PC or server where the analysis tool is located will deteriorate. Therefore, a third-party packet capture and analysis tool is recommended to detect microbursts.
• The discarded packet capture function can capture only discarded known unicast packets. Because the device sends the captured discarded packets to the CPU at a maximum rate of 1000 pps, this function is applicable to networks with a few packets discarded. If a great number of packets are discarded on a network, this function cannot capture all the discarded packets. Therefore, the discarded packet capture function is recommended to detect microbursts.

The following describes how to use these methods to monitor microbursts in detail:

Using Telemetry to Monitor Microbursts

Telemetry is a device feature in a narrow sense while is a closed automatic O&M system in a board sense. It is divided into the device side and OSS side, and consists of network devices, collector, analyzer, and controller, which can be provided by Huawei or third parties. In Huawei Telemetry system, the network devices refer to CloudEngine switches, the collector and analyzer both refers to iMaster NCE-FabricInsight, and the controller refers to iMaster NCE-Fabric.

Telemetry can be used to detect microbursts on a network. Figure 1-7 shows the Telemetry system architecture and data processing flow.

Figure 1-7 Telemetry system architecture and data processing flow

The following details operations and data processing flow of Huawei Telemetry system.

1. Create a Telemetry static subscription on the device for microburst monitoring.
   1. Create a destination group to which a destination collector belongs, and configure the IP address and port number of the destination collector as well as the protocol and encryption mode for data sending.

      system-view
      telemetry        //Enter the Telemetry view.
      destination-group destination-group-name       //Create a destination group to which a destination collector belongs and enter the destination group view.
      ipv4-address ip-address port port [ vpn-instance vpn-instance ] [ protocol grpc [ no-tls ] ]      //Configure the IP address and port number of the destination collector as well as the protocol and encryption mode for data sending.
      commit

   2. Configure a sampling sensor group, including the sampling path and filter criteria. When the configured sampling path meets the filter criteria, the device reports the sampling path to the collector in a timely manner.

      The following describes how to configure the sampling path and filter criteria for a customized event.

      1. Create a sampling sensor group.

         system-view
         telemetry   //Enter the Telemetry view.
         sensor-group sensor-name     //Create a sampling sensor group and enter the sensor group view.

      2. Configure a sampling path for the customized event and enter the customized event view.

         CloudEngine switches only support the following sampling path for microburst monitoring: huawei-qos: qos/qosQueueBufUsageStats/qosQueueBufUsageStat. The minimum accuracy is 100 ms.

         sensor-path huawei-qos:qos/qosQueueBufUsageStats/qosQueueBufUsageStat self-defined-event

      3. Create a filter for the sampling path.

         filterfilter-name    //Create a filter for the sampling path and enter the filter view.
         op-field field op-type { eq | gt | ge | lt | le } op-value value     //Configure filter criteria.
         condition-relation { and | or }     //Configure the logical operation mode between filter criteria.
         commit

   3. Create a static subscription.

      The static subscription is used to associate the destination group where the destination collector is located with a sampling sensor group and configure gRPC information.

      1. Create a subscription.

         system-view
         telemetry     //Enter the Telemetry view.
         subscription subscription-name       //Create a subscription for associating the destination group where the destination collector is located with a sampling sensor group, and enter the subscription view.
         destination-group destination-name   //Associate the subscription with the destination group where the destination collector is located.
         sensor-group sensor-name   //Associate the subscription with a sampling sensor group.

      2. (Optional) Configure gRPC information.

         All the following configuration items have default values. If the default values meet the requirements, skip this step.

         protocol gRPC [ no-tls ]         //Configure the encryption mode for gRPC. By default, no encryption mode is configured for gRPC.
         local-source-address ipv4 ip-address     //Configure the source IP address for packets to be sent to the CPU. By default, the source IP address is the IP address of the route's outbound interface selected by the socket.
         dscp value     //Configure the DSCP value for data packets to be sent to the CPU. By default, the DSCP value is 0.
         encoding { json | gpb }    //Configure the encoding format for data packets to be sent to the CPU. By default, the encoding format is GPB.

      3. (Optional) Configure the maximum usage of the MPU's CPU resources for Telemetry data sampling.

         By default, Telemetry data sampling can occupy at most 5% of the MPU's CPU resources.

         cpu-usage max-percent usage

      4. Commit the configuration.

         commit

2. Configure sampling parameters for microburst monitoring on the device, including the upper and lower buffer thresholds and sampling interval.
   1. Configure the lower and upper buffer thresholds.

      system-view
      interface interface-type interface-number
      qos [ queue queue-index ] buffer-monitoring percent low low-percent high high-percent    //Configure the lower and upper buffer thresholds for a queue.
      quit     //Exit the interface view.

      When any of the following conditions is met, the device considers that a microburst occurs and then records microburst information, including the microburst occurrence time and corresponding queue buffer usage:

      • If the obtained queue buffer usage is higher than the upper buffer threshold or the lower buffer threshold is set to 0:
        • The difference between the obtained queue buffer usage and the last recorded queue buffer usage is greater than 2%.
        • The difference between the current microburst occurrence time and the last recorded microburst occurrence time is greater than 1 second.
      • If the obtained queue buffer usage is between the upper and lower buffer thresholds:
        • The difference between the obtained queue buffer usage and the last recorded queue buffer usage is greater than 2%, and a microburst occurred last time.
        • The difference between the current microburst occurrence time and the last recorded microburst occurrence time is greater than 1 second, and a microburst occurred last time.
   2. Configure the sampling interval.

      telemetry         //Enter the Telemetry view.
      subscription subscription-name       //Enter the view of the created static subscription.
      sensor-group sensor-name sample-interval sample-interval          //Configure the sampling interval for the associated sampling sensor group.
      commit

3. When the Telemetry sampling interval is reached, the device sends microburst information recorded during this period in Telemetry data format to the collector.

   Frequent reading the buffer usage of interface queues affects the CPU usage of a device. In this case, the device adjusts the Telemetry sampling interval based on the CPU usage.

   • When the CPU usage is lower than or equal to 60%, the device reads the buffer usage of interface queues at the configured sampling interval.
   • When the CPU usage is higher than or equal to 80%, the device stops reading the buffer usage of interface queues.
   • When the CPU usage is higher than 60% and lower than 80%, the device reads the buffer usage of interface queues based on the sampling ratio of 10:1. That is, the device reads the buffer usage of interface queues once every 10 sampling intervals.

4. The analyzer reads and then analyzes the data sent by the collector (Huawei iMaster NCE-FabricInsight).
   1. On the device side, configure the interconnection between iMaster NCE-FabricInsight and a device.
      1. Configure a route for the iMaster NCE-FabricInsight collector cluster to access the data center network.
      2. Configure SNMPv3 to add the device to iMaster NCE-FabricInsight for management.

         system-view 
         snmp-agent sys-info version v3 
         snmp-agent mib-view included iso-view iso    //Create a MIB view and name it iso-view. To ensure that iMaster NCE-FabricInsight can properly manage the device, the MIB view must contain the iso node.
         snmp-agent group v3 snmpv3group privacy write-view iso-view notify-view iso-view    //Create an SNMPv3 group, name it snmpv3group, and associate the write view and notify view named iso-view with the group. By default, the write view has the read permission. Therefore, you do not need to set the read view. The notify view is used to restrict the MIB objects of the device for sending traps to iMaster NCE-FabricInsight.
         snmp-agent usm-user v3 snmpv3user group snmpv3group      //Create an SNMPv3 user and name it snmpv3user, which must be consistent with the security name on iMaster NCE-FabricInsight. The user's security level cannot be lower than the security level of the user group to which the user belongs. Otherwise, the user cannot communicate normally. For example, if the security level of the user group snmpv3group is privacy, the security level of the user snmpv3user must be authentication and encryption.
         snmp-agent usm-user v3 snmpv3user authentication-mode sha2_512     //Set the authentication protocol and authentication password of the user, which must be the same as those on iMaster NCE-FabricInsight. The authentication protocol is SHA2_512 and the authentication password  is 8937561bc.
         Please configure the authentication password (8-255) 
         Enter Password: 
         Confirm Password:
         snmp-agent usm-user v3 snmpv3user privacy-mode aes256     //Set the encryption protocol and encryption password of the user, which must be the same as those on iMaster NCE-FabricInsight. The encryption protocol is AES256 and the encryption password is 68283asd.
         Please configure the privacy password (8-255) 
         Enter Password: 
         Confirm Password: 
         snmp-agent trap enable 
         snmp-agent target-host trap address udp-domain destip-address source loopback 0 udp-port 10162 params securityname snmpv3user v3 privacy     //destip-address is the IP address of iMaster NCE-FabricInsight. (If both the analyzer and collector are deployed, set destip-address to the floating IP address of the collector. If only the analyzer is deployed, set destip-address to the southbound floating IP address of the analyzer.) Loopback 0 is the interface where the source address of trap packets is located. 10162 is the port number of trap packets. You are not advised to change the port number. If you need to, contact technical support personnel. Set securityname to the user name.
         snmp-agent packet max-size 12000    //Set the maximum size of an SNMP packet that the SNMP agent can receive and send to 12000 bytes. The defaul value is 12000 bytes. Since this parameter may be modified, you need to reset this parameter.
         commit

      3. Configure the Syslog protocol so that the device reports exception logs to iMaster NCE-FabricInsight for analysis and display. The logs can be used for issue analysis and troubleshooting assistance.

         system-view 
         info-center enable     //Enable the information center.
         info-center channel 6 name fabricSyslog     //Configure the information center channels, set the channel number to 6, 7, or 8, and set the channel name to fabricSyslog.
         info-center loghost destip-address source-ip source-ip-address { public-net | vpn-instance vpn-instance-name } port 28305 channel fabricSyslog level debugging transport tcp ssl-policy policy1    //Configure the device to send logs of the debugging level to iMaster NCE-FabricInsight.
         undo info-center statistic-suppress enable //Disable suppression of statistics about consecutive repeated logs.
         commit

   2. On iMaster NCE-FabricInsight, perform the configuration for interconnecting with devices, that is, add devices to iMaster NCE-FabricInsight. The procedure is as follows:
      1. Configure an SNMPv3 template.
         1. Choose Inventory from the navigation tree and choose Protocol Template > SNMP.
         2. Click Create and set parameters in the template based on the SNMP protocol information. SNMP parameter settings on iMaster NCE-FabricInsight must be consistent with those on devices.
         3. Click Confirm to create the SNMP protocol parameter template.
      2. Add a single device to iMaster NCE-FabricInsight.
         1. Choose Inventory from the navigation tree and choose Network Resource > Device.
         2. Click Add Device.
         3. In the Basic Information area, set IP Address, Name, Fabric, and Device Role for the device.
         4. In the SNMP area, set SNMP parameters.
         5. Click Confirm. The device is added.
   3. On the following pages of iMaster NCE-FabricInsight, you can view queue metrics.
      1. Choose from the navigation tree and click the Queue tab. Check the used buffer distribution of queues.

         Figure 1-8 Queue Used Buffer Distribution
         

      2. View queue details, including the queue buffer trend, packet loss trend due to queue congestion, and packet loss details due to queue congestion.

         Figure 1-9 Queue Details
         

   4. When a microburst occurs, iMaster NCE-FabricInsight identifies it as a health issue based on abnormal events.
      1. Choose from the navigation tree and choose the Issue tab.
      2. When a microburst occurs, you can check the Service Affected by Switch Port Congestion issue to view the trends of packet loss and packet loss details due to queue congestion, that is, the number of lost packets at different time points.
5. Based on the analysis result, the controller (Huawei iMaster NCE-Fabric) adjusts the network accordingly.
   1. On the device side, complete the configuration for interconnecting with iMaster NCE-Fabric.
      1. Configure a route for iMaster NCE-Fabric to access the data center network.
      2. Configure SNMPv3 to add devices to iMaster NCE-Fabric for management. For details, see 4.a.ii.
   2. On iMaster NCE-Fabric, perform the configuration for interconnecting with the device, that is, add the device to iMaster NCE-Fabric. The procedure is as follows:
      1. Access the Auto Discovery page.

         Choose Network Planning and Construction > Physical Resource > Device > Device Management. The Device Management page is displayed. Select Auto Discovery from the Device Discovery drop-down list box.

      2. Set SNMPv3 parameters.

         Select Customized and set SNMPv3 parameters.

      3. Click Start. The controller starts to discover devices.

         The discovered devices are displayed on the Discovered Devices tab page.

   3. Open the Loop Closure app and choose Event Management from the main menu.
   4. On the Fault Object tab page, view Service Affected by Switch Port Congestion and rectify the fault based on the suggestions provided on the page.

      Figure 1-10 Handling suggestions for Service Affected by Switch Port Congestion
      

Monitoring Microbursts Using Packet Capture and Traffic Analysis Tools

You can use packet capture and traffic analysis tools (such as Wireshark) to monitor microbursts on a network. The procedure is as follows:

1. Use the packet capture function to capture packets and generate a packet file.
   1. Configure outbound mirroring on the interface where packets are discarded.
   2. Mirror traffic to the observing port working at the same rate.
   3. On a PC or server, use the packet capture and traffic analysis tool to capture packets on the observing port and generate a packet file.

      Due to performance limitations, a PC can only capture packets at a maximum rate of 1 Gbps, and a server can capture packets at a maximum rate of 10 Gbps. You are advised to use a high-performance PC. Otherwise, the packet capture function may be affected.

2. Use the traffic analysis function to monitor microbursts.
   1. On the packet capture and traffic analysis tool, open the file of captured packets.
   2. You can view the traffic graph in the IO graph.
   3. Change the traffic unit for the Y axis in the IO graph to Bits and the interval to 1 ms. In this way, you can view millisecond-level traffic bursts.

      Figure 1-11 Wireshark IO graph
      

Monitoring Microbursts Using the Discarded Packets Capture Function

You can configure the discarded packet capture function to detect microbursts. The procedure is as follows:

1. Enable the discarded packet capture function so that the device captures outgoing packets on the ports where microbursts occur.

   system-view 
   qos capture drop-packet enable tcb egress

2. Check information about captured discarded packets.

   display qos capture statistics drop-packet { ip | ipv6 | ethernet } [ tcb ] slot slot-id

How to Mitigate Microbursts on a Data Center Network?

A data center network runs a variety of services, such as the management, storage, big data, computing, and video services, as shown in Figure 1-12.

Figure 1-12 Typical data center networking and traffic paths

• Management services: As there is a small volume of management service traffic on the network, packet loss will rarely affect the services.
• Storage services: The storage backend primarily uses the fibre channel networking and does not involve data center network optimization. The storage frontend accesses compute nodes through the Ethernet and the service traffic is heavy. Once packet loss occurs, services are greatly affected.
• Big data services: Big data services, such as Hadoop clusters, encounter microbursts frequently within the clusters. Data services have a certain tolerance for packet loss. However, a cluster will split once the cluster heartbeat is lost, causing a great impact on services.
• Video services: Service jitter may occur. Once packet loss occurs, user experience is affected, which may cause service complaints. Therefore, the risk of microbursts is high.

Traffic Analysis

1. Access layer
   1. North-south traffic: For example, when each access device has two uplink 40GE interfaces connected to the aggregation layer and 48 downlink 10GE interfaces connected to servers, the oversubscription ratio is 6:1, indicating a high risk of microbursts.
   2. East-west traffic: In the storage, computing, and Hadoop clusters, multiple servers may transmit traffic to one server at the same time, posting a high risk of microbursts.
2. Aggregation layer
   1. North-south traffic: North-to-south traffic has no risk of microbursts because the access traffic is greater than public network traffic. South-to-north traffic may pass through a firewall or not. South-to-north traffic passing through a firewall has a low oversubscription ratio and accounts for only 10% to 25% of the total traffic in the data center, indicating a low risk of microbursts. In addition, switches have much higher forwarding capabilities than firewalls, so the bottleneck lies in the firewalls. Similarly, the risk of microbursts is also low for south-to-north traffic that does not pass through a firewall.
   2. East-west traffic: Devices in the east and west directions have the same bandwidth, and the oversubscription ratio is 1:1. Therefore, the risk of microbursts is low.

Preventive Measures

The most fundamental preventive measure is to reduce microbursts by optimizing server traffic. On the network side, you can take the following methods to reduce the possibility of microbursts and mitigate the impact of them. The following is an example of configurations on Huawei CloudEngine switches:

1. Enable traffic shaping on the device.

   When the latency is controllable and the buffer is sufficient, run the qos queue queue-index shaping { percent cir cir-percent-value [ pir pir-percent-value ] | cir cir-value [ kbps | mbps | gbps ] [ cbs cbs-value [ bytes | kbytes | mbytes ] | pir pir-value [ kbps | mbps | gbps ] [ cbs cbs-value [ bytes | kbytes | mbytes ] pbs pbs-value [ bytes | kbytes | mbytes ] ] ] } command on the downlink interfaces of the upstream switch connected to the congested device to enable the traffic shaping function. This function reduces the instantaneous peak of traffic and controls the burst. Be aware that this measure will increase the packet forwarding latency.

2. Set the burst traffic buffer mode to enhanced on the congested interface.

   Run the qos burst-mode enhanced command on the congested interface to enable enhanced burst traffic buffering, so as to relieve network congestion.

3. Access layer
   1. Reduce the uplink oversubscription ratio. Configure access devices to constitute a Multichassis Link Aggregation Group (M-LAG) so that uplinks of the M-LAG master and backup devices work in per-flow load balancing mode. In this way, the uplink oversubscription ratio is reduced from 6:1 to 3:1.
   2. Avoid the scenario where multiple servers access the same server simultaneously. During network service traffic planning, avoid the scenario where multiple servers access the same server simultaneously. Additionally, expand the capacity of outbound interfaces where microburst traffic occurs frequently to eliminate burst traffic bottlenecks.
   3. Increase the priority of key services. On the device connected to the big data area, configure a traffic classifier to identify heartbeat packets transmitted in the Hadoop storage cluster, and run the remark local-precedence command in the traffic behavior view to increase the priority of the packets, for example, setting the priority value to 4. Then bind the traffic classifier and traffic behavior to a traffic policy, and apply the traffic policy to the interface to protect key services from the impact of microbursts.
4. Aggregation layer

   Configure fixed priorities for different services. On all core and aggregation devices, configure a traffic classifier to differentiate services, and configure a traffic behavior to remark packet priorities (run the remark local-precedence command to re-mark the internal priority, run the remark 8021p command to re-mark the 802.1p priority, and run the remark dscp command to remark the DSCP priority of IP packets). For example, set the priority of management services to 0, the priority of other services and high-performance computing services to 1, the priority of video services to 2, and the priority of storage services to 3. Then bind the traffic classifier and traffic behavior to a traffic policy, and apply the traffic policy to an interface.

How Do I Determine Whether a Microburst Occurs on an Interface Where Packets Are Discarded and the Bandwidth Usage Is Low?

Question

How do I determine whether a microburst occurs on an interface where packets are discarded and the bandwidth usage is low?

Products and Versions Involved

Products involved: CE6851HI, CE6855HI, CE6856HI, CE6860EI, and CE6865EI

Versions involved: V200R005C10 and later versions

Answer

1. Install the patch of V200R005C10SPH025 or a later version.
2. Log in to the device and enter the user view.
3. Enable high-precision microburst detection on the interface, and set a microburst detection duration.

   high-precision rate detection interface interface duration monitortime

   In this command, monitortime specifies the microburst detection duration. The value is an integer in the range from 1 to 1440, in minutes. Set this parameter based on the frequency at which the value of DISCARD is updated on an interface on the live network.

   An example is as follows:

   <HUAWEI>high-precision rate detection interface 10GE 2/0/20 duration 10
   Info: High-precision rate detection is enabled, duration 10 minutes.

4. If the value of DISCARD on the interface increases, check the peak rate of the interface to determine whether a microburst occurs.

   display high-precision rate detection interface interface

   An example is as follows:

   <HUAWEI>display high-precision rate detection result interface 10GE 2/0/12 
   Interface                     : 10GE2/0/12
   Configured detection duration : 10 Min
   Detection start time          : 2020-09-28 10:24:19
   Detection end time            : --
   Average input rate            : 0 Mbits/sec
   Average output rate           : 0 Mbits/sec
   
   Top 10 input rate:
   -----------------------------------------------
   Time                     Input  Rate(Mbits/sec)
   -----------------------------------------------
   2020-09-28 10:24:38.912                       0
   2020-09-28 10:24:38.915                       0
   2020-09-28 10:24:38.909                       0
   2020-09-28 10:24:38.903                       0
   2020-09-28 10:24:38.900                       0
   2020-09-28 10:24:38.893                       0
   2020-09-28 10:24:38.896                       0
   2020-09-28 10:24:38.884                       0
   2020-09-28 10:24:38.887                       0
   2020-09-28 10:24:38.881                       0
   -----------------------------------------------
   
   Top 10 output rate:
   -----------------------------------------------
   Time                     Output Rate(Mbits/sec) //Check the Output Rate column. In this example, the peak rate of the 10GE interface reached 10000 Mbit/s, indicating that a microburst occurred.
   -----------------------------------------------
   2020-09-28 10:24:29.345                    10000 
   2020-09-28 10:24:21.799                    10000
   2020-09-28 10:24:31.751                    10000
   2020-09-28 10:24:38.648                    10000
   2020-09-28 10:24:20.178                    10000
   2020-09-28 10:24:24.954                    10000
   2020-09-28 10:24:26.718                    10000
   2020-09-28 10:24:22.270                    10000
   2020-09-28 10:24:22.129                    10000
   2020-09-28 10:24:20.767                    10000
   -----------------------------------------------

5. Disable high-precision microburst detection on the interface.

   undo high-precision rate detection interface interface

   An example is as follows:

   <HUAWEI>undo high-precision rate detection interface 10GE 2/0/12
   Info: High-precision rate detection is disabled.

Common Misunderstandings About a Microburst

Misunderstanding 1. Why is no alarm reported for microbursts upon buffer exhaustion?

This is because the CPU uses the polling mechanism to query the buffer usage. The CPU will be overloaded if it frequently polls the buffer usage. As a result, the switch responds slowly or even does not respond.

Misunderstanding 2. Currently, the rate and utilization of an interface are low. Therefore, microbursts rarely occur on this interface.

This is a wrong understanding. There is no linear relationship between the average rate and burst rate. A low rate or utilization of an interface does not mean a low rate of burst traffic.

Misunderstanding 3. Switches record the number of packets discarded due to congestion. Therefore, switches cause microbursts or packet loss.

This is also a wrong understanding. Burst traffic is generated by service terminals. Except for a small number of protocol packets, switches do not generate other traffic. However, a traffic burst may be aggravated on switches. For example, if multiple interfaces send data to a single port concurrently, the oversubscription ratio is improper, exacerbating the burst peak. Therefore, the source of traffic bursts needs to be located based on the networking.

Misunderstanding 4: Service traffic is random on servers. Therefore, servers do not encounter heavy traffic bursts.

The NIC of a server sends service packets at the maximum rate supported by the physical interface. For example, a 10GE interface sends service packets at the rate of 10 Gbps, and waits for subsequent packets from the application layer after it finishes sending the preceding packets. Therefore, the physical link transmits packets based on the link rate. That is, the physical link works for a period of time and is idle for a period of time. The overall average bandwidth usage may be low, for example, 20% to 30%. However, the actual bandwidth usage is either 100% or 0%. Such scenario is considered as a traffic burst. When the server sends packets, the bandwidth usage is 100%.

• What Is a Microburst?
• Causes of a Microburst
• Impact and Generation Process of a Microburst
• How to Evaluate the Anti-Burst Capability of a Switch?
• Why Cannot a Microburst Be Monitored Through an NMS or the Observing Port of a Device?
• How to Detect a Microburst?
• How to Mitigate Microbursts on a Data Center Network?
• How Do I Determine Whether a Microburst Occurs on an Interface Where Packets Are Discarded and the Bandwidth Usage Is Low?
• Common Misunderstandings About a Microburst