Ethernet bonding with Linux and 802.3ad

Nowadays, most desktop mainboards provide more than one gigabit ethernet port. Connecting them both to the same switch causes most Linux distros by default to get a individual IP on each device and route traffic only on the primary device (based on device metric) or round-robin. A single connection always starts at one IP and so all traffic goes through one device, limiting maximum bandwidth to 1 GBit.

Here comes bonding (sometimes called (port) trunking or link aggregation) to play. It connects two ore more ethernet ports to one virtual port with only one MAC and so mostly one IP address. Wheres earlier only two hosts (with the same OS running) or two switches (from the same vendor) could be connected, nowadays there's a standard protocol which makes it easy: LACP which is part of IEEE 802.3ad. Linux supports difference bonding mechanisms including 802.3ad. To enable bonding at all there are some kernel settings needed:

Device Drivers  --->
[*] Network device support  --->
<*>   Bonding driver support

After compiling and rebooting, we need a userspace tool for configuring the virtual interface. It's called ifenslave and provided with the Linux kernel. You can either compile it by hand

/usr/src/linux/Documentation/networking
gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave
cp ifenslave /sbin/ifenslave

or install it by emerge if you run Gentoo Linux:

emerge -va ifenslave

Now we can configure the bonding device, called bond0. Firstofall we need to set the 802.3ad mode and the MII link monitoring frequency by

echo "802.3ad" > /sys/class/net/bond0/bonding/mode
echo 100 >/sys/class/net/bond0/bonding/miimon

Now we can up the device and add some ethernet ports:

ifconfig bond0 up
ifenslave bond0 eth0
ifenslave bond0 eth1

Now bond0 is ready to be used. Run a dhcp client or set an IP by

ifconfig bond0 192.168.1.2 netmask 255.255.255.0

These steps are needed on each reboot. If you're running gentoo, you can use baselayout for this. Add

config_eth0=( "none" )
config_eth1=( "none" )
preup() {
 # Adjusting the bonding mode / MII monitor
 # Possible modes are : 0, 1, 2, 3, 4, 5, 6,
 #     OR
 #   balance-rr, active-backup, balance-xor, broadcast,
 #   802.3ad, balance-tlb, balance-alb
 # MII monitor time interval typically: 100 milliseconds
 if [[ ${IFACE} == "bond0" ]] ; then
  BOND_MODE="802.3ad"
  BOND_MIIMON="100"
  echo ${BOND_MODE} >/sys/class/net/bond0/bonding/mode
  echo ${BOND_MIIMON}  >/sys/class/net/bond0/bonding/miimon
  einfo "Bonding mode is set to ${BOND_MODE} on ${IFACE}"
  einfo "MII monitor interval is set to ${BOND_MIIMON} ms on ${IFACE}"
 else
  einfo "Doing nothing on ${IFACE}"
 fi
 return 0
}
slaves_bond0="eth0 eth1"
config_bond0=( "dhcp" )

to your /etc/conf.d/net. I found this nice preup part in the Gentoo Wiki Archive.

Now you have to configure the other side of the link. You can either use a Linux box and configure it the same way or a 802.3ad-capable switch. I used an HP Procurve 1800-24G switch. You have to enable LACP on the ports you're connected:

Now everything should work and you can enjoy a 2 GBits (or more) link. Further details can be found in the kernel documentation.

Read More

EtherChannel vs LACP vs PAgP

What is EtherChannel?

EtherChannel links formed when two or more links budled together for the purposes of aggregating available bandwidth and providing a measure of physical redundancy. Without EtherChannel, only one link will be available while the rest of the links will be disabled by STP, to prevent loop.

p/s# Etherchannel is a term normally used by Cisco, other vendors might calling this with a different term such as port trunking, trunking (do not confuse with cisco’s trunk port definition), bonding, teaming, aggregation etc.

What is LACP

Standards-based negotiation protocol, known as IEEE 802.1ax Link Aggregation Control Protocol, is simply a way to dynamically build an EtherChannel. Essentially, the “active” end of the LACP group sends out special frames advertising the ability and desire to form an EtherChannel. It’s possible, and quite common, that both ends are set to an “active” state (versus a passive state). Once these frames are exchanged, and if the ports on both side agree that they support the requirements, LACP will form an EtherChannel.

What is PAgP

Cisco’s proprietary negotiation protocol before LACP is introduced and endorsed by IEEE. EtherChannel technology was invented in the early 1990s. They were later acquired by Cisco Systems in 1994. In 2000 the IEEE passed 802.3ad (LACP) which is an open standard version of EtherChannel.

EtherChannel Negotiation

An EtherChannel can be established using one of three mechanisms:

• PAgP - Cisco’s proprietary negotiation protocol
• LACP (IEEE 802.3ad) – Standards-based negotiation protocol
• Static Persistence (“On”) – No negotiation protocol is used

Any of these three mechanisms will suffice for most scenarios, however the choice does deserve some consideration. PAgP, while perfectly able, should probably be disqualified as a legacy proprietary protocol unless you have a specific need for it (such as ancient hardware). That leaves LACP and “on“, both of which have a specific benefit.

PAgP/LACP Advantages over Static

a) Prevent Network Error

LACP helps protect against switching loops caused by misconfiguration; when enabled, an EtherChannel will only be formed after successful negotiation between its two ends. However, this negotiation introduces an overhead and delay in initialization. Statically configuring an EtherChannel (“on”) imposes no delay yet can cause serious problems if not properly configured at both ends.

b) Hot-Standby Ports

If you add more than the supported number of ports to an LACP port channel, it has the ability to place these extra ports into a hot-standby mode. If a failure occurs on an active port, the hot-standby port can replace it.

c) Failover

If there is a dumb device sitting in between the two end points of an EtherChannel, such as a media converter, and a single link fails, LACP will adapt by no longer sending traffic down this dead link. Static doesn’t monitor this. This is not typically the case for most vSphere environments I’ve seen, but it may be of an advantage in some scenarios.

d) Configuration Confirmation

LACP won’t form if there is an issue with either end or a problem with configuration. This helps ensure things are working properly. Static will form without any verification, so you have to make sure things are good to go.

To configure an EtherChannel using LACP negotiation, each side must be set to either active or passive; only interfaces configured in active mode will attempt to negotiate an EtherChannel. Passive interfaces merely respond to LACP requests. PAgP behaves the same, but its two modes are refered to as desirable and auto.

3750X(config-if)#channel-group 1 mode ?
  active     Enable LACP unconditionally
  auto       Enable PAgP only if a PAgP device is detected
  desirable  Enable PAgP unconditionally
  on         Enable Etherchannel only
  passive    Enable LACP only if a LACP device is detected

Conclusion

Etherchannel/port trunking/link bundling/bonding/teaming is to combine multiple network interface.

PAgP/LACP is just a protocol to form the etherchannel link. You can have etherchannel without protocol, but not advisable.

Sources:

http://en.wikipedia.org/wiki/EtherChannel
http://packetlife.net/blog/2010/jan/18/etherchannel-considerations/
http://wahlnetwork.com/2012/05/09/demystifying-lacp-vs-static-etherchannel-for-vsphere/

Read More

VDSL2 vectoring explained

Several system vendors including Adtran, Alcatel-Lucent and ZTE have announced vectoring technology that boosts the performance of very-high-bit-rate digital subscriber line (VDSL2) broadband access technology. Vectoring is used to counter crosstalk - signal leakage between the telephony twisted wire pairs that curtails VDSL2's bit rate performance – as is now explained.

Technology briefing

There is a large uncertainty in the resulting VDSL2 bit rate for a given loop length. With vectoring this uncertainty is almost removed

Paul Spruyt, Alcatel-Lucent

Two key characteristics of the local loop limit the performance of digital subscriber line (DSL) technology: signal attenuation and crosstalk.

Attenuation is due to the limited spectrum of the telephone twisted pair, designed for low frequency voice calls not high-speed data transmission. Analogue telephony uses only 4kHz of spectrum, whereas ADSL uses 1.1MHz and ADSL2+ 2.2MHz. The even higher speed VDSL2 has several flavours: 8b is 8.5MHz, 17a is 17.6MHz while 30a spans 30MHz.

The higher frequencies induce greater attenuation and hence the wider the spectrum, the shorter the copper loop length over which data can be sent. This is why higher speed VDSL2 technology requires the central office or, more commonly, the cabinet to be closer to the user, up to 2.5km away - although in most cases VDSL2 is deployed on loops shorter than 1.5km.

The second effect, crosstalk, describes the leakage of the signal in a copper pair into neighbouring pairs. “All my neighbours get a little bit of the signal sent on my pair, and vice versa: the signal I receive is not only the useful signal transmitted on my pair but also noise, the contributed components from all my active VDSL2 neighbours,” says Paul Spruyt, xDSL technology strategist at Alcatel-Lucent.

Typical a cable bundle comprises several tens to several hundred copper pairs. The signal-to-noise ratio on each pair dictates the overall achievable data rate to the user and on short loops it is the crosstalk that is the main noise culprit.

Vectoring boosts VDSL2 data rates to some 100 megabits-per-second (Mbps) downstream and 40Mbps upstream over 400m. This compares to 50Mbps and 20Mbps, respectively, without vectoring. There is a large uncertainty in the resulting VDSL2 bit rate for a given loop length. "With vectoring this uncertainty is almost removed," says Spruyt.

Vectoring

The term vectoring refers to the digital signal processing (DSP) computations involved to cancel the crosstalk. The computation involves multiplying pre-coder matrices with Nx1 data sets – or vectors – representing the transmit signals.

The crosstalk coupling into each VDSL2 line is measured and used to generate an anti-noise signal in the DSLAM to null the crosstalk on each line.

To calculate the crosstalk coupling between the pairs in the cable bundle, use is made of the ‘sync’ symbol that is sent after every 256 data symbols, equating to a sync symbol every 64ms or about 16 a second.

Each sync symbol is modulated with one bit of a pilot sequence. The length of the pilot sequence is dependent on the number of VDSL2 lines in the vectoring group. In a system with 192 VDSL2 lines, 256-bit-long pilot sequences are used (the next highest power of two).

Moreover, each twisted pair is assigned a unique pilot sequence, with the pilots usually chosen such that they are mutually orthogonal. “If you take two orthogonal pilots sequences and multiply them bit-wise, and you take the average, you always find zero,” says Spruyt. "This characteristic speeds up and simplifies the crosstalk estimation.”

A user's DSL modem expects to see the modulated sync symbol, but in reality sees a modulated sync symbol distorted with crosstalk from the modulated sync symbols transmitted on the neighbouring lines. The modem measures the error – the crosstalk – and sends it back to the DSLAM. The DSLAM correlates the received error values on the ‘victim’ line with the pilot sequences transmitted on all other ‘disturber’ lines. By doing this, the DSLAM gets a measure of the crosstalk coupling for every disturber – victim pair.

The final step is the generation of anti-noise within the DSLAM.

This anti-noise is injected into the victim line on top of the transmit signal such that it cancels the crosstalk signal picked up over the telephone pair. This process is repeated for each line.

VDSL2 uses discrete multi-tone (DMT) modulation where each DMT symbol consists of 4096 tones, split between the upstream (from the DSL modem to the DSLAM) and the downstream (to the user) transmissions. All tones are processed independently in the frequency domain. The resulting frequency domain signal including the anti-noise is converted back to the time domain using an inverse fast Fourier transform.

The above describes the crosstalk pre-compensation or pre-coding in the downstream direction: anti-noise signals are generated and injected in the DSLAM prior to transmission of the signal on the line.

For the upstream, the inverse occurs: the DSLAM generates and adds the anti-noise after reception of the signal distorted with crosstalk. This technique is known as post-compensation or post-coding. In this case the DSL modem sends the pilot modulated sync symbols and the DSLAM measures the error signal and performs the correlations and anti-noise calculations.

Challenges

One key challenge is the amount of computations to be performed in real-time. For a fully-vectored 200-line VDSL2 system, some 2,600 billion multiply-accumulates per second - 2.6TMAC/s - need to be calculated. A system of 400 lines would require four times as much processing power, about 10TMAC/s.

Alcatel-Lucent’s first-generation vectoring system that was released end 2011 could process 192 lines. At the recent Broadband World Forum show in October, Alcatel-Lucent unveiled its second-generation system that doubles the capacity to 384 lines.

For larger cable bundles, the crosstalk contributions from certain more distant disturbers to a victim line are negligible. Also, for large vectoring systems, pairs typically do not stay together in the same cable but get split over multiple smaller cables that do not interfere with each other. “There is a possibility to reduce complexity by sparse matrix computations rather than a full matrix,” says Spruyt, but for smaller systems full matrix computation is preferred as the disturbers can’t be ignored.

There are other challenges.

There is a large amount of data to be transferred within the DSLAM associated with the vectoring. According to Alcatel-Lucent, a 48-port VDSL2 card can generate up to 20 Gigabit-per-second (Gbps) of vectoring data.

There is also the need for strict synchronization – for vectoring to work the DMT symbols of all lines need to be aligned within about 1 microsecond. As such, the clock needs to be distributed with great care across the DSLAM.

Adding or removing a VDSL2 line also must not affect active lines which requires that crosstalk is estimated and cancelled before any damage is done. The same applies when switching off a VDSL2 modem which may affect the terminating impedance of a twisted pair and modify the crosstalk coupling. Hence the crosstalk needs to be monitored in real-time.

Zero touch

A further challenge that operators face when upgrading to vectoring is that not all the users' VDSL2 modems may support vectoring. This means that crosstalk from such lines can’t be cancelled which significantly reduces the vectoring benefits for the users with vectoring DSL modems on the same cable.

To tackle this, certain legacy VDSL2 modems can be software upgraded to support vectoring. Others, that can't be upgraded to vectoring, can be software upgraded to a ‘vector friendly’ mode. Crosstalk from such a vector friendly line into neighbouring vectored lines can be cancelled, but the ‘friendly’ line itself does not benefit from the vectoring gain.

Upgrading the modem firmware is also a considerable undertaking for the telecom operators especially when it involves tens or hundreds of thousands of modems.

Moreover, not all the CPEs can be upgraded to friendly mode. To this aim, Alcatel Lucent has developed a 'zero-touch' approach that allows cancelling the crosstalk from legacy VDSL2 lines into a vectored lines without CPE upgrade. “This significantly facilitates and speeds up the roll-out of vectoring” says Spruyt.

Read More

How-To Configure NIC Teaming on Windows for HP Proliant Server

NIC Teaming means you are grouping two or more physical NIC (network interface controller card) and it will act as a single NICs. You may call it as a Virtual NICs. The minimum number of NICs which can be grouped (Teamed) is Two and the maximum number of NICs which you can group is Eight.

HP Servers are equipped with Redundant Power Supply, Fan, Hard drive (RAID) etc. As we have redundant hardware components installed on same server, the server will be available to its users even if one of the above said components fails. In the similar manner, by doing NIC Teaming (Network Teaming), we can achieve Network Fault tolerance and Load balancing on your HP Proliant Server.

HP Proliant Network Adapter Teaming (NIC Teaming) allows Server administrator to configure Network Adapter, Port, Network cable and switch level redundancy and fault tolerance. Server NIC Teaming will also allows Receive Load balancing and Transmit Load balancing. Once you configure NIC teaming on a server, the server connectivity will not be affected when Network adapter fails, Network Cable disconnects or Switch failure happens.

To create NIC Teaming in Windows 2008/2003 Operating System, we need to use the HP Network Configuration Utility. This utility is available for download at Driver & Download page of your HP Server (HP.com). Please install the latest version of Network card drivers before you install the HP Network Configuration Utility. In Linux, Teaming (NIC Bonding) function is already available and there is no HP tools which you need to use to configure it. This article will focus only on Windows based NIC teaming.

HP Network Configuration Utility (HP NCU) is a very easy-to-use tool available for Windows Operating System. HP NCU allows you to configure different types of Network Team, here are the few: 

1. Network Fault Tolerance Only (NFT)
2. Network Fault Tolerance Only with Preference Order
3. Transmit Load Balancing with Fault Tolerance (TLB)
4. Transmit Load Balancing with Fault Tolerance and Preference Order
5. Switch-assisted Load Balancing with Fault Tolerance (SLB)
6. 802.3ad Dynamic with Fault Tolerance

Network Fault Tolerance Only (NFT)

In NFT team, you can group two to eight NIC ports and it will act as one virtual network adapter. In NFT, only one NIC port will transmit and receive data and its called as primary NIC. Remaining adapters are non-primary and will not participate in receive and transmit of data. So if you group 8 NICs and create a NFT Team, then only 1 NIC will transmit and receive data, remaining 7 NICs will be in standby mode. If the primary NIC fails, then next available NIC will be treated as Primary, and will continue the transmit and receive of data. NFT supports switch level redundancy by allowing the teamed ports to be connected to more than one switch in the same LAN.

Network Fault Tolerance Only with Preference Order:

This mode is identical to NFT, however here you can select which NIC is Primary NIC. You can configure NIC Priority in HP Network Configuration Utility. This team type allows System Administrator to prioritize the order in which teamed ports should failover if any Network failure happens. This team supports switch level redundancy.

Transmit Load Balancing with Fault Tolerance (TLB):

TLB supports load balancing (transmit only). The primary NIC is responsible for receiving all traffic destined for the server, however remaining adapters will participate in transmission of data. Please note that Primary NIC will do both transmit and receive while rest of the NIC will perform only transmission of data. In simpler words, when TLB is configured, all NICs will transmit the data but only the primary NIC will do both transmit and receive operation. So if you group 8 NICs and create a TLB Team, then only 1 NIC will transmit and receive data, remaining 7 NICs will perform transmission of data. TLB supports switch level redundancy.

Transmit Load Balancing with Fault Tolerance and Preference Order:

This model is identical to TLB, however you can select which one is the Primary NIC. This option will help System Administrator to design network in such a way that one of the teamed NIC port is more preferred than other NIC port in the same team. This model also supports switch level redundancy.

Switch-assisted Load Balancing with Fault Tolerance (SLB):

SLB allows full transmit and receive load balancing. In this team, all the NICs will transmit and receive data hence you have both transmit and receive load balancing. So if you group 8 NICs and create a SLB Team, all the 8 NICs will transmit and receive data. However, SLB does not support Switch level redundancy as we have to connect all the teamed NIC ports to the same switch. Please note that SLB is not supported on all switches as it requires Ether Channel, MultiLink Trunking etc.

802.3ad Dynamic with Fault Tolerance

This team is identical to SLB except that the switch must support IEEE 802.3ad Link Aggregation Protocol (LACP). The main advantage of 802.3ad is that you do not have to manually configure your switch. 802.3ad does not support Switch level redundancy but allows full transmit and receive load balancing.

How to team NICs on HP Proliant Server:

To configure NIC teaming on your Windows based HP Proliant Server, you need to download HP Network Configuration Utility (HP NCU). This utility is available for download at HP.com. Once you download and install NCU, please open it. To know how to open NCU on your HP Server, please check my guide provided below.

Guide: Different ways to open HP NCU on your server

If you are using Windows 2012 Server Operating System on your HP Server, then you could not use HP Network Configuration Utility. We need to use the inbuilt network team software of Windows here. Please check the below provided article about Windows 2012 Network team to learn more.

Guide: NIC Teaming in Windows Server 2012

Let us continue with our Windows 2008/2003 based HP NCU here. Once you open NCU, you will find all the installed network cards are listed in it. As you can find from below provided screenshot, we have 4 NICs installed. Here, we will team first two NICs in NFT mode.

Let’s start

1. The HP Network Configuration Utility Properties window will look like the one provided below.

2. Select 2 NICs by clicking on it and then click Team button.

3. HP Network Team #1 will be created as shown below.
4. Select HP Network Team #1 and click on Properties button to change team properties

5. The Team Properties Window will open now.

6. Here you can select the type of NIC team you want to implement (See below screenshot).

7. Here, I will select NFT from the Team Type Selection drop down list.
8. Click OK once you selected the desired Team type.

9. Now you will be at below provided screen now. Click OK to close HP NCU.

10. You will receive confirmation window prompting you to save changes, Click Yes.

11. HP NCU will configure NIC teaming now, the screen may look like the one provided below.

12. This may take some time, once Teaming is done, below provided window will be shown.

13. Open HP NCU, you could find that HP Network Team is in Green color. Congrats

Read More

Windows 7 Link aggregation / NICs Teaming

Intel NIC’s 802.3ad Link Aggregation in Windows 7? – [H]ard|Forum

http://hardforum.com/showthread.php?t=1762818

If anyone else is trying to do this, I figured it out. Follow these directions for Intel NIC’s. The feature is not included in Windows 7, so the NIC drivers have to support it. You have to be logged…

Network Connectivity — How do I use Teaming with Advanced Networking Services (ANS)?

http://www.intel.com/support/network/sb/cs-009747.htm

Adapter teaming with Intel® Advanced Network Services (ANS) uses an intermediate driver to group multiple physical ports. Teaming can be used to add fault tolerance, load balancing, and link…

Read More