Ubuntu 14.04 iperf3 Upstart Script

I had a requirement for running an iperf3 server in daemon mode. This needed to survive server reboots, and therefore start automatically at boot. This example might help network engineers that have limited linux skills (like myself).

I took this example from DigitalOcean, but for the stopping of the service to work properly I had to add expect fork to the code.

You’ll first need to install iperf3 on the machine.

Once iperf3 is installed place the following code in a new text file in /etc/init/ with a .conf extension. I used /etc/init/iperf3.conf

description "Iperf 3 Upstart Script"
author      "Author"

start on filesystem or runlevel [2345]
stop on shutdown
expect fork

script

    exec /usr/bin/iperf3 -s -D
    echo $$ > /var/run/iperf3.pid

end script

pre-start script
    echo "[`date`] Iperf 3 Server Starting" >> /var/log/iperf3.log
end script

pre-stop script
    rm /var/run/iperf3.pid
    echo "[`date`] Iperf 3 Server Stopping" >> /var/log/iperf3.log
end script

The Iperf3 daemon should now run automatically at boot. You can use the sudo service iperf3 stop, sudo service iperf3 start and sudo service iperf3 status commands to manage the service.

ACS 5.2.0.26 Upgrade Hangs

I had a problem when upgrading from 5.1.0.44 to 5.2.0.26.

I’d see the following:

Saved the running configuration to startup successfully

Then the CLI would just hang, after an hour I CTRL+C ‘d out of it.

Stopping the ACS process and re-running the install fixed the issue.

ACS/user# application stop acs

Stopping ACS.
Stopping Management and View............................................................
Stopping Runtime........................
Stopping Database..............
Cleanup....

ACS/user# application upgrade ACS_5.2.0.26.tar.gz repo
Do you want to save the current configuration ? (yes/no) [yes] ?
Generating configuration...
Saved the running configuration to startup successfully

     <Still hung for a very long time here>

% CARS Install application required post install reboot...

Broadcast message from root (pts/0) (Tue May 29 13:48:54 2016):

The system is going down for reboot NOW!






Application upgrade successful
ACS/user#

Running a show process from another terminal after starting the upgrade showed that the install was working fine this time. I could see processes like tar, gzip and rpm running.

Launch an AWS EC2 Instance from an iPad or iPhone

I couldn’t find any instructions online on how to launch an AWS EC2 instance from an iPad or iPhone.

The problem is generating an RSA key from an iPad. The AWS web interface doesn’t lend it’s self to allowing iOS devices to download the private RSA key.

It turns out you can generate an RSA key pair with the free Serverauditor SSH client for iOS. Heres how…

Open up the Serverauditor app on your iPhone / iPad and select ‘Keychain’ from the left hand menu. Serverauditor Keychain

Click the ‘+’ symbol. Plus Symbol

Select ‘New Key’. New Key

Give the new key a name and select ‘Save’. Create RSA Key

Now click ‘Edit’ at the top right. Click Edit

Select the key you just created, this time click ‘Edit’ at the top left. Edit Again

Select the ‘Public’ tab for the public key. Then select ‘Copy’. Public RSA Key

Now log into your AWS EC2 console and select ‘Key Pairs’ from the left hand menu. EC2 Menu

Select ‘Import Key Pair’. Import RSA Key

Enter a key pair name and paste in the public key you copied out of Serverauditor. Select ‘Import’. AWS Key Import Screen

You can see the new public key has been imported. Select ‘Instances’ from the left hand menu. AWS Menu - Instances

Select ‘Launch Instance’. Launch Instance

Select the image that you’d like to use for the new instance. I selected a 64-bit Ubuntu image. AWS Ubuntu Image

Select an instance type. I selected the ‘Free tier eligible’ ‘t2.micro’ instance type. Then select ‘Review and Launch’. AWS Instance Type

Scroll down to ‘Securty Groups’ and ensure that inbound SSH is allowed. If not you’ll need to select ‘Edit security groups’ and add an entry to allow inbound SSH connections.

Select ‘Launch’. Review Instance

From the first drop downs select ‘Choose an existing key pair’ and then from the second drop down, select the key we imported earlier. Tick the checkbox to indicate that understand you’ll lock yourself out of the newly created instance if this goes wrong. Select ‘Launch Instances’. Select Keypair

From the next screen select ‘View Instances’. View Instances

At the bottom of the instances page, select the ‘Description’ tab. Copy the ‘Public IP’ value. Instance IP

Open up Serverauditor, select ‘Quick Connect’ from the left hand menu. Enter the IP address you just copied into the ‘Host’ field. Enter the default username for the instance image you selected. I used Ubuntu, the default username for this image is ‘ubuntu’. Select the key icon. Quick Connect

Select the key you created in the app earlier. Key for connection

Select ‘Connect’. Quick Connect - Connect

Select ‘Continue’. SSH Warning

And there you are… SSH Session

Sipgate on Cisco CME

I saw a fantastic deal on a Cisco 1861 SRST on eBay, so decided that our home needed one.

Beautiful, isn’t it?

Cisco 1861

After registering for a new Sipgate account and using the config from www.netconf.co.uk, I found that the new user IDs couldn’t be entered into CME as a ‘clid network-number’.

Cisco IOS only takes a numerical value as the ‘clid network-number’ but Sipgate user IDs have a letter in them.

As a workaround for this, I manually amended ‘From’ field the SIP Invite message sent to Sipgate.

Instead of:

dial-peer voice 1 voip
  clid network-number xxxxxxxx

I’ve used the following:

voice class sip-profiles 1
 request INVITE sip-header From modify "(<.*:)(.*@)" "\1xxxxxxxx@"

dial-peer voice 1 voip
  voice-class sip profiles 1

This was taken from Cisco’s example found here.

MPLS on 2600XM Series Routers

I had difficulty finding an IOS version that would run MPLS on my 2611XM routers with 96MB RAM and 32MB flash, so thought I’d post the version that I’ve had success with.

The image that worked successfully was c2600-jk9o3s-mz.123-24.bin.

The routers are connected together on Fa0/1 on each router. Fa0/0 goes up to an ESXI box running two IOS XRv PE routers. It’s interesting to watch modern MPLS applications run over these old routers.

Due to the age of the IOS image, the default is Cisco’s TDP as opposed to LDP. Here’s the basic config I used to test MPLS:

R1

mpls label protocol ldp

interface Loopback0
  ip address 172.16.0.3 255.255.255.255

interface FastEthernet0/0
  ip address 10.0.0.9 255.255.255.252
  tag-switching ip

interface FastEthernet0/1
  ip address 10.0.0.6 255.255.255.252
  tag-switching ip

router ospf 1
  network 0.0.0.0 255.255.255.255 area 0

R2

mpls label protocol ldp

interface Loopback0
  ip address 172.16.0.2 255.255.255.255

interface FastEthernet0/0
  ip address 10.0.0.1 255.255.255.252
  tag-switching ip

interface FastEthernet0/1
  ip address 10.0.0.5 255.255.255.252
  tag-switching ip

router ospf 1
  network 0.0.0.0 255.255.255.255 area 0

BSR Hash Calculation

I couldn’t find much information on how the Bootstrap Router hash function is calculated. Most sources teach that the mask dictates how much of the group IP address is used in the hash calculation and that the RP IP address that returns the highest value from the BSR hash algorithm becomes the chosen RP for that group, but don’t give detail on how the BSR hash is calculated.

I found the following algorithm in RFC 4601:

Value(G,M,C(i))=(1103515245 * ((1103515245 * (G&M)+12345) XOR C(i)) + 12345) mod 2^31

Mushing together various python examples from the Internet gave me the following script:

mask=int('11111111111111111111111111111110', 2)
group='239.1.1.1'
rpip='150.1.8.8'

groupbin=''.join(format(int(x), '08b') for x in group.split('.'))
groupdec=int(groupbin, 2)

rpipbin=''.join(format(int(x), '08b') for x in rpip.split('.'))
rpipdec=int(rpipbin, 2)

bsrhash=(1103515245*((1103515245*(groupdec & mask)+12345)^rpipdec)+12345)%(2**31)

print bsrhash

The output generated by the script when amending the ‘mask’, ‘group’ and ‘rpip’ variables matches the hash generated by the ‘show ip pim rp-hash‘ command on IOS.

VPLS Unicast Flooding

Unicast flooding problems, usually associated with switched networks, can also impact VPLS.

If traffic is forwarded asymmetrically through a VPLS instance, unicast flooding of unknown frames can occur. I’ll step through a scenario where this could happen.

I set a LAB up with two CSR 1000V routers acting as PE routers, providing a VPLS instance. GNS3 was used to run the IOS routers acting as CE and C routers.

Base VPLS

In this scenario we could imagine that CE3 and CE4 are Internet routers, they run HSRP on the VPLS facing interfaces for redundancy. CE3 is the Active HSRP device with a HSRP address of 10.0.0.5. On the Internet side of CE3 and CE4 would typically be a BGP connection, although we use static routes in our scenario to easily cause asymmetric routing. With regular BGP connections to an ISP, it’s easy to imagine how asymmetric forwarding could arise.

CE1 and CE2 have static default routes pointing to the HSRP address to provide connectivity. CE3 points to C1 for the address 4.2.2.2. C1 has a static route for 10.0.0.0/24 pointing to CE4 to cause asymmetric routing.

This configuration forces traffic from / to CE1 (10.0.0.1) to 4.2.2.2 to be forwarded in the following direction.

VPLS Loop

This initially works fine. Looking at the MAC address table associated with the bridge domain connected to the VPLS instance on PE2, we can see that an entry for CE1 (CA02.02A0.0008) exists. Any traffic PE2 receives from CE4 destined for CE1 is forwarded across the pseudowire towards the neighbouring PE1 router.

PE2#show bridge-domain 200
Bridge-domain 200 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet2 service instance 20
    GigabitEthernet3 service instance 20
    vfi LAB neighbor 172.16.0.1 300
   MAC address    Policy  Tag       Age  Pseudoport
   0000.0C07.AC01 forward dynamic   297  LAB.1001010
   CA02.02A0.0008 forward dynamic   252  LAB.1001010
   FFFF.FFFF.FFFF flood   static    0    OLIST_PTR:0xe8e8bc00
   CA00.016C.0008 forward dynamic   300  GigabitEthernet2.EFP20
   CA01.01F9.0008 forward dynamic   253  LAB.1001010
   CA02.016C.0008 forward dynamic   253  GigabitEthernet3.EFP20

In order to test for any flooding, we set up an access list egress on the PE2 to CE2 link.

ICMP ACL

We then send 100 pings from CE1 to 4.2.2.2.

CE1#ping 4.2.2.2 rep 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 16/40/172 ms

The ICMP echo reply packets are routed back asymmetrically via PE2, the icmptest ACL does not increment and the unicast ICMP packets are not flooded out of the PE2 to CE2 link.

PE2#show access-lists     
Extended IP access list icmptest
    10 permit icmp any any
    20 permit ip any any (3881 matches)

By default, the MAC address entries are only in place for 300 seconds. As with a switch, this is updated based on the source address of frames received by the PE router.

As traffic is being forwarded asymmetrically and no communication occurs between CE1 and CE2, PE2 does not receive any frames to update the MAC address entry for CE1 (CA02.02A0.0008).

We see that the timer for the CE1 (CA02.02A0.0008) entry is decremented.

PE2#show bridge-domain 200
Bridge-domain 200 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet2 service instance 20
    GigabitEthernet3 service instance 20
    vfi LAB neighbor 172.16.0.1 300
   MAC address    Policy  Tag       Age  Pseudoport
   0000.0C07.AC01 forward dynamic   299  LAB.1001010
   CA02.02A0.0008 forward dynamic   3    LAB.1001010
   FFFF.FFFF.FFFF flood   static    0    OLIST_PTR:0xe8e8bc00
   CA00.016C.0008 forward dynamic   298  GigabitEthernet2.EFP20
   CA01.01F9.0008 forward dynamic   238  LAB.1001010
   CA02.016C.0008 forward dynamic   238  GigabitEthernet3.EFP20

It then times out and is removed.

PE2#show bridge-domain 200
Bridge-domain 200 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet2 service instance 20
    GigabitEthernet3 service instance 20
    vfi LAB neighbor 172.16.0.1 300
   MAC address    Policy  Tag       Age  Pseudoport
   0000.0C07.AC01 forward dynamic   298  LAB.1001010
   FFFF.FFFF.FFFF flood   static    0    OLIST_PTR:0xe8e8bc00
   CA00.016C.0008 forward dynamic   300  GigabitEthernet2.EFP20
   CA01.01F9.0008 forward dynamic   229  LAB.1001010
   CA02.016C.0008 forward dynamic   229  GigabitEthernet3.EFP20

This results in a problem where traffic being received be PE2, destined for CE1 has no entry in the MAC address table of PE2. The frames are therefore flooded by PE2.

ICMP Flooded

To prove that the frames are being flooded, 100 more echo requests are sent to 4.2.2.2.

CE1#ping 4.2.2.2 rep 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/59/288 ms

The asymmetrically forwarded echo reply packets can now be seen on the ACL on the PE2 to CE2 link.

PE2#show access-lists     
Extended IP access list icmptest
    10 permit icmp any any (100 matches)
    20 permit ip any any (4114 matches)

Needless to say, this is at best very inefficient. Bandwidth on all links connected to this VPLS instance on PE2 is wasted.

If we generate any traffic from CE1 that reaches PE2 directly across the VPLS instance, the problem is resolved for a further 300 seconds.

A permanent solution can be put in place by lowering the ARP timeout on the VPLS facing interface on CE4 (although it would be recommended to apply on CE3 VPLS interface too).

Lowering the ARP timeout to 300 seconds on CE4 causes CE4 to send out an ARP request for any IP addresses is it forwarding traffic towards before it’s 300 second timer expires. The ARP reply that comes from CE1 towards CE4 causes the MAC address table on PE2 to be updated, preventing unicast flooding of frames destined for CE1.

Configuration applied as follows.

CE4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CE4(config)#int fa0/0
CE4(config-if)#arp timeout 300

We see the MAC address entry timeout for CE1 decrement down to 89 seconds.

PE2#show bridge-domain 200
Bridge-domain 200 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet2 service instance 20
    GigabitEthernet3 service instance 20
    vfi LAB neighbor 172.16.0.1 300
   MAC address    Policy  Tag       Age  Pseudoport
   0000.0C07.AC01 forward dynamic   299  LAB.1001010
   CA02.02A0.0008 forward dynamic   89   LAB.1001010
   FFFF.FFFF.FFFF flood   static    0    OLIST_PTR:0xe8e8bc00
   CA00.016C.0008 forward dynamic   300  GigabitEthernet2.EFP20
   CA01.01F9.0008 forward dynamic   246  LAB.1001010
   CA02.016C.0008 forward dynamic   278  GigabitEthernet3.EFP20

An ARP request and reply is then exchanged between CE4 and CE1, and the timer is renewed.

PE2#show bridge-domain 200
Bridge-domain 200 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet2 service instance 20
    GigabitEthernet3 service instance 20
    vfi LAB neighbor 172.16.0.1 300
   MAC address    Policy  Tag       Age  Pseudoport
   0000.0C07.AC01 forward dynamic   299  LAB.1001010
   CA02.02A0.0008 forward dynamic   300  LAB.1001010
   FFFF.FFFF.FFFF flood   static    0    OLIST_PTR:0xe8e8bc00
   CA00.016C.0008 forward dynamic   300  GigabitEthernet2.EFP20
   CA01.01F9.0008 forward dynamic   245  LAB.1001010
   CA02.016C.0008 forward dynamic   277  GigabitEthernet3.EFP20

When thousands of ICMP messages are sent to 4.2.2.2 from CE1, the icmptest ACL shows no unicast flooding.

PE2#show access-lists 
Extended IP access list icmptest
    10 permit icmp any any
    20 permit ip any any (179 matches)

MPLS MRU

The purpose of the MPLS MRU (Maximum Receive Unit) is to indicate the maximum size of a packet, including MPLS labels, that the local router router can forward without fragmenting. MRU is only locally significant.

If an incoming packet belonging to a particular FEC (Forwarding Equivalence Class) exceeds the MRU calculated for that FEC, the packet will require fragmentation prior to it being transmitted on the outgoing interface.

The MRU for each FEC varies depending on the MTU of the outgoing interface as well as the number of labels that are added / removed by the local router.

As an example please see the diagram below. R1 is sending an MPLS labeled packet destined to the loopback interface of R3 (172.16.0.3/32).

MPLS MRU

R1 has a local label of 18 for 172.16.0.3/32 and calculates an MRU of 1500 bytes based on a swap operation for R2’s advertised label of label 16.

R2 is popping a label off due to the penultimate hop pop before the IP packet is sent to R3. Based on the knowledge that the outgoing packet to R3 will be 4 bytes shorter, R2 calculates an MRU of 1504.

In this example the maximum sized unlabelled IP packet that could be sent without fragmentation would be 1496 bytes.

The MRU values along a path can be seen by performing an MPLS traceroute. The below traceroute is taken from R1 in the above example.

R1#traceroute mpls ipv4 172.16.0.3/32
Tracing MPLS Label Switched Path to 172.16.0.3/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
  0 10.0.0.0 MRU 1500 [Labels: 16 Exp: 0]
L 1 10.0.0.1 MRU 1504 [Labels: implicit-null Exp: 0] 16 ms
! 2 10.0.0.3 232 ms

In addition to this, the LFIB can also be checked on each of the routers, showing the MRU for FEC 172.16.0.3/32.

R1#show mpls forwarding-table 172.16.0.3 32 detail 
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface          
18     16            172.16.0.3/32     0             Fa1/0      10.0.0.1
    MAC/Encaps=14/18, MRU=1500, Label Stack{16}
    CA0428E00006CA0328E0001C8847 00010000
    No output feature configured


R2#show mpls forwarding-table 172.16.0.3 32 detail 
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface          
16     Pop Label     172.16.0.3/32     36886         Fa0/0      10.0.0.3
    MAC/Encaps=14/14, MRU=1504, Label Stack{}
    CA0628E10008CA0428E000088847 
    No output feature configured


R3#show mpls forwarding-table 172.16.0.3 32 detail 
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface          
None   No Label      172.16.0.3/32     0             aggr-punt
    MAC/Encaps=0/0, MRU=0, Label Stack{}
    No output feature configured

If a push operation occurred along the forwarding path, this would cause a lower MRU on that router. If R2 was to push a label onto the stack for 172.16.0.3/32 rather than pop one off, the MRU on R2 would be 1496 bytes. The interface MTU of 1500 bytes would mean that an ingress packet larger than 1496 bytes would be larger than the 1500 byte interface MTU and therefore require fragmentation.

First Post

I’m Matt, a CCNP engineer working towards a CCNP SP certification.

The purpose of this blog is to help with my studies. I’ll be sharing what I learn with the idea that writing about a subject forces me to know it.

Please feel free to comment on any of my posts, I’m sure I’ll read them all and reply to as many as possible!