Difference between revisions of "Linux: Network Debugging"

Revision as of 11:05, 23 January 2013

There are several steps to go through to debug issues with the networking.

ping

first and foremost see if you can ping another system on the same network, and system you know you can ping, or a public system. packet dropping will show if there is a problem. Only being able to ping some systems may show a configuration problem - eg if you ping bbc.co.uk it may throw an error that it can't resolve the name, in which case the issue is to do with the DNS server.

[root@head ~]# ping bbc.co.uk
PING bbc.co.uk (212.58.241.131) 56(84) bytes of data.
64 bytes from 212.58.241.131: icmp_seq=1 ttl=243 time=24.2 ms
64 bytes from 212.58.241.131: icmp_seq=2 ttl=243 time=23.9 ms
64 bytes from 212.58.241.131: icmp_seq=3 ttl=243 time=21.9 ms
64 bytes from 212.58.241.131: icmp_seq=4 ttl=243 time=22.4 ms
^C
--- bbc.co.uk ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3452ms
rtt min/avg/max/mdev = 21.970/23.157/24.288/0.984 ms

ifconfig

The first thing you need to do is check ifconfig - this will tell you if something is wrong. ifconfig shows only active connection, adding the -a flag will show all possible connections.

Apparent from teh obvious settings, you should check the traffic over the network. if RX and TX are zero then they are not on the network.

eth0      Link encap:Ethernet  HWaddr 00:25:90:00:82:E0
          inet addr:172.16.1.1  Bcast:172.16.255.255  Mask:255.255.0.0
          inet6 addr: fe80::225:90ff:fe00:82e0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2744484870 errors:0 dropped:16 overruns:16 frame:0
          TX packets:221119412 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4030984039588 (3.6 TiB)  TX bytes:24413935421 (22.7 GiB)
          Memory:fbce0000-fbd00000

eth1      Link encap:Ethernet  HWaddr 00:25:90:00:82:E1
          inet addr:10.16.1.1  Bcast:10.255.255.255  Mask:255.0.0.0
          inet6 addr: fe80::225:90ff:fe00:82e1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15595414 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10112232 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:17678102580 (16.4 GiB)  TX bytes:1805575547 (1.6 GiB)
          Memory:fbc60000-fbc80000

network-scripts

Each interface has a config script. These are found in /etc/sysconfig/network-scripts/

[root@head ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
HWADDR="00:25:90:00:82:E0"
ONBOOT="yes"
IPADDR=172.16.1.1
NETMASK=255.255.0.0
BOOTPROTO="static"

The script is read each time the network interface is started. The scirpts defines what settings will be used.

The HWADDDR defines the mac address of the interface to be used. This must be correct is it is in the file of the network will fail to start.

other network files

resolve.conf

/etc/resolve.conf

Defines the DNS servers for the network. These can also be defined in the network-scripts

network

/etc/sysconfig/

Defines the Hostname of the system, and determines if the networking is avaialable.

70 persistent rules

/etc/udev/rules.d/70-persistent-net.rules

The file defines which mac address is related to which network interface. This is the file that the network-script checks the mac address against. Each interface will have a line defining the drivers and mac address of each interface. This file is auto generated an can get it wrong.

The file can either be edited directly to correct any mistakes. The file can be deleted as it will be regenerated - this may also correct the problem.

# PCI device 0x8086:0x10c9 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:00:82:e0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

kernel modules

The kernel makes use of several possible network modules. The full list of network modules can be found in a location based on the kernel version of the system.

less /lib/modules/2.6.32-220.el6.x86_64/modules.networking

##extract of file##

dummy.ko
dvb-core.ko
e100.ko
e1000.ko
e1000e.ko
enic.ko
epic100.ko
et1011c.ko
ethoc.ko
fakehard.ko
fealnx.ko
firewire-net.ko
fmvj18x_cs.ko

Ethernet Modules

igb
e1000
e1000e
mlx4_en

Infiniband Modules

mlx4_ib

Module Commands

modprobe <module name>

Can be used to install, remove and edit modules. It is the main command when it comes to modules - but it is easier to look at the man pages for full details of its abilities.

module load <module name> / insmod <module name>

Alternative command to install a module

module unload <module name> / rmmod <module name>

Alternative command to remove a module

lsmod

Lists all of the available modules

Module                  Size  Used by

iw_nes                205304  0
libcrc32c               1246  1 iw_nes
iw_cxgb4              119289  0
cxgb4                 346764  1 iw_cxgb4
iw_cxgb3              133370  0
cxgb3                 196201  1 iw_cxgb3
ib_qib                606030  1
mlx4_ib                76784  0
mlx4_en                74570  0
mlx4_core             123595  2 mlx4_ib,mlx4_en
ib_mthca              141055  0

modinfo <module name>

Show all of the settings for a module. Can be useful to check that you are using the correct version.

[root@blade1 ~]# modinfo mlx4_ib
filename:       /lib/modules/2.6.32-220.el6.x86_64/extra/ofa_kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko
version:        1.0-mlnx_ofed1.5.3
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA InfiniBand driver
author:         Roland Dreier
srcversion:     4EA6D043CEB8ED80480B5C8
depends:        mlx4_core,ib_core,ib_mad
vermagic:       2.6.32-220.el6.x86_64 SMP mod_unload modversions
parm:           debug_level:Enable debug tracing if > 0 (int)

lspci

List all of the pci connections in a system. Combines with grep you can check that the connections have been found.

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
06:01.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)

dmesg

dmesg shows all of the boot and kernel messages from the system. Useful to check that nmodules are loaded correctly, and that network interfaces are correct. It can show what mac address is being used for the interface, and which driver.

[root@head ~]# dmesg | grep eth0
igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x4) 00:25:90:00:82:e0
igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
eth0: no IPv6 routers present
8021q: adding VLAN 0 to HW filter on device eth0

ethtool

Gives full information as to the state of a network interface. Useful to check that the connection is working

[root@head ~]# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000003 (3)
        Link detected: yes