Troubleshooting

Introduction

This section covers troubleshooting of problems in FreeNAC.

Troubleshooting PC end-devices

1. Use ‘ipconfig /all’ to get the IP address and MAC address.
2. If the IP address seems wrong
   • what network does it belong to, what is the domain name?
   • can you ping the router?
   • what is the MAC of the router? ‘arp –a’
3. Search the VMPS GUI for the mac of the PC and or its router:
   • Systems: When were they last seen, on what vlan, on which port/switch.
   • Ports: what time was the port last used? What vlan did it have?
4. if the IP address empty:
   • Is there a link light, have we connectivity?
   • If a static address is configured, can pings get through?
   • Ports: what time was the port last used? What vlan did it have?
5. if the IP address is OK
6. can the default router be pinged, does DNS work?

NAC server (advanced)

syslog

Syslog is the most important source of info of what is going on. First try "log" and "logv" commands, which tail all of syslog, and vmps related entries.

Are any MAC being constantly denied access?
Are any switches having problems contacting the vmps servers? They may often send
empty vmps packets (sort of like vmps pings):

tail -f /var/log/messages | egrep "MACNOTRECONFIRMED|000000000000" 

egrep "MACNOTRECONFIRMED|000000000000"  /var/log/messages   

Watch the key stuff: what is allowed or not:

tail -f /var/log/messages | egrep "DENY|ALLOW"

Watch a summary of key events (V3):

tail -f /var/log/messages | egrep "Note"

Watch the logic of decisions:

tail -f /var/log/messages | egrep "vmpsd_external|postconnect"

Watch all relevant events

tail -f /var/log/messages | grep vmpsd

Are syslog messages arriving from the Switches in /var/log/messages? Lets find out what other systems are sending messages to this syslog.

egrep -v `uname -n` /var/log/messages |awk '{print $4}'| sort| uniq -c |sort   

Show a list of device MACs and the vlans attributed to them recently

tail -1000 /var/log/messages | egrep "DENY|ALLOW"  | awk '{print $6, $7, $9}'| sort -u

Checking Running Daemons

Checking SQL:

 pgrep -l mysqld

31419 mysqld_safe
31460 mysqld

Vmps daemon

  pgrep -l vmpsd

31478 vmpsd
31479 vmpsd_external

Lastseen daemon (master only)

  pgrep -l vmps_lastseen

27616 vmps_lastseen

Restarting daemons

In an urgency, if you suspect vmps to be "hung", the easier thing may be simply to reboot the server, which shutsdown all daemons and restarts them:

reboot 

Or restart specific daemons. If you restart mysql, make sure to restart them all:

/etc/init.d/mysql restart
/etc/init.d/vmps restart
/etc/init.d/vmps_lastseen restart

The VMPS daemon is silent dying or will not start

On starting vmps, there are no syslog messages and it is not in the list of running processes:

/etc/init.d/vmps start

Try and start vmps from the command line:

 /opt/nac/bin/vmpsd -e /opt/nac/bin/vmpsd_external.php  -l 0x0204

If you have more than one network interface (check with 'ifconfig -a'), add the '-a A.B.C.D' parameter

 /opt/nac/bin/vmpsd -e /opt/nac/bin/vmpsd_external.php  -l 0x0204 -a 10.10.10.10

Start without the error flags and watch syslog

/opt/nac/bin/vmpsd -e /opt/nac/bin/vmpsd_external.php

Start the vmpsd external script with some dummy input, to show PHP or include errors:

 echo test | /opt/nac/bin/vmpsd_external.php

Network

To see of VMPS requests are really coming from switches:

  tcpdump -n port 1589                   [sniff vmps requests]

Trace requests from one switch:

  tcpdump -i eth1 host MYSWITCH and not port telnet

Note: use '-i' when you have several interfaces.

Example on Solaris (you may have snoop, not tcpdump)

  snoop –d qfe0 port 1589

MySQL Database

Check that the DB is running
mysql opennac

mysql>

// On the master, show the last 5 devices seen on the network
// (This proves that the 'vmps_lastseen' daemon is working)
select name,mac,LastSeen,LastVlan,r_ip from systems WHERE LastSeen IS NOT NULL ORDER BY LastSeen DESC LIMIT 5;
+---------+----------------+---------------------+-----------------+--------------+
| name | mac | LastSeen | LastVlan | r_ip |
+---------+----------------+---------------------+-----------------+--------------+
| u147769 | 0014.2240.fe38 | 2006-06-17 09:05:01 | tbw-drucker/cli | 10.12.59.227 |
| u147869 | 0014.2240.8caa | 2006-06-17 08:56:03 | tbw-drucker/cli | 10.12.59.23 |
| u147814 | 0014.2240.e7bb | 2006-06-17 08:52:41 | tbw-drucker/cli | 10.12.59.144 |
| u147862 | 0014.2240.ccbe | 2006-06-17 08:50:12 | tbw-drucker/cli | 10.12.59.217 |
| u147775 | 0014.2240.f749 | 2006-06-17 08:48:07 | tbw-drucker/cli | 10.12.59.131 |
+---------+----------------+---------------------+-----------------+--------------+

// Show recent server activity
select * from vmpslog ORDER BY datetime DESC LIMIT 100;

// Show server activity for a specific mac:
select * from vmpslog where what like '%000b.dbdc.d6f5%' limit 10;

// Show recent GUI changes / activity
select who,host,datetime,what from history ORDER BY datetime DESC LIMIT 100;

// Show GUI changes for a specif MAC:
select * from history where what like '%000b.dbdc.d6f5%' limit 10;

// If Hub Checking is enabled ($detect_hub=true in config.inc),
// look at the last 5 authentications. Works on Master or slave.
select * from vmpsauth ORDER BY AuthLast DESC LIMIT 5;
+----------------+---------------------+----------+-------------+------------+----------+
| mac | AuthLast | AuthPort | AuthSw | vlan_group | AuthVlan |
+----------------+---------------------+----------+-------------+------------+----------+
| 000b.dbdd.0d62 | 2006-06-08 19:04:18 | 2/11 | 10.12.56.33 | NULL | 100 |
| 0014.2240.e322 | 2006-06-08 19:29:18 | 2/32 | 10.12.56.23 | NULL | 106 |
| 0014.2240.8699 | 2006-06-08 19:29:18 | 2/24 | 10.12.56.23 | NULL | 106 |
| 0008.749d.ad72 | 2006-06-08 19:29:31 | 2/6 | 10.12.56.39 | NULL | 100 |
| 0014.222b.4410 | 2006-06-08 21:29:40 | 2/23 | 10.12.56.43 | NULL | 106 |
+----------------+---------------------+----------+-------------+------------+----------+
5 rows in set (0.00 sec)

Testing VMPS requests

Listen only on localhost, to test:
./vmpsd -d -p 10000 -e decide -l 0x0804 -a 127.0.0.1

Sending test VMPS requests yourself: The tool vqpcli.pl can be used to send test VMPS requests to a designated server, and thus test the server answer if did receive a real quest from a switch.

vqpcli.pl -v domain1 -s 192.168.245.19 -w 10.0.0.1 -i 2/4 -m 0010.0000.0000

Vlan: defaultvlan
MAC Address: 001000000000
Status: ALLOW

Explanation:
- send a request from the ctcs VTP domain, to the VMPS server 192.168.245.19 and request authorisation for mac 0010.a49f.30e3 on port 2/4 of switch 10.0.0.1 .
- the VMPS server allows access and attributes the “defaultvlan”

vqpcli.pl [-s=server -v=VTP domain, -w=Switch -i=Port -m=Mac -c=Last Vlan]

./vqpcli.pl -s 192.168.245.40 -v VTP_DOM -w 193.5.222.1 -i 'Fa0/17' -m '0800.20b0.cb95' -c VLAN1

./vqpcli.pl -s 127.0.0.1  -v VTP_DOM -w 193.5.222.1 -i 'Fa0/17' -m '0800.20b0.cb95' -c VLAN1

Troubleshooting FreeRadius/Samba

Summary of radius logs:

tail -500 /usr/local/var/log/radius/radius.log|egrep "User-Name|Exec-Program output|NAS-|Calling-Station-Id|check_mac" 

tail /usr/local/var/log/radius/radius.log

tail -1000 /usr/local/var/log/radius/radius.log 

Sniff radius requests (to verify radius/switch connectivity)

tcpdump -i eth0 -n host 10.0.0.1    (IP=Access Point)

Watch Samba logs, if there are problems authenticating in the domain:

tail /opt/samba/var/log.nmbd
tail /opt/samba/var/log.smbd
tail /opt/samba/var/log.winbindd
tail /opt/samba/var/log.wb-DOMAINANME