Netapp monitoring with check_netapp.pl

In my previous post  I showed how to enable Nagios to monitor a Netapp device. The only issue that you may have noticed is the output of the Nagios check. Having Nagios return “SMNP 73 OK” is not a particularly interesting result. We would also like some performance metrics so RRDTool can graph it on Nagios.

So how can we achieve this? Nagios Exchange provides netapp.pl which is a Perl script which allows you to monitor disk usage and formats the results in to a reasonable format. The script takes several arguments by default:

/opt/nagios/libexec/check_netapp.pl
Missing arguments!
check_netapp -H <ip_address> -v variable [-w warn_range] 

[-c crit_range]
 [-C community] [-t timeout] [-p port-number]
 [-P snmp version] [-L seclevel] [-U secname] [-a authproto]
 [-A authpasswd] [-X privpasswd] [-o volume]

The ones we are interested in are

-H <ip_address>
-v <variable>
-o <volume>
-w -c <warn_range crit_range>
-t <timeout>

So for our test volume we would like to check the disk usage so we would execute the following command to check the diskspace used:

/opt/nagios/libexec/check_netapp.pl -H netapp -v DISKUSED 

-o /vol/test/ -w 60 -c 70 -t 60
 DISKUSED CRITICAL - /vol/test/ - total: 600 Gb - used 501 

Gb (84%) - free: 98 Gb|NetApp 

/vol/test/ Used Space=501GB;360;420;0;600

This provides a much nicer output that the standard SNMP result. So to have Nagios use this command we add the command to the commands.cfg file:

define command{
        command_name check-test-volume
        command_line $USER1$/check_netapp.pl -H netapp -v 

DISKUSED -o /vol/test/ 

-w 80 -c 90 -t 60
}

After this is added we add the service to our groups config file:

define service
        use                             netapp-template-unix
        host_name                       netapp
        service_description             Check Test Volume Disk 

Space
        check_command                   check-test-volume
}

To activate this check restart Nagios and wait for the monitoring results to appear on our Nagios site. If you have RRDTool and pnpnagios installed they should begin graphing the results shortly after the first checks are recieved.

Keeping your NetApp in check with Nagios

Recently we took delivery of a new NetApp. Two 3210′s with roughly 50TB of storage across five disk shelves. For monitoring we use Nagios which can check the NetApp effectively and alert us of any problems quickly. Luckily enough our existing NetApp is already being monitored by Nagios so getting the checks up and running is not a problem. Nagios relies on SNMP to monitor NetApp storage. Normally SNMP checking runs fine but we have been seeing issues with Nagios timing out when the NetApp is heavily utilized. According to NetApp this is a known bug and we should not see it in this new setup (DATA ONTAP 8).

Our first step is to retrieve all the SNMP OIDs using our snmpwalk command:

       snmpwalk -v 1  -c public netapp 1.3.6.1.4.1.789 > netapp-snmpwalk1-oid.txt

The 1.3.6.1.4.1.789 number is the top-level OID. This is our starting point for finding information such as CPU usage, disk space and other system status information. If you have created a volume on your NetApp then you should be able to find its corresponding OID. To find this we need to download the MIB zip folder from the NetApp site. This folder contains a file called traps.dat which lists all the OIDs needed. After unzipping the file and opening the traps.dat the first keyword we look for is dfFileSys which is the base OID for the filesystem.

       dfFileSys snmp.1.3.6.1.4.1.789.1.5.4.1.2.1

The base OID we are interested in is 1.5.4.1.2.1 so any volumes created will have the OID 1.5.4.1.2.2 or 1.5.4.1.2.3 and so on with the final digit incrementing as new volumes are added. Searching through our snmpwalk file we find the OID:

        SNMPv2-SMI::enterprises.789.1.5.4.1.2.5 = STRING: "/vol/test/"

This corresponds to the dfFileSys base OID with 1.5.4.1.2.5 being the corresponding OID for our test volume. Now we have the volume OID so we must look for the disk space utilized OID. Searching through the traps.dat file find dfPerCentKBytesCapacity which indicates in percentage the amount of disk space utilized on the volume.

        dfPerCentKBytesCapacity snmp.1.3.6.1.4.1.789.1.5.4.1.6.1
 

The test volume identifier is 1.5.4.1.2.5 and the reference number needed is the last digit 5. The dfPerCentKBytesCapacity in the traps.dat file shows the base OID 1.5.4.1.6.1 . The final digit will be the reference number so in the above OID we replace the 1 with 5  which will become 1.5.4.1.6.5.  Let’s query it and see what happens:

        snmpwalk -v 1  -c public netapp 1.3.6.1.4.1.789.1.5.4.1.6.5


result:
        SNMPv2-SMI::enterprises.789.1.5.4.1.6.5 = INTEGER: 1

Looking at FilerView I can see that 1% is in fact being used. But to be sure lets check the other volumes. Searching our OID list I find the other volumes at:

    SNMPv2-SMI::enterprises.789.1.5.4.1.2.3 = STRING: "/vol/test1/" 

    SNMPv2-SMI::enterprises.789.1.5.4.1.2.7 = STRING: "/vol/test2/" 

So with our reference numbers “3″ and “7″ let us see how much disk space is used:

        snmpwalk -v 1 -c public swe-filer1 1.3.6.1.4.1.789.1.5.4.1.6.3

result:

        SNMPv2-SMI::enterprises.789.1.5.4.1.6.3 = INTEGER: 2

and:

        snmpwalk -v 1 -c public swe-filer1 1.3.6.1.4.1.789.1.5.4.1.6.7

result:

        SNMPv2-SMI::enterprises.789.1.5.4.1.6.7 = INTEGER: 0

Looking at FilerView I can see that indeed test1 has 2% used and test3 has 0% used. Now that we have the commands needed let’s plug them in to Nagios.

In the commands.cfg file create the command as follows: 

    define command{ 

    command_name check-test-diskspace 

    command_line $USER1$/check_snmp -H netapp -C public -o .1.3.6.1.4.1.789.1.5.4.1.6.5 -w 80 -c 90 

    } 

Note the ‘.’ preceeding the OID.  Next we add it as a service: 

          define service{ 

    use                      netapp-template 

    host_name                netapp service_description      

    Check Test Volume Disk Space check_command            

    check_test_diskspace

    }

Restart Nagios and it should start checking the new service. This is only one of many services you can monitor through Nagios. For example we monitor NFS operations, CIFS operations, CPU load, Uptime, Fans, Global Status and many more. All you need to find is the corresponding OID in the traps.dat file, send some test queries to make sure you are hitting the right object and then add it as a Nagios service. You could also create a script which lists all OIDs and its corresponding object. With this all you would need is the object name (/vol/test for example) in your Nagios command rather than the OID string. This would centralize all your OIDs so you could just add new ones to your script as new volumes are created.