NSX-v Central CLI Packet Capture

Share on:

In NSX-v 6.2.3 a new feature to aid troubleshooting and operations got introduced, called Central CLI for Packet Capture. The feature is intended to reduce the administrative burden of logging onto any ESXi host to start a packet capture. The ability to perform packet captures for troubleshooting network issues is something all network guys do from time to time and using a network virtualisation platform such as VMware NSX for vSphere it’s no different. Therefore, in this post, I will go through the process of initiating a packet capture using the NSX-v Central CLI for a VM that is misbehaving.

Setting the scene

So let’s begin by first setting the scene, I have the following components in play:

Server NameFunctionDescription
vcsa01vCenter ServerVersion 6.0 U2
nsx01aNSX ManagerVersion 6.2.4
vsphereESXi HostVersion 6.0 U2
ubuntu1Virtual MachineTest VM
CentOSSCP ServerRemote Transfer Server

Identifying the VNIC

The objective is to identify the HOST-ID and VNIC-ID of the VM that we will be performing the packet capture on. The subsequent steps will take you through the process and assumes you already have a SSH connection open to the NSX Manager.

Identify which clusters you have the Distributed Firewall installed on, by executing the command:

1nsx01a.tonysangha.com> show cluster all
2No.  Cluster Name     Cluster Id               Datacenter Name   Firewall Status          
31    MGMT             domain-c26               Datacenter        Enabled                  
42    Compute%2fEdge   domain-c33               Datacenter        Enabled

Identify the hosts in the cluster where the VM is running:

1nsx01a.tonysangha.com> show cluster domain-c26 
2Datacenter: Datacenter               
3Cluster: MGMT                     
4No.  Host Name                Host Id                  Installation Status                               
51    vsphere.tonysangha.com   host-36                  Enabled        

Identify the virtual machines running on the ESXi host:

1nsx01a.tonysangha.com> show host host-36 
2Datacenter: Datacenter               
3Cluster: MGMT                     
4Host: vsphere.tonysangha.com   
5No.  VM Name               VM Id     Power Status
6... 
75    ubuntu1               vm-69     on

Now that we have the VM we want to run the packet capture on, we need to identify the VNIC_ID of the VM‘s vNIC:

 1nsx01a.tonysangha.com> show vm vm-69  
 2Datacenter: Datacenter               
 3Cluster: MGMT                     
 4Host: vsphere.tonysangha.com   
 5Host-ID: host-36                  
 6VM: ubuntu1                                                                    
 7Virtual Nics List:
 81.
 9Vnic Name      ubuntu1 - Network adapter 1                                                
10Vnic Id        5037962b-668d-bfb4-fc48-8d1063000fb6.000                                   
11Filters        nic-47369-eth0-vmware-sfw.2 

Performing the Packet Capture

We have the HOST-ID and VNIC-ID, therefore we can perform the packet capture, however prior to doing so, it’s a good idea to check that the NSX Manager has enough disk space to store the associated capture onto.

To check the disk space, execute the command:

1nsx01a.tonysangha.com# show filesystems 
2Filesystem      Size  Used Avail Use% Mounted on
3/dev/root       5.6G  1.2G  4.1G  23% /
4tmpfs           7.9G  240K  7.9G   1% /run
5devtmpfs        7.9G     0  7.9G   0% /dev
6/dev/sda6        44G   19G   24G  44% /common
7/dev/loop0       16G   45M   15G   1% /common/vdisk_mnt

It goes without saying, before proceeding in a production environment, ensure that the packet capture you are performing does not overwhelm the NSX Manager. If it does, you have most likely filled up the disk space! To rectify this issue, you can try one of two things:

  1. Identify the sessions and delete using the following commands:
1show packet capture sessions
2no debug packet capture session CAPTURE_ID discard
  1. Reboot the NSX Manager Appliance, and all associated packet capture files will be purged from the /tmp/pktcap folder on the NSX Manager Appliance.

Now that I have the warning out of the way let’s crack on with the actual capture.

Enter enable mode; you are only able to perform a packet capture with elevated priviliges

1nsx01a.tonysangha.com> enable 
2Password: 
3nsx01a.tonysangha.com# 

Start the capture on the HOST-ID and VNIC-ID you identified in the previous section

1nsx01a.tonysangha.com# debug packet capture host host-36 vnic 5037962b-668d-bfb4-fc48-8d1063000fb6.000 dir input parameters 
2Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
3Request:
4        Capture host: host-36
5        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
6        Capture point: vnic
7        Capture direction: input
8Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
9Session status: started

What you should see from the excerpt above is that the packet capture starts, a session ID generates and the session file is stored as a pcap in the /tmp/pktcap/ folder. The packet capture in the example above captures all packets inbound to the vSwitch from the vNIC of the VM. Lastly, we didn’t include any capture parameters for any specific traffic that we are looking for, in a production implementation it’s usually a good idea to narrow your focus.

Once you are satisfied you have captured enough traffic, stop the capture:

1nsx01a.tonysangha.com# no debug packet capture session cde461a0-4a2f-4051-8999-ecf552dd28e8 
2Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
3Request:
4        Capture host: host-36
5        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
6        Capture point: vnic
7        Capture direction: input
8Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
9Session status: stopped

Next display the packet capture on the console for quick analysis. Pay attention to the text after parameters switch, I am using TCPDUMP syntax to narrow my focus to traffic to the host 172.16.32.1.

 1debug packet capture display session cde461a0-4a2f-4051-8999-ecf552dd28e8 parameters host 172.16.32.1
 2Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
 3Request:
 4        Capture host: host-36
 5        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
 6        Capture point: vnic
 7        Capture direction: input
 8Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
 9Session status: finished
10Capture packets:
11reading from file /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap, link-type EN10MB (Ethernet)
1222:17:32.869545 ARP, Request who-has 172.16.32.1 tell 172.16.32.11, length 46
1322:17:33.869539 ARP, Request who-has 172.16.32.1 tell 172.16.32.11, length 46
14...

Exporting the capture is a good idea as you most likely will want to open the file in WireShark for further analysis. In this example, I have a CentOS VM (SCP Server) setup for transferring my captures onto. You will need to specify the session ID from your previous capture.

 1nsx01a.tonysangha.com# debug packet capture scp session cde461a0-4a2f-4051-8999-ecf552dd28e8 url vmware@192.168.2.52:file1.pcap
 2Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
 3Request:
 4        Capture host: host-36
 5        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
 6        Capture point: vnic
 7        Capture direction: input
 8Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
 9Session status: finished
10Begin SCP:
11...
12vmware@192.168.2.52's password: 

Lastly, discard the capture session(s) to save disk space on the NSX Manager Appliance

1nsx01a.tonysangha.com# no debug packet capture session cde461a0-4a2f-4051-8999-ecf552dd28e8 discard 
2Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
3Request:
4        Capture host: host-36
5        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
6        Capture point: vnic
7        Capture direction: input
8Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap

There you have it, a quick 101 on how to use the NSX Central CLI to perform packet captures without ever logging onto the ESXi host. Below are some other commands which may prove useful.

 1debug packet capture host HOST_ID vnic VNIC_ID dir (input|output) parameters [options] .
 2debug packet capture host HOST_ID vmknic VMKNIC_NAME dir (input|output) parameters [options] .
 3debug packet capture host HOST_ID vmnic VMNIC_NAME dir (input|output) parameters [options] .
 4debug packet capture host HOST_ID vdrport dir (input|output) parameters [options] .
 5show packet capture session CAPTURE_ID
 6no debug packet capture session CAPTURE_ID
 7no debug packet capture session CAPTURE_ID discard
 8debug packet capture display session CAPTURE_ID parameters [options] .
 9debug packet capture scp session CAPTURE_ID url URL
10debug packet capture clear session CAPTURE_ID
11show packet capture sessions
12show packet capture help host HOST_ID
13show interface