NSX-v Central CLI Packet Capture

In NSX-v 6.2.3 a new feature to aid troubleshooting and operations got introduced, called Central CLI for Packet Capture. The feature is intended to reduce the administrative burden of logging onto any ESXi host to start a packet capture. The ability to perform packet captures for troubleshooting network issues is something all network guys do from time to time and using a network virtualisation platform such as VMware NSX for vSphere it’s no different. Therefore, in this post, I will go through the process of initiating a packet capture using the NSX-v Central CLI for a VM that is misbehaving.

Setting the scene

So let’s begin by first setting the scene, I have the following components in play:

Server Name Function Description
vcsa01 vCenter Server Version 6.0 U2
nsx01a NSX Manager Version 6.2.4
vsphere ESXi Host Version 6.0 U2
ubuntu1 Virtual Machine Test VM
CentOS SCP Server Remote Transfer Server

Identifying the VNIC

The objective is to identify the HOST-ID and VNIC-ID of the VM that we will be performing the packet capture on. The subsequent steps will take you through the process and assumes you already have a SSH connection open to the NSX Manager.

Identify which clusters you have the Distributed Firewall installed on, by executing the command:

nsx01a.tonysangha.com> show cluster all
No.  Cluster Name     Cluster Id               Datacenter Name   Firewall Status          
1    MGMT             domain-c26               Datacenter        Enabled                  
2    Compute%2fEdge   domain-c33               Datacenter        Enabled

Identify the hosts in the cluster where the VM is running:

nsx01a.tonysangha.com> show cluster domain-c26 
Datacenter: Datacenter               
Cluster: MGMT                     
No.  Host Name                Host Id                  Installation Status                               
1    vsphere.tonysangha.com   host-36                  Enabled        

Identify the virtual machines running on the ESXi host:

nsx01a.tonysangha.com> show host host-36 
Datacenter: Datacenter               
Cluster: MGMT                     
Host: vsphere.tonysangha.com   
No.  VM Name               VM Id     Power Status
... 
5    ubuntu1               vm-69     on

Now that we have the VM we want to run the packet capture on, we need to identify the VNIC_ID of the VM‘s vNIC:

nsx01a.tonysangha.com> show vm vm-69  
Datacenter: Datacenter               
Cluster: MGMT                     
Host: vsphere.tonysangha.com   
Host-ID: host-36                  
VM: ubuntu1                                                                    
Virtual Nics List:
1.
Vnic Name      ubuntu1 - Network adapter 1                                                
Vnic Id        5037962b-668d-bfb4-fc48-8d1063000fb6.000                                   
Filters        nic-47369-eth0-vmware-sfw.2 

Performing the Packet Capture

We have the HOST-ID and VNIC-ID, therefore we can perform the packet capture, however prior to doing so, it’s a good idea to check that the NSX Manager has enough disk space to store the associated capture onto.

To check the disk space, execute the command:

nsx01a.tonysangha.com# show filesystems 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       5.6G  1.2G  4.1G  23% /
tmpfs           7.9G  240K  7.9G   1% /run
devtmpfs        7.9G     0  7.9G   0% /dev
/dev/sda6        44G   19G   24G  44% /common
/dev/loop0       16G   45M   15G   1% /common/vdisk_mnt

It goes without saying, before proceeding in a production environment, ensure that the packet capture you are performing does not overwhelm the NSX Manager. If it does, you have most likely filled up the disk space! To rectify this issue, you can try one of two things:

  1. Identify the sessions and delete using the following commands:
    show packet capture sessions
    no debug packet capture session CAPTURE_ID discard
    
  2. Reboot the NSX Manager Appliance, and all associated packet capture files will be purged from the /tmp/pktcap folder on the NSX Manager Appliance.

Now that I have the warning out of the way let’s crack on with the actual capture.

Enter enable mode; you are only able to perform a packet capture with elevated priviliges

nsx01a.tonysangha.com> enable 
Password: 
nsx01a.tonysangha.com# 

Start the capture on the HOST-ID and VNIC-ID you identified in the previous section

nsx01a.tonysangha.com# debug packet capture host host-36 vnic 5037962b-668d-bfb4-fc48-8d1063000fb6.000 dir input parameters 
Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
Request:
        Capture host: host-36
        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
Session status: started

What you should see from the excerpt above is that the packet capture starts, a session ID generates and the session file is stored as a pcap in the /tmp/pktcap/ folder. The packet capture in the example above captures all packets inbound to the vSwitch from the vNIC of the VM. Lastly, we didn’t include any capture parameters for any specific traffic that we are looking for, in a production implementation it’s usually a good idea to narrow your focus.

Once you are satisfied you have captured enough traffic, stop the capture:

nsx01a.tonysangha.com# no debug packet capture session cde461a0-4a2f-4051-8999-ecf552dd28e8 
Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
Request:
        Capture host: host-36
        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
Session status: stopped

Next display the packet capture on the console for quick analysis. Pay attention to the text after parameters switch, I am using TCPDUMP syntax to narrow my focus to traffic to the host 172.16.32.1.

debug packet capture display session cde461a0-4a2f-4051-8999-ecf552dd28e8 parameters host 172.16.32.1
Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
Request:
        Capture host: host-36
        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
Session status: finished
Capture packets:
reading from file /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap, link-type EN10MB (Ethernet)
22:17:32.869545 ARP, Request who-has 172.16.32.1 tell 172.16.32.11, length 46
22:17:33.869539 ARP, Request who-has 172.16.32.1 tell 172.16.32.11, length 46
...

Exporting the capture is a good idea as you most likely will want to open the file in WireShark for further analysis. In this example, I have a CentOS VM (SCP Server) setup for transferring my captures onto. You will need to specify the session ID from your previous capture.

nsx01a.tonysangha.com# debug packet capture scp session cde461a0-4a2f-4051-8999-ecf552dd28e8 url vmware@192.168.2.52:file1.pcap
Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
Request:
        Capture host: host-36
        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
Session status: finished
Begin SCP:
...
vmware@192.168.2.52's password: 

Lastly, discard the capture session(s) to save disk space on the NSX Manager Appliance

nsx01a.tonysangha.com# no debug packet capture session cde461a0-4a2f-4051-8999-ecf552dd28e8 discard 
Session: cde461a0-4a2f-4051-8999-ecf552dd28e8
Request:
        Capture host: host-36
        Vnic: 5037962b-668d-bfb4-fc48-8d1063000fb6.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/cde461a0-4a2f-4051-8999-ecf552dd28e8.pcap
Session status: deleted

There you have it, a quick 101 on how to use the NSX Central CLI to perform packet captures without ever logging onto the ESXi host. Below are some other commands which may prove useful.

debug packet capture host HOST_ID vnic VNIC_ID dir (input|output) parameters [options] .
debug packet capture host HOST_ID vmknic VMKNIC_NAME dir (input|output) parameters [options] .
debug packet capture host HOST_ID vmnic VMNIC_NAME dir (input|output) parameters [options] .
debug packet capture host HOST_ID vdrport dir (input|output) parameters [options] .
show packet capture session CAPTURE_ID
no debug packet capture session CAPTURE_ID
no debug packet capture session CAPTURE_ID discard
debug packet capture display session CAPTURE_ID parameters [options] .
debug packet capture scp session CAPTURE_ID url URL
debug packet capture clear session CAPTURE_ID
show packet capture sessions
show packet capture help host HOST_ID
show interface
Advertisements

1 thought on “NSX-v Central CLI Packet Capture”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s