Troubleshooting Flash VM Boot

Product version: 10.9

Last modified: 24 April 2023

Problem

Flash VM boot or screenshot verification fails with an error.

Background

NAKIVO Backup & Replication uses iSCSI technology to connect VM disks stored in the backup to a target hypervisor: an ESXi host or a Hyper-V server.

VM disks are exposed by the Transporter that is assigned to the Backup Repository and mounted to the target hypervisor host.

In turn, the disks are mounted to the hypervisor host and presented to the VMware VM as RDM disks in virtual compatibility mode or to the Hyper-V VM as virtual disks.

The most common reasons that prevent the feature from working successfully are listed below:

iSCSI Bindings on ESXi Hosts

ESX/ESXi hosts allow binding the software iSCSI adapter (HBA) to specific VM Kernel network(s).

If such binding is configured for an ESXi host used for Flash VM Boot, connectivity between the host and the Transporter might not be available at the iSCSI layer.

If an iSCSI binding is configured for a host that you plan to use as a target for Flash VM Boot, verify the connectivity at the iSCSI layer.

iSCSI Port Busy on Transporter

NAKIVO Backup & Replication uses TCP port 3260 for VMware and port 10000 for Hyper-V to expose iSCSI targets to hosts.

In addition, NAKIVO Backup & Replication uses TCP port 9446 for Director-Transporter and Transporter-Transporter communication, as well as the port range 9448-10000 for cross-Transporter data transfer. These must be opened on the Transporter machine.

If a required TCP port is busy on the machine on which the NAKIVO Backup & Replication Transporter is installed, the feature will not work.

The default port number cannot be changed since VMware/Hyper-V is always using it.

Windows Firewall Blocking iSCSI

On some Windows Server systems, the firewall is configured to prevent outbound iSCSI connections.

If the firewall has this configuration, Flash VM Boot will not be functional.

iSCSI Initiator Service Not Running

NAKIVO Backup & Replication requires the Microsoft iSCSI Initiator service to be enabled and running on a target Hyper-V host.

If the service is not running, the Flash VM Boot will not be available with such a target host.

Troubleshooting VMware Flash VM Boot

Follow the steps below to troubleshoot NAKIVO Backup & Replication Flash VM Boot for VMware virtual machines.

Root Cause Analysis

iSCSI Bindings on ESXi Hosts

Use the vSphere Client to check if you have iSCSI bindings in your ESXi host:

Navigate to the ESXi host.
Go to the Configure tab and click Storage Adapters.
Locate iSCSI Software Adapter and select its entry.
Go to the Network Port Binding tab.
Check for VMkernel port bindings in the table below. If you have iSCSI bindings, you will see something similar to this:

iSCSI Port Busy on Transporter

In most cases, the iSCSI port is busy when NAKIVO Backup & Replication is running on a NAS and there are iSCSI targets already exposed from this NAS.

For example, to check if TCP port 3260 is busy, perform the following steps:

Log in to the system console of your NAS or server where NAKIVO Backup & Replication is installed.
Run the following console command: netstat -ant
Check if the command output contains TCP port 3260.

If the command output contains TCP port 3260, this port is already in use and cannot be used by NAKIVO Backup & Replication.

Repeat as needed for other required ports.

Windows Firewall Blocking iSCSI

If your ESXi host does not have iSCSI bindings, check if your Transporter is installed on Windows.

If yes, check your firewall settings.

Solution

iSCSI Bindings

Transporter VM is running on the ESXi host that is used for Flash VM Boot

Configure the Transporter to have access to the iSCSI targets:

Create a new VM network (or use an existing one that has a connection to the VMkernel network used for iSCSI traffic).
Add a new network adapter to the VM with the Transporter, which will be used for Flash VM Boot.
Configure a static (or dynamic) IP for the newly created adapter.
Create a new VMkernel network on the ESXi host (or use an existing VMkernel network on the given NIC).
Add binding to the software iSCSI to this VMkernel.

Transporter is not running on the ESXi host that is used for Flash VM Boot

If an ESXi host has a non-bridged NIC that has connectivity to the Transporter:

Create a new VMkernel network on the ESXi host (or use an existing VMkernel network on the given NIC).
Add binding to the software iSCSI to this VMkernel.

If an ESXi host has an unused NIC that has connectivity to the Transporter:

Create a new VMkernel network and a new vSwitch.
Add binding to the software iSCSI to this VMkernel.

If an ESXi host has bridged NICs that cannot participate in binding:

Create a new VMkernel network (or use an existing one) on the ESXi host.
Enable SSH access to the ESXi host.
Run the following command via SSH on the target host to create binding:

esxcli iscsi networkportal add -n vmkX -A vmhbaY -f 1

Where vmkX is the ID of the created VMkernel network, and vmhbaY is the ID of the software iSCSI HBA (usually it’s vmhba34).

iSCSI Port Busy on Transporter

If a required TCP port is busy on the machine on which the NAKIVO Backup & Replication Transporter is installed, do one of the following:

Remove existing third-party iSCSI targets exposed on this port; or
Reconfigure existing third-party iSCSI targets to use a different port.

Windows Firewall

If you are using NAKIVO Backup & Replication on Windows and a firewall is blocking iSCSI traffic, follow the steps below:

Open the Command Prompt as Administrator.
Run the following commands:

netsh advfirewall firewall add rule name="iSCSI" dir=in action=allow protocol=TCP localport=3260

netsh advfirewall firewall add rule name="iSCSI" dir=out action=allow protocol=TCP localport=3260

If the issue persists, run the above command to open port 9446 and the port range 9448-10000 by replacing localport=3260 with localport=9446 and localport=9448-10000, respectively.

Other Issues

If you do not have network adapter iSCSI bindings and do not have issues with Windows firewall, contact us and provide us with detailed information:

Create a support bundle.
Take screenshots of your iSCSI adapter configuration and networking.
Contact NAKIVO support and provide the collected information.

Troubleshooting Hyper-V Flash VM Boot

Follow the steps below to troubleshoot NAKIVO Backup & Replication Flash VM Boot for Hyper-V virtual machines.

Root Cause Analysis

Microsoft iSCSI Initiator Service Not Running

Follow the steps below to determine whether Microsoft iSCSI Initiator Service is running:

Log in to the Hyper-V server.
Start the Services snap-in:
- On Windows Server 2016, click Start. In the Start Search box, enter services.msc and press Enter.
- On Windows Server 2012 and Windows Server 2012 R2, go into the Desktop mode, hover the pointer over the lower-left corner of the screen, and right-click the Start pane. In the menu that opens, click Run, enter services.msc in the Run box, and press Enter.

In the list of services, locate the Microsoft iSCSI Initiator Service and check its status. The service must be running.

iSCSI Port Busy on Transporter

In most cases, the iSCSI port is busy when NAKIVO Backup & Replication is running on a NAS and there are iSCSI targets already exposed from this NAS.

For example, to check if TCP port 10000 is busy, perform the following steps:

Log in to the system console of your NAS or server where NAKIVO Backup & Replication is installed.
Run the following console command: netstat -ant
Check if the command output contains TCP port 10000.

If the command output contains TCP port 10000, this port is already in use and cannot be used by NAKIVO Backup & Replication.

Repeat as needed for other required ports.

Windows Firewall Blocking iSCSI

If the NAKIVO Backup & Replication Transporter is installed on Windows, check your firewall settings.

Solution

Microsoft iSCSI Initiator Service Not Running

Follow the steps below to run the Microsoft iSCSI Initiator Service:

In the Services snap-in, locate the Microsoft iSCSI Initiator Service and double-click it.
The Microsoft iSCSI Initiator Service Properties dialog box opens. Make sure Automatic or Manual is chosen for the Startup type, and click Start.

iSCSI Port Busy on Transporter

If a required TCP port is busy on the machine on which the NAKIVO Backup & Replication Transporter is installed, do one of the following:

Remove existing third-party iSCSI targets exposed on this port; or
Reconfigure existing third-party iSCSI targets to use a different port.

Windows Firewall Blocking iSCSI

If you are using NAKIVO Backup & Replication on Windows and firewall is blocking iSCSI traffic, follow the steps below to enable the ports necessary for iSCSI network traffic:

Open the Command Prompt as Administrator.
Run the following commands to allow the necessary ports:

netsh advfirewall firewall add rule name="iSCSI" dir=in action=allow protocol=TCP localport=10000

netsh advfirewall firewall add rule name="iSCSI" dir=out action=allow protocol=TCP localport=10000

If the issue persists, run the above command to open port 9446 by replacing localport=10000 with localport=9446.

Other Issues

If you do not have network adapter iSCSI bindings and do not have issues with Windows firewall, contact us and provide us with detailed information:

Create a support bundle.
Take screenshots of your iSCSI adapter configuration and networking.
Contact NAKIVO support and provide the collected information.