Proxmox Backup Failure

Product version: 11.0.1

Last modified: 14 February 2025

Problem

During a full or incremental VM backup on Proxmox, the default Temporary file location (/tmp) runs out of space. This prevents the VM from writing new data to the overlay file, causing it to become stuck in a "Blocked" state. Additionally, this can make the host unavailable.

Note

During the incremental backup, the changed data size is smaller. As a result, the overlay file contains less data.

If the backup process is interrupted, the Director may be unable to clean up the overlay file properly due to potential failures in the Proxmox API.

After stopping the backup job and restarting the VM, only the data from before the backup remains, leading to the loss of any changes made during the backup.

Background

During Proxmox VM backups NAKIVO Backup & Replication creates temporary overlay files for raw disk images, to guarantee the backup consistency. The backup job may fail if there is insufficient space for the temporary file, leading to VM instability and potential data loss.

The issue typically results in unresponsive virtual machines (VMs) and backups failing to complete successfully.

The temporary overlay files may not be committed further increasing the risk of data loss. In this case, data validation and recovery are required to restore missing data.

Important

If a backup fails due to space issues, there is no automated recovery for lost data.

Solution

  1. Change the Temporary file location to a different directory (not on the root volume) if backing up large amounts of data or if the data change rate is high.

    Specify a new location with sufficient free space for temporary overlay files: go to Settings > Inventory, select the Proxmox VE Item, and edit Options > Temporary file location.

  2. To avoid similar issues in the future, periodically clean up unnecessary files from the temporary folder.

You may need to commit the overlay file manually to get merge the new data during backup:

  1. Stop the backup job that failed if it is still running.

  2. Ensure the Director service is not running (nkv-dirsvc).

  3. Open the Director log file C:\Program Files\NAKIVO Backup & Replication\logs\0\all_logs_except_inventory.log (/opt/nakivo/director/logs/all_logs_except_inventory.log) to locate the overlay file of the VM. Search for PROXMOX_OVERLAY_FILE. Example of log entry:

2025-02-14T06:36:09,669 [INFO ][Main] [SYSTEM] [JOB-148/Backup job for Proxmox VE][PROXMOX_VM-160/Centos-VM]

New volatile object was memorized: PROXMOX_OVERLAY_FILE/drive-scsi3~/tmp/8096966f-f23f-450a-a6a1-f505c26222b3.qcow2,

child of: PROXMOX_VM/171~ab72e1ab-9b33-4881-b46c-68d8a6777730, linked parent VO: N/A.

  1. Unlock the VM with the following command:

qm unlock <VM ID>

For example:

qm unlock 171

  1. Ensure the VM is Powered off.

  2. Run the following command to commit the overlay file to the original disk. Make sure the output log is Image committed.

qemu-img commit <path to overlay file>

Output log: Image committed.

For example:

qemu-img commit /tmp/8096966f-f23f-450a-a6a1-f505c26222b3.qcow2

Output log: Image committed.

Note

The disk type or storage type does not need to be considered since the overlay file contains all the information from the parent disk.

  1. Power on the VM to restore functionality.

  2. Verify the data on the original disk. If all data is correct, the overlay file can be removed.