ZFS – errors: Permanent errors have been detected in the following files: – ICESQUARE – Solve Computer Server Problems, Computer Help, Server Support, Server Help

I got the following messages today when I inspected my ZFS:

errors: Permanent errors have been detected in the following files:

        /mypool/data/file1.dat
        /mypool/data/file2.dat
        /mypool/data/file3.dat
        /mypool/data/file4.dat
        /mypool/data/file5.dat

As usual, the first thing I did was to scrub the entire pool, i.e.,

sudo zpool scrub mypool

Unfortunately, it didn’t work. The error still existed even there was no checksum error.

Keep in mind that this tutorial is about removing the error message, it is not about rescuing your files. ZFS is not 100% fail proof. It is highly recommended to have a backup copy if the data is important.

Since I don’t care about the file (backup copy), I decided to delete the files manually, and it ended up like this:

errors: Permanent errors have been detected in the following files:

        mypool/data:<0x1fa3a>
        mypool/data:<0x1fa45>
        mypool/data:<0x1fa46>
        mypool/data:<0x1f354>
        mypool/data:<0x1f664>

That’s because when the files were deleted, it simply removed the file pointers. Since ZFS no longer has the file names, it decided to report the location instead. To solve this problem, you will need to go through the following:

First, make sure that you have no checksum error and the pool is healthy, i.e., all hard drives are online, and all counts are zero.

sudo zpool status mypool

Next, try to scrub the pool again:

sudo zpool scrub mypool

Within a minute, try to stop the process:

sudo zpool scrub -s mypool

Check the status again. The error should be gone:

sudo zpool status -v
  pool: mypool
 state: ONLINE
  scan: scrub canceled on Sun Feb  3 12:18:06 2019
errors: No known data errors

If the error still presents, you may need to scrub the pool again.

Our sponsors:

7 Replies to “ZFS – errors: Permanent errors have been detected in the following files:”

Pingback: When does "zpool scrub" delete files automatically? - Onooks

Don Daley October 5, 2020 at 1:24 am

Thanks very much. Worked perfect

Reply ↓

AAA 4A January 7, 2021 at 12:34 am

I have this:
[root@localhost somedir-feb12020]# zpool status -v
pool: datapool
state: UNAVAIL
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Jan 7 10:49:13 2021
2.78T scanned at 238M/s, 2.77T issued at 237M/s, 2.78T total
108G resilvered, 99.65% done, 0 days 00:00:42 to go
config:

NAME STATE READ WRITE CKSUM
datapool UNAVAIL 0 0 0 insufficient replicas
mirror-0 DEGRADED 538 8 0
1921203353047932876 FAULTED 0 0 0 was /dev/sda1
ata-HGST_HUS728T8TALE6L4_VAJ69RRL DEGRADED 599 8 176 too many errors
mirror-1 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4_VAJ6HWVL ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4_VAHV3B7L ONLINE 0 0 0 (resilvering)

What i need to recover is the data is:
errors: Permanent errors have been detected in the following files:
/datapool/home/somedir/sharedvm/ssss-feb12020/ssss-disk1-000004.vmdk

I have a backup but over a month old.

Reply ↓

Derrick Post authorJanuary 8, 2021 at 1:28 pm

Typically if a device status is faulted or degraded, you can try to run: “zpool clear pool_name” first. If it doesn’t work, try to shut down the machine and power on the machine again (rebooting the server doesn’t work). This will force the ZFS to recognize the missing (status= removed) devices again. If it doesn’t work, try to check the cable connections (both power and SATA). Good luck!

Reply ↓