ZFS – errors: Permanent errors have been detected in the following files:

I got the following messages today when I inspected my ZFS:

errors: Permanent errors have been detected in the following files:

        /mypool/data/file1.dat
        /mypool/data/file2.dat
        /mypool/data/file3.dat
        /mypool/data/file4.dat
        /mypool/data/file5.dat

As usual, the first thing I did was to scrub the entire pool, i.e.,

sudo zpool scrub mypool

Unfortunately, it didn’t work. The error still existed even there was no checksum error. So I decided to delete the files manually, and it ended up like this:

errors: Permanent errors have been detected in the following files:

        mypool/data:<0x1fa3a>
        mypool/data:<0x1fa45>
        mypool/data:<0x1fa46>
        mypool/data:<0x1f354>
        mypool/data:<0x1f664>

That’s because when the files were deleted, it simply removed the file pointer. Since ZFS no longer has the file names, it decided to report the location.

To solve this problem, you will need to go through the following:

First, make sure that you have no checksum error and the pool is healthy, i.e., all hard drives are online, and all counts are zero.

Next, try to scrub the pool again:

sudo zpool scrub mypool

Within a minute, try to stop the process:

sudo zpool scrub -s mypool

Check the status again. The error should be gone:

sudo zpool status -v
  pool: mypool
 state: ONLINE
  scan: scrub canceled on Sun Feb  3 12:18:06 2019
errors: No known data errors

If the error still presents, you may need to scrub the pool again.

Our sponsors:

4 Replies to “ZFS – errors: Permanent errors have been detected in the following files:”

  1. Pingback: When does "zpool scrub" delete files automatically? - Onooks

  2. AAA 4A

    I have this:
    [[email protected] somedir-feb12020]# zpool status -v
    pool: datapool
    state: UNAVAIL
    status: One or more devices is currently being resilvered. The pool will
    continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
    scan: resilver in progress since Thu Jan 7 10:49:13 2021
    2.78T scanned at 238M/s, 2.77T issued at 237M/s, 2.78T total
    108G resilvered, 99.65% done, 0 days 00:00:42 to go
    config:

    NAME STATE READ WRITE CKSUM
    datapool UNAVAIL 0 0 0 insufficient replicas
    mirror-0 DEGRADED 538 8 0
    1921203353047932876 FAULTED 0 0 0 was /dev/sda1
    ata-HGST_HUS728T8TALE6L4_VAJ69RRL DEGRADED 599 8 176 too many errors
    mirror-1 ONLINE 0 0 0
    ata-HGST_HUS728T8TALE6L4_VAJ6HWVL ONLINE 0 0 0
    ata-HGST_HUS728T8TALE6L4_VAHV3B7L ONLINE 0 0 0 (resilvering)

    What i need to recover is the data is:
    errors: Permanent errors have been detected in the following files:
    /datapool/home/somedir/sharedvm/ssss-feb12020/ssss-disk1-000004.vmdk

    I have a backup but over a month old.

    Reply
    • Derrick Post author

      Typically if a device status is faulted or degraded, you can try to run: “zpool clear pool_name” first. If it doesn’t work, try to shut down the machine and power on the machine again (rebooting the server doesn’t work). This will force the ZFS to recognize the missing (status= removed) devices again. If it doesn’t work, try to check the cable connections (both power and SATA). Good luck!

      Reply

Leave a Reply to Don Daley Cancel reply

Your email address will not be published. Required fields are marked *