I got the following messages today when I inspected my ZFS:
errors: Permanent errors have been detected in the following files: /mypool/data/file1.dat /mypool/data/file2.dat /mypool/data/file3.dat /mypool/data/file4.dat /mypool/data/file5.dat
As usual, the first thing I did was to scrub the entire pool, i.e.,
sudo zpool scrub mypool
Unfortunately, it didn’t work. The error still existed even there was no checksum error.
Keep in mind that this tutorial is about removing the error message, it is not about rescuing your files. ZFS is not 100% fail proof. It is highly recommended to have a backup copy if the data is important.
Since I don’t care about the file (backup copy), I decided to delete the files manually, and it ended up like this:
errors: Permanent errors have been detected in the following files: mypool/data:<0x1fa3a> mypool/data:<0x1fa45> mypool/data:<0x1fa46> mypool/data:<0x1f354> mypool/data:<0x1f664>
That’s because when the files were deleted, it simply removed the file pointers. Since ZFS no longer has the file names, it decided to report the location instead. To solve this problem, you will need to go through the following:
First, make sure that you have no checksum error and the pool is healthy, i.e., all hard drives are online, and all counts are zero.
sudo zpool status mypool
Next, try to scrub the pool again:
sudo zpool scrub mypool
Within a minute, try to stop the process:
sudo zpool scrub -s mypool
Check the status again. The error should be gone:
sudo zpool status -v pool: mypool state: ONLINE scan: scrub canceled on Sun Feb 3 12:18:06 2019 errors: No known data errors
If the error still presents, you may need to scrub the pool again.
Pingback: When does "zpool scrub" delete files automatically? - Onooks
Thanks very much. Worked perfect
I have this:
[[email protected] somedir-feb12020]# zpool status -v
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Jan 7 10:49:13 2021
2.78T scanned at 238M/s, 2.77T issued at 237M/s, 2.78T total
108G resilvered, 99.65% done, 0 days 00:00:42 to go
NAME STATE READ WRITE CKSUM
datapool UNAVAIL 0 0 0 insufficient replicas
mirror-0 DEGRADED 538 8 0
1921203353047932876 FAULTED 0 0 0 was /dev/sda1
ata-HGST_HUS728T8TALE6L4_VAJ69RRL DEGRADED 599 8 176 too many errors
mirror-1 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4_VAJ6HWVL ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4_VAHV3B7L ONLINE 0 0 0 (resilvering)
What i need to recover is the data is:
errors: Permanent errors have been detected in the following files:
I have a backup but over a month old.
Typically if a device status is faulted or degraded, you can try to run: “zpool clear pool_name” first. If it doesn’t work, try to shut down the machine and power on the machine again (rebooting the server doesn’t work). This will force the ZFS to recognize the missing (status= removed) devices again. If it doesn’t work, try to check the cable connections (both power and SATA). Good luck!
Pingback: Resolved: When does "zpool scrub" delete files automatically? - Resolved Problem