{"id":4830,"date":"2020-11-13T21:15:30","date_gmt":"2020-11-14T03:15:30","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=4830"},"modified":"2020-11-13T21:17:54","modified_gmt":"2020-11-14T03:17:54","slug":"zfshow-to-repair-a-zfs-pool-if-one-device-was-damaged","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/zfshow-to-repair-a-zfs-pool-if-one-device-was-damaged\/","title":{"rendered":"[ZFS]How to repair a ZFS pool if one device was damaged"},"content":{"rendered":"<p>Today, I accidentally dd&#8217;ed a disk which was part of an active ZFS pool on my test server. I dd&#8217;ed the first and the last 10 sectors of the disk. Technically I didn&#8217;t lose any data because my ZFS configuration was RAIDZ. However once I rebooted my computer, my ZFS complained:<\/p>\n<pre>\r\n#This is what I did:\r\nsudo dd if=\/dev\/zero of=\/dev\/sda bs=512 count=10\r\nsudo dd if=\/dev\/zero of=\/dev\/sda bs=512 seek=$(( $(blockdev --getsz \/dev\/sda) - 4096 )) count=1M\r\n<\/pre>\n<pre>\r\nsudo zpool status\r\n  pool: storage\r\n state: DEGRADED\r\nstatus: One or more devices could not be used because the label is missing or\r\n        invalid.  Sufficient replicas exist for the pool to continue\r\n        functioning in a degraded state.\r\naction: Replace the device using 'zpool replace'.\r\n   see: http:\/\/zfsonlinux.org\/msg\/ZFS-8000-4J\r\n  scan: resilvered 2.40T in 1 days 00:16:34 with 0 errors on Fri Nov 13 20:05:53 2020\r\nconfig:\r\n\r\n        NAME                                 STATE     READ WRITE CKSUM\r\n        storage                              DEGRADED     0     0     0\r\n          raidz1-0                           DEGRADED     0     0     0\r\n            ata-ST4000DM000-1F2168_S30076XX  ONLINE       0     0     0\r\n            ata-ST4000DX001-1CE168_Z3019CXX  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0S9YY  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXZZ  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXDD  ONLINE       0     0     0\r\n            412403026512446213               UNAVAIL      0     0     0  was \/dev\/disk\/by-id\/ata-ST4000NM0033-9ZM170                                                                                                                         _Z1Z3RR74-part1\r\n<\/pre>\n<p>So I checked the problematic device, and I see the problem:<\/p>\n<pre>\r\nls \/dev\/disk\/by-id\/\r\n\r\n#This is normal disk:\r\nlrwxrwxrwx 1 root root  10 Nov 13 20:58 ata-ST4000DX001-1CE168_Z3019CXX-part1 -> ..\/..\/sdd1\r\nlrwxrwxrwx 1 root root  10 Nov 13 20:58 ata-ST4000DX001-1CE168_Z3019CXX-part9 -> ..\/..\/sdd9\r\n\r\n\r\n#This is the problematic disk, part1 and part9 are missing.\r\nlrwxrwxrwx 1 root root   9 Nov 13 20:58 ata-ST4000NM0033-9ZM170_Z1Z3RR74 -> ..\/..\/sdf\r\n<\/pre>\n<p>It is pretty easy to fix this problem. All you need is to bring the device offline and bring it back.<\/p>\n<pre>\r\n#First, offline the problematic device:\r\nsudo zpool offline storage 412403026512446213\r\n<\/pre>\n<pre>\r\nsudo zpool status\r\n  pool: storage\r\n state: DEGRADED\r\nstatus: One or more devices could not be used because the label is missing or\r\n        invalid.  Sufficient replicas exist for the pool to continue\r\n        functioning in a degraded state.\r\naction: Replace the device using 'zpool replace'.\r\n   see: http:\/\/zfsonlinux.org\/msg\/ZFS-8000-4J\r\n  scan: resilvered 2.40T in 1 days 00:16:34 with 0 errors on Fri Nov 13 20:05:53 2020\r\nconfig:\r\n\r\n        NAME                                 STATE     READ WRITE CKSUM\r\n        storage                              DEGRADED     0     0     0\r\n          raidz1-0                           DEGRADED     0     0     0\r\n            ata-ST4000DM000-1F2168_S30076XX  ONLINE       0     0     0\r\n            ata-ST4000DX001-1CE168_Z3019CXX  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0S9YY  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXZZ  ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXDD  ONLINE       0     0     0\r\n            412403026512446213               OFFLINE      0     0     0\r\n<\/pre>\n<pre>\r\n#Then bring back the device:\r\nsudo zpool online ata-ST4000NM0033-9ZM170_Z1Z3RR74 \r\n\r\n#Resilver it\r\nsudo zpool scrub storage\r\n\r\nsudo zpool status\r\n  pool: storage\r\n state: ONLINE\r\n  scan: resilvered 36K in 0 days 00:00:01 with 0 errors on Fri Nov 13 21:03:01 2020\r\nconfig:\r\n\r\n        NAME                                  STATE     READ WRITE CKSUM\r\n        storage                               ONLINE       0     0     0\r\n          raidz1-0                            ONLINE       0     0     0\r\n            ata-ST4000DM000-1F2168_S30076XX   ONLINE       0     0     0\r\n            ata-ST4000DX001-1CE168_Z3019CXX   ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0S9YY   ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXZZ   ONLINE       0     0     0\r\n            ata-ST4000DM000-2AE166_WDH0SXDD   ONLINE       0     0     0\r\n            ata-ST4000NM0033-9ZM170_Z1Z3RR74  ONLINE       0     0     0\r\n\r\nerrors: No known data errors\r\n\r\n<\/pre>\n<p>That&#8217;s it.<\/p>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today, I accidentally dd&#8217;ed a disk which was part of an active ZFS pool on my test server. I dd&#8217;ed the first and the last 10 sectors of the disk. <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/zfshow-to-repair-a-zfs-pool-if-one-device-was-damaged\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,144,6,69,109],"tags":[177],"class_list":["post-4830","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-sever-problem","category-technology","category-troubleshooting-technology","category-zfs","tag-zfs"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-1fU","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":3242,"url":"https:\/\/icesquare.com\/wordpress\/freebsdhow-to-remove-zfs-meta-data\/","url_meta":{"origin":4830,"position":0},"title":"[FreeBSD\/Linux]How To Remove ZFS Meta Data","author":"Derrick","date":"May 14, 2013","format":false,"excerpt":"I have many hard drives circulating among my servers for testing purpose. For example, I took a hard drive from one server and put it on a different server. After doing this for many times, I've noticed that the ZFS has put many header information \/ meta data left on\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2012\/01\/400px-broom_icon.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4286,"url":"https:\/\/icesquare.com\/wordpress\/how-to-install-zfs-on-rhel-centos-7\/","url_meta":{"origin":4830,"position":1},"title":"How to install ZFS on RHEL \/ CentOS 7","author":"Derrick","date":"January 4, 2017","format":false,"excerpt":"A friend of mine likes to try ZFS on CentOS 7, therefore I decide to make a guide for him. The following instructions have been well tested on CentOS 7. Before you decide to put ZFS in a production use, you should be aware of the following: ZFS is originally\u2026","rel":"","context":"In &quot;Everyday Computer Problem&quot;","block_context":{"text":"Everyday Computer Problem","link":"https:\/\/icesquare.com\/wordpress\/category\/everyday-computer-problem\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":4648,"url":"https:\/\/icesquare.com\/wordpress\/zfs-on-linux-what-to-do-when-resilver-takes-very-long\/","url_meta":{"origin":4830,"position":2},"title":"[ZFS On Linux] What to Do When Resilver Takes Very Long","author":"Derrick","date":"January 13, 2020","format":false,"excerpt":"If you check your Zpool health status and you notice an error like the following: sudo zpool status pool: myzpool state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":4830,"position":3},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3458,"url":"https:\/\/icesquare.com\/wordpress\/how-to-stress-test-zfs-system\/","url_meta":{"origin":4830,"position":4},"title":"How to Stress Test ZFS System","author":"Derrick","date":"January 26, 2014","format":false,"excerpt":"If you have set up a ZFS system, you may want to stress test the system before putting it in a production environment. There are many different ways to stress test the system. The most common way is to fill the entire pool using dd. However, I think scrubbing the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1380,"url":"https:\/\/icesquare.com\/wordpress\/how-to-improve-zfs-performance\/","url_meta":{"origin":4830,"position":5},"title":"How to Improve ZFS performance","author":"Derrick","date":"August 24, 2014","format":false,"excerpt":"First Edition: March 14, 2010 Last Edited: January 10, 2026 \u00a0 In this tutorial, I will show you how to improve the performance of your ZFS using the affordable consumer-grade hardware (e.g., Gigabit network card, standard SATA non-SSD hard drives, consumer-grade motherboard etc.). Many people found a problem on their\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4830","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=4830"}],"version-history":[{"count":6,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4830\/revisions"}],"predecessor-version":[{"id":4836,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/4830\/revisions\/4836"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=4830"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=4830"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=4830"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}