{"id":3342,"date":"2013-10-21T23:39:07","date_gmt":"2013-10-22T04:39:07","guid":{"rendered":"http:\/\/icesquare.com\/wordpress\/?p=3342"},"modified":"2020-09-10T16:54:24","modified_gmt":"2020-09-10T21:54:24","slug":"freebsd-zfs-faulted-kernel-ada0ahcich4000-lost-device","status":"publish","type":"post","link":"https:\/\/icesquare.com\/wordpress\/freebsd-zfs-faulted-kernel-ada0ahcich4000-lost-device\/","title":{"rendered":"[FreeBSD] ZFS FAULTED \/ kernel: (ada0:ahcich4:0:0:0): lost device"},"content":{"rendered":"<p>I am so happy that I finally solved a problem today. This problem had existed in my server farms for few months already. In the past few months, I had absolutely no idea what was causing the issues. Here is my story:<\/p>\n<p>I built a file server using FreeBSD and ZFS. The hardware components are consumer grade hardware. Basically, it is a desktop computer with multiple hard drives. The hard drives are connected to the SATA ports on the motherboard. It is a very simple setup.<\/p>\n<p>After I set up a ZFS pool, I started to load the data and stress tested the system. I noticed that the ZFS pool turned into a fault state after 15 mins, e.g., <\/p>\n<pre>\r\n#sudo zpool status -v\r\n  pool: storage\r\n state: FAULTED\r\nstatus: One or more devices could not be opened.  There are insufficient\r\n        replicas for the pool to continue functioning.\r\naction: Attach the missing device and online it using 'zpool online'.\r\nconfig:\r\n\r\n        NAME        STATE     READ WRITE CKSUM\r\n        storage     ONLINE       0     0     0\r\n          raidz1-0  ONLINE       0     0     0\r\n            ada4    ONLINE       0     0     0\r\n            ada0    REMOVED      0     0     0 \r\n            ada2    ONLINE       0     0     0\r\n            ada5    ONLINE       0     0     0\r\n            ada3    ONLINE       0     0     0\r\n\r\nerrors: Permanent errors have been detected in the following files:\r\n\r\n<\/pre>\n<p>In short, a drive is missing. So I tried to run dmesg <\/p>\n<pre>\r\nOct 17 20:24:00 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 17 20:24:00 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 20:42:14 kernel: (ada0:ahcich3:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): lost device\r\nOct 19 23:19:18 kernel: (ada0:ahcich4:0:0:0): removing device entry\r\nOct 19 23:33:02 kernel: (ada0:ahcich4:0:0:0): Synchronize cache failed\r\nOct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 30 af 4e 40 7a 00 00 00 00 00\r\nOct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): CAM status: ATA Status Error\r\nOct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): ATA status: 61 (DRDY DF ERR), error: 04 (ABRT )\r\nOct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): RES: 61 04 00 00 00 40 00 00 00 00 00\r\nOct 20 22:53:40 kernel: (ada0:ahcich4:0:0:0): Retrying command\r\n<\/pre>\n<p>I also verified the content in \/dev\/, e.g.,<\/p>\n<pre>\r\n#ls \/dev\/ad0\r\n\r\ncrw-r-----  1 root  operator   0x6f Oct 18 22:10 \/dev\/ada0\r\n<\/pre>\n<p>Interestingly, the device is still available. This is very confusing because the system sees the hardware (with some error), but the application could not see the hardware. So I decided to perform the following tests:<\/p>\n<ul>\n<li>Reboot<\/li>\n<li>Shutdown and reboot<\/li>\n<li>Replacing the SATA cables<\/li>\n<li>Connecting the hard drive to a different SATA port<\/li>\n<li>Replacing the hard drive with a new one that is certified by the manufacturer<\/li>\n<li>Replacing the motherboard<\/li>\n<li>Replacing the hard drive power cable<\/li>\n<li>Replacing the power supply<\/li>\n<li>Updating the firmware of the motherboard<\/li>\n<\/ul>\n<p>Unfortunately, none of these fixed the problem. I also checked the S.M.A.R.T. status of the hard drives but I didn&#8217;t see any issues. One thing I noticed is that the error is not consistent. It happens randomly on different SATA port and hard drive. That doesn&#8217;t make any sense to me because I saw the same thing on 3 different motherboards. Finally, I decided to try one last thing: I connected the hard drives through an USB port. Guess what, my FreeBSD box stopped complaining.<\/p>\n<p>So I decided to a different approach. I used the same hardware setup (the hard drives were connected using SATA instead of USB) and loaded a Windows. The system worked perfectly fine and stable. I didn&#8217;t experience any issue at all. So I believed the problem had nothing to do with the hardware. It must be the software settings issue.<\/p>\n<p>Finally, I decided to try one last thing, something I hadn&#8217;t paid attention before: BIOS Settings<\/p>\n<h2>AHCI\/IDE<\/h2>\n<p>Typically there is a setting to control how the motherboard interacts with the hard drives: IDE or AHCI. I usually stick with the default settings. In my case, the default value of my motherboard is IDE. After I changed the settings to AHCI, I found that the problem is gone. Yes, it&#8217;s gone and my headache is gone too.<\/p>\n<p>That was easy!<\/p>\n<p>&#8211;Derrick<\/p>\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I am so happy that I finally solved a problem today. This problem had existed in my server farms for few months already. In the past few months, I had <a class=\"more-link\" href=\"https:\/\/icesquare.com\/wordpress\/freebsd-zfs-faulted-kernel-ada0ahcich4000-lost-device\/\">Continue Reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[94,146,7,144,6,109],"tags":[171],"class_list":["post-3342","post","type-post","status-publish","format-standard","hentry","category-computer-problems-computer-help-pc-support","category-everyday-computer-problem","category-freebsd","category-sever-problem","category-technology","category-zfs","tag-freebsd"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/p1sqzr-RU","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2562,"url":"https:\/\/icesquare.com\/wordpress\/freebsd-9-zfs-one-or-more-devices-has-experienced-an-error-resulting-in-data-corruption-applications-may-be-affected\/","url_meta":{"origin":3342,"position":0},"title":"[FreeBSD+ZFS]One or more devices has experienced an error resulting in data corruption.  Applications may be affected.","author":"Derrick","date":"January 17, 2012","format":false,"excerpt":"When you check the ZFS status, you may find the following error message: One or more devices has experienced an error resulting in data corruption. Applications may be affected.. There can be million reasons to cause this error message showing up. Of course, 99% of them are caused by hardware\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4648,"url":"https:\/\/icesquare.com\/wordpress\/zfs-on-linux-what-to-do-when-resilver-takes-very-long\/","url_meta":{"origin":3342,"position":1},"title":"[ZFS On Linux] What to Do When Resilver Takes Very Long","author":"Derrick","date":"January 13, 2020","format":false,"excerpt":"If you check your Zpool health status and you notice an error like the following: sudo zpool status pool: myzpool state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3242,"url":"https:\/\/icesquare.com\/wordpress\/freebsdhow-to-remove-zfs-meta-data\/","url_meta":{"origin":3342,"position":2},"title":"[FreeBSD\/Linux]How To Remove ZFS Meta Data","author":"Derrick","date":"May 14, 2013","format":false,"excerpt":"I have many hard drives circulating among my servers for testing purpose. For example, I took a hard drive from one server and put it on a different server. After doing this for many times, I've noticed that the ZFS has put many header information \/ meta data left on\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/icesquare.com\/wordpress\/wp-content\/uploads\/2012\/01\/400px-broom_icon.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1214,"url":"https:\/\/icesquare.com\/wordpress\/building-a-super-large-and-reliable-file-server-with-mixed-size-of-harddisks\/","url_meta":{"origin":3342,"position":3},"title":"Building a Super Large and Reliable File Server with Mixed Size of Harddisks","author":"Derrick","date":"January 17, 2011","format":false,"excerpt":"In this article, I am going to show you how to build a super large, ultra reliable, expandable file server with mixed size of hard drives. Here is my criteria: Large Storage Capacity. At this moment, the largest hard drive is 2TB. I am looking for at least 10TB. Great\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"\/wordpress\/wp-content\/uploads\/2011\/01\/zfs_logo-150x150.jpg","width":350,"height":200},"classes":[]},{"id":3914,"url":"https:\/\/icesquare.com\/wordpress\/zfs-linux-vs-freebsd\/","url_meta":{"origin":3342,"position":4},"title":"Running ZFS on Linux: Things you should know and be aware of","author":"Derrick","date":"February 29, 2016","format":false,"excerpt":"First Published: Feb 29, 2016 Last Revised: January 10, 2026 ZFS is a next-generation file system renowned for its reliability and advanced features. However, due to legal and licensing issues, it is not included by default in most Linux distributions. Fortunately, you can still install (ZFS on Linux) with just\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3458,"url":"https:\/\/icesquare.com\/wordpress\/how-to-stress-test-zfs-system\/","url_meta":{"origin":3342,"position":5},"title":"How to Stress Test ZFS System","author":"Derrick","date":"January 26, 2014","format":false,"excerpt":"If you have set up a ZFS system, you may want to stress test the system before putting it in a production environment. There are many different ways to stress test the system. The most common way is to fill the entire pool using dd. However, I think scrubbing the\u2026","rel":"","context":"In &quot;Computer Problems&quot;","block_context":{"text":"Computer Problems","link":"https:\/\/icesquare.com\/wordpress\/category\/technology\/computer-problems-computer-help-pc-support\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3342","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/comments?post=3342"}],"version-history":[{"count":22,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3342\/revisions"}],"predecessor-version":[{"id":4823,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/posts\/3342\/revisions\/4823"}],"wp:attachment":[{"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/media?parent=3342"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/categories?post=3342"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/icesquare.com\/wordpress\/wp-json\/wp\/v2\/tags?post=3342"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}