We have a setup at work where we have two Hyper-V servers in a failover cluster mounting a VHD on a NAS1 as an iSCSI target running a number of VMs that get backed up by Veeam.
Recently we had a hard drive failure in the NAS. The NAS had RAID 5 with a hot spare2 so no problem, in this case, the drive died on a Friday so I shut the server down on the weekend, popped a new drive in and rebuilt the array offline.
All looked good, the RAID controller said all drives were OK and the servers started back up fine, but the next night the Veeam backups failed with the error message.
2017-07-23 9:16:59 PM :: Processing HWFS1 Error: Incorrect function.
Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{8144a28c-459c-41a7-a274-b03ae6a3d493}\HWFS1_D.vhd].
Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{8144a28c-459c-41a7-a274-b03ae6a3d493}\HWFS1_D.vhd].
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Incorrect function.
Failed to read data from the file [\\?\GLOBALROOT\Device\CSV{8144a28c-459c-41a7-a274-b03ae6a3d493}\HWFS1_D.vhd].
After some digging, I found events in the event view on the NAS that said.
Logical drive 2 configured on array controller P212 located in server slot 3 returned a fatal error during a read/write request from/to the volume.
Logical block address 4123043672, block count 8 and command 32 were taken from the failed logical I/O request.
Array controller P212 located in server slot 3 is also reporting that the last physical drive to report a fatal error condition (associated with this logical request), is located on bus 0 and ID 11.
So I ran chkdsk /f e: on the server and it said
Windows has checked the file system and found no problems.
But I was still seeing the error messages in event viewer. Then I tried chkdsk /r e: which took several hours but eventually came back and reported that it had found and repaired (moved) several unreadable sectors.
So even when every thing seems ok, and chkdsk reports no errors if you get a Failed to read data from the file message from Veeam it could be bad sectors on the underlying disk.
