File failing during backup job

[justanotheradmin]

Hello,

During one of our backup jobs, a recently added file is being skipped.

I have verified permissions on the drive and run consistency checks on the disk.

The error returned is...

Reading (1117) The request could not be performed due to an IO device error.

It should be noted that the file is fairly large, just shy of 90GB.

Any ideas on where to go from here?

Thank you

[Stuart]

Hi there,

Thanks for posting on our forum.

Typically an I/O error is related directly to a hardware issue so the checks you've performed were a good start. Did you run these checks on the source disks or the backup destination?
The error seems to be occurring when the backup process is reading the file - so it's likely that it's occurring on the source more so than the destination.

Have you checked Event Viewer (in particular the System events) for any NTFS, Disk or volsnap errors/warnings?
There should be something logged by the operating system that provides extra detail as to what's occurred.

[justanotheradmin]

Hey Stuart,

Just wanted to update...

It does appear to be a hardware issue. Who would have thought it was what it said on the tin?

This server has 4 drives in it & I tracked it down to which one was causing the fault via eventviewer.

It was in fact the source drive, not the destination drive... this makes sense as our onsite and offsite backup jobs both fussed about the same file.

I am currently stuck in a chkdsk /r test which has been running for a few hours now, but making progress. It did find bad sectors on the large file I'd mentioned.

Thank you for your help - I'll update when I have things resolved & am taking backups again without warning.

[Stuart]

Hi again,

Glad to hear that you've made some progress! Hopefully chkdsk can sort out the bad sectors and sort everything out.

[justanotheradmin]

I wanted to follow up so there would be some documentation for this issue should another user run into this.

Per Stuart's advice, I began to look into the storage device more closely. Manufacturer's tools were passing the drive on both Short, SMART, and Extended tests.

However, reading into the Event Viewer, I was seeing MANY errors:

Error - volsnap - Event ID 25
Error - atapi - Event ID 11
Error - Ntfs - Event ID 57

These are all indicators that something is not working properly on the storage device, despite what the Manufacturers device tools were indicating.

It took me a bit to work with this device as it is used in our Stage environment & I ended up getting the ok to come in later one day to perform after hours work.

I performed a chkdsk /f and was informed the file system had made corrections. Upon booting the system, I once again was greeted with the above errors.

I opted to perform a "surface scan" with the chkdsk /r option (thus implying /f also) which scans the surface of the device, attempts to recover data from bad read areas & moves them to a "spare" area on the drive. This was performed successfully & took about 4 hours (1TB). Upon completion, the chkdsk log in Event Viewer showed there were corrections to the file system & that data was recovered.

The results of the event viewer are:

Code: Select all

A disk check has been scheduled.
Windows will now check the disk.                         

CHKDSK is verifying files (stage 1 of 5)...
  2394880 file records processed.                                          File verification completed.
  317 large file records processed.                                      0 bad file records processed.                                        0 EA records processed.                                              0 reparse records processed.                                       CHKDSK is verifying indexes (stage 2 of 5)...
  2539492 index entries processed.                                         Index verification completed.
  0 unindexed files scanned.                                           0 unindexed files recovered.                                       CHKDSK is verifying security descriptors (stage 3 of 5)...
  2394880 file SDs/SIDs processed.                                         Cleaning up 4 unused index entries from index $SII of file 0x9.
Cleaning up 4 unused index entries from index $SDH of file 0x9.
Cleaning up 4 unused security descriptors.
Security descriptor verification completed.
  72307 data files processed.                                            CHKDSK is verifying Usn Journal...
  1064632 USN bytes processed.                                             Usn Journal verification completed.
CHKDSK is verifying file data (stage 4 of 5)...
Read failure with status 0xc0000185 at offset 0x4bf91e1000 for 0x10000 bytes.
Read failure with status 0xc0000185 at offset 0x4bf91ed000 for 0x1000 bytes.
Windows replaced bad clusters in file 183727
of name \NOPE~1\NOPE~1\LARGEF_~1.VHD.
  2394864 files processed.                                                 File data verification completed.
CHKDSK is verifying free space (stage 5 of 5)...
  151798965 free clusters processed.                                         Free space verification is complete.
Adding 1 bad clusters to the Bad Clusters File.
CHKDSK discovered free space marked as allocated in the
master file table (MFT) bitmap.
Correcting errors in the Volume Bitmap.
Windows has made corrections to the file system.

I then reran our "on hold" BackupAssist jobs with no errors.

Thank you again for your time, Stuart. I should have verified the disk was having trouble from the get go and looked at the Event Viewer... but this is a rare case where the tools were showing an incorrect result. Jumped ahead of myself.