When we are asked by our Clients to setup their servers we usually go with virtualization. We usually choose Hyper-V for multiple reasons but this article is not about this but about a problem that we've recently fixed for our Client. In the last few days we've been receiving strange error in backups on our email mailboxes [Windows Error 1]: Incorrect function.
Problem Description
During Altaro Hyper-V Backups we've noticed strange error:
The operation encountered an error. (ALTERR_BASEBACKUPCONTROLLER_018)
There was a problem while transferring data to or from the backup location. (ALTERR_DTPROVIDER_003)
(ALTERR_DATAPIPE_005)
(ALTERR_FILTER_001)
Cannot read from file – I/O error. (ALTERR_BASE_FILEOP_039)
[Windows Error 1]: Incorrect function.
Since it's not very uncommon for things to hang a bit for different reasons we decided to take the shortest approach by doing reboot and server updates but Altaro was still giving an error. Since there was no backup being made we decided to go with non-destructive test of file system, to verify if that could be the issue, with chkdsk.
But that also shows no problems. After that next step was to try and move machine to different place. Unfortunately during machine move it also displays Incorrect function. Next step was to find which file was actually giving an error. Since this virtual hyper-v machine was made up of 3 files we've tried to convert all 3 from VHD to VHDX type. Unfortunately it worked for 2 out of 3 files.
The error for APFILE1.vhd was there. Incorrect function.
Solution
Since we couldn't move, convert machine, creating new machine just by attaching old files didn't solve anything and there was no backup made there must be some corruption in the file itself causing the problem. We've decided to simply replace VHD drive with another drive by copying data. Since the drive had several folders, files with different folders and had some shared directories we had to do couple of steps:
Stop Server (LanmanServer) service (disable for peace of mind so that this setting survives reboots) to block access of file shares by users.
Run robocopy to copy all files and folders with permissions properly.
Our command of choice:
robocopy E: F: /E /ZB /DCOPY:T /COPYALL /R:1 /W:1 /V /TEE /LOG:Robocopy.log
Where:
E: is source drive (can be drive:\path or \\server\share\path).
F: is destination drive (can be drive:\path or \\server\share\path).
switch /E copy subdirectories, including Empty ones.
switch /ZB use restartable mode; if access denied use Backup mode.
switch /DCOPY:T copies directory timestamps.
switch /COPYALL copies all file information (equivalent to /COPY:DATSOU).
Copies the Data, Attributes, Timestamps, Owner, Permissions and Auditing info.
switch /R:n number of retries on failed copies. Default is 1 million.We recommend only 1.
switch /W:n wait time between retries. Default is 30 seconds, we set it to only 1 second.
switch /V produce verbose output, showing skipped files.
switch /TEE output to console window, as well as the log file.
switch /LOG:file output status to LOG file (overwrite existing log).
It seems one of the fails has failed to copy. But that was expected.
Next step was to remove drive letter from the old drive, and assign that drive letter to new drive we just copied files to
Shutdown server and remove old drive
Start SERVER back up
When it's back up and running, Start Server (LanmanServer) service and verify shares functionality.
And that's it. After that action we were able to make proper backups.
Notes
For the sake of additional verification we tried to use Test-VHD PowerShell command.
The command should return “True” if Test-VHD PowerShell cmdlets does not find any issues with the virtual hard disk file. In our case it shows everything is fine, so it must be something different. But since backup works, move works our job is done and we can go back to other tasks.