I was recently tasked with creating a cleanroom restore process in Azure to recover Active Directory servers in the event of a catastrophic event, such as an entire AD domain being crypto-locked.  We needed to confirm that we could restore all the domain controllers, and that the domain controllers came up and were able to immediately replicate AD/DNS, etc. 

The production environment:

  • One AD server is already in Azure
  • One AD server is hosted on-premises in a VMware hypervisor
  • Backup is done on both servers with CommVault

We created a Resource Group and populated with the following:

  • Virtual Network separate from our production network with 2 subnets with no Public IP
    • Cleanroom Subnet
    • AzureBastionSubnet
  • Bastion
  • StorageAccount

Using CommVault both servers were restored to the vnet subnet for the Cleanroom.

The Azure server came up right away with no issues.  AD DS was healthy.

The VMware hosted server did not come up – it could not be booted and gave a BSOD of 0xc00002e2 which is Directory Services could not be started.  Uh oh – this is a problem.

Azure can be a little difficult when it comes to a BSOD on a server.  There is no ‘console’ per-say so there’s no way to reboot the server and start pounding on the F8 key to start in Directory Services Recovery Mode.  Even after an exhaustive Google search session, there wasn’t a lot to be found.  Lots of articles on how to enable DSRM on a failed domain controller if the option wasn’t available, but very little on how to actually get a VM to boot into DSRM.

The good news is it is doable, but it’s quite complex and involved.  Here are the steps I performed to achieve my goal:

Step 1 – create a recovery VM

This is done by utilizing Azure CLI commands.

Repair a Windows VM by using the Azure Virtual Machine repair commands – Virtual Machines | Microsoft Learn

I used the CLI option in the top right of Azure to open a CLI shell into Azure.  This is the simplest way to get a shell without having to install a whole bunch of Azure modules into PowerShell and navigate MFA authentication, etc.

az extension add -n vm-repair

This installs the extension so you can run the repair and create the repair environment

If you’ve already installed the extension at some point you may need to update it:

az extension update -n vm-repair

You will be prompted to enable a Public IP – if you aren’t using Bastion you might as well indicate Y – you will need it to access your recovery VM.

Now you run some simple commands 

az vm repair create -g MyResourceGroup -n myVM –repair-username username –repair-password ‘password!234’ –enable-nested –verbose

  • where MyResourceGroup is the resource group the failed VM is currently residing
  • where myVM is the name of the failed VM
  • where repair-username is the is admin user you want to use for your recovery VM 
  • where repair-password is the admin password you want to use for your recovery VM (I suggest you use a strong password)
  • where –enable-nested sets up your recovery VM (a 2016 server) with Hyper-V installed

Once these commands complete, you’ll have a new Resource Group called repair-myVM–timestamp which has your bootable 2016 recovery VM and a copy of your failed drives – the non-bootable AD controller system drives and whatever other disks were attached to the failed VM from the restore.

Boot up the myVM recovery VM if it isn’t already and use RDP (or Bastion if you have it) to connect to the server using the repair-username and repair-password you specified in the previous command.

  • Your Windows 2016 Server should have Hpyer-V installed to it.  If not, install it.
  • Open Computer Management -> Disk Management and make sure the drive(s) you need to use for Hyper-V VM are set to “Offline”
  • Open Hyper-V and create a VM with appropriate vCPU cores and memory (acknowledging the limits of your Server 2016 recovery instance)
    • Do not create a drive – choose to add a drive later
  • Edit your new Hyper-V VM
    • Click on IDE Controller 1
    • Click on Add Hard Drive
    • Select the physical disk available
    • Click on Apply then OK
  • Right click on the VM and click on Connect
  • Start the VM
  • When it starts booting up start tapping F8 to get into the boot menu
  • Select Directory Services Recovery Mode
  • Log in with your DSRM (Administrator) password

You can now perform the steps to correct whatever issues you’re having with AD.

When completed and you can boot the VM up and log in with your domain credentials, shut down the VM, then shut down your Recovery VM.

Go back to your Azure CLI 

az vm repair restore -g MyResourceGroup -n MyVM –verbose

Obviously, you will again substitute your values for MyResourceGroup and MyVM (same as above).

This final command will completely remove the recovery environment including the repair-ResourceGroup that was created from your Azure tenant.  

You should now be able to boot your recovered AD Domain Controller in Azure.

If your issue was not with Active Directory or you’re having issues recovering Active Directory due to disk issues:

Troubleshoot Windows stop error – directory service initialization failure – Virtual Machines | Microsoft Learn

I found this to be an extremely helpful article on fixing disks with NTDS installed. 

  • You can use the 2016 Server to run DISKPART on the attached drive to set the active partition (Gen 1 VMs)
  • You can use REG QUERY to find the location of the NTDS (typically C:\Windows\NTDS but it can be moved such as I found in my case which complicated things considerably)
  • You can run BCDEDIT commands to force the VM to boot to DSRM (safeboot dsrepair) so you can re-attach the disk to your VM in Azure and not worry about not being able to hit F8 on a console
    • Note – you will have to also reverse this or the BCD will always boot to DS Recovery Mode
    • Note – you still have to have your DSRM password to log in

Other Helpful Resources

I used all of these online resources to solve my problem.

Blue screen errors when booting an Azure VM – Virtual Machines | Microsoft Learn

Troubleshoot a Windows VM in the Azure portal – Virtual Machines | Microsoft Learn

How to reset the Directory Services Restore Mode administrator account password – Windows Server | Microsoft Learn

How to fix Error 0xc00002e2 after rebooting Windows Domain Controller – Hostway Help Center

Azure Serial Console for Windows – Virtual Machines | Microsoft Learn

Fixing non-bootable Azure Virtual Machine (cloud-aware.net)

How to boot Windows Server in Directory Services Restore Mode (DSRM) (kapilarya.com)

Recovering the Active Directory database in Windows Server 2012 R2 | Dell Canada

How to Deploy Hyper-V Nested Virtualization on Azure: Full Overview (nakivo.com)

Attach a managed data disk to a Windows VM – Azure – Azure Virtual Machines | Microsoft Learn