Typically I use the EmailAddress (Mail) attribute to cross-link the accounts.

Example:

Azure AD user:  Smiles J. McDuff

Azure AD email:  smiley@mydomain.com, sjmcduff@mydomain.onmicrosoft.com

In AD I then create the user –

AD User:   Smiles McDuff

SamAccountName:  sjmcduff

Set Email Address:  smiley@mydomain.com

When I do an Azure AD Connect Sync the new AD user should match to the Azure AD user and overwrite the user and attributes to that based in Active Directory.

Sometimes that doesn’t happen, such as making a typo with the email address, and instead of cross-linking the accounts between AD and AAD, a new user account is created in AAD.  Now things get tricky, because no matter how many times you delete the incorrect account in Azure AD, the next sync will just recreate it.

The solution is to capture the ObjectGUID attribute for the user in Active Directory and set that as the ImmutableID for the user in Azure.

Command:

Get-ADUser sjmcduff | select-object userPrincipalName, objectGuid

Result:

UserPrincipleName  :  sjmcduff@mywindowsdomain.com

objectGuid : b316d357-25fd-4912-9896-faf007a16289

Now convert that Guid to something we can use as an ImmutableID –

[Convert]::ToBase64String([guid]::New(“b316d357-25fd-4912-9896-faf007a16289”).ToByteArray())

Result:  

V9MWs/0lEkmYlvrwB6FiiQ==

This is our new ImmutableID value for the Azure AD user account.

connect-msolservice

Get-MsolUser -UserPrincipalName “sjmcduff@mydomain.onmicrosoft.com” | select-object userPrincipalName, ImmutableId

Result:

UserPrincipalName : sjmcduff@mydomain.onmicrosoft.com
ImmutableId :                              [Confirm ImmutableID is blank -if not record it in your notes]

Command:

Set-MsolUser -UserPrincipalName “sjmcduff@mydomain.onmicrosoft.com” -ImmutableId “V9MWs/0lEkmYlvrwB6FiiQ==”

Comand:

Get-MsolUser -UserPrincipalName “sjmcduff@mydomain.onmicrosoft.com” | fl userPrincipalName,ImmutableId

Confirm ImmutableID matches

Now when you sync, the accounts should pair up properly.

The post Azure AD Connect – Fixing the Sync appeared first on Wiredwolf Canada.

This post is more specific to creating a clean room, restoring your controllers to the clean room to run tests on them, particularly to make sure they’re replicating with each other, and restore them to the original Resource Group.  In this post we are using Commvault as the backup medium and restoring to Azure using a Commvault Media Agent.

Step 1: Create a Clean Room in Azure

The clean room in Azure is a Resource Group into which Commvault will restore backups of the Active Directory Domain Controllers. This Resource Group should be created in the same region as the production environment.

Resource Group:                RG-CACENTRAL-AD-Cleanroom
Within the Resource Group create:
Virtual Network:               VNET-CACENTRAL-AD-Cleanroom
Scope:                                  10.200.0.0/16
Subnet 1:                             SNET-CACENTRAL-AD-Cleanroom
                                              10.200.0.0/24
Subnet 2:                            AzureBastionSubnet
                                             10.200.254.0/26
Storage Account:              cleanroomstorage1
Public IP:                           PIP-Bastion-Cleanroom

NOTES

• Subnet 2 is automatically created when Bastion is deployed, and the name of the subnet must be AzureBastionSubnet
• The storage account is a requirement of Commvault as a place holder for the recovery VHDs
• Commvault also requires the name of the virtual network/subnet
• Throughout this exercise the vnet was never given its own security group or public IP address. This is a sandbox area with no internet access to ensure there can be no tampering with the domain controllers while they’re being recovered.
Bastion Deployment

Step 2 – Add Bastion

• In the resource group click on Create
• Enter “Bastion” in the search field
• Select Microsoft Bastion
• Click on Create to add the Bastion Plan to the Resource Group

o Subscription: pre-selected
o Resource Group: pre-selected
o Name: RG-CACENTRAL-CLEANROOM-BASTION
o Region: pre-selected (but make sure it matches)
o Tier: Basic
o Virtual Network: select VNET-CACENTRAL-AD-Cleanroom
o Subnet: select AzureBastionSubnet
o Public IP select ‘Create new’
o Public IP Name PIP-Bastion-Cleanroom

It can take a few minutes for Bastion to deploy.

The Clean Room has now been set up. Please note that the only Public IP to this environment is with Bastion and Bastion cannot be accessed outside of the Azure Portal.

Step 3 – Restoring the VMs to the Clean Room

Using Commvault the restoration process will deposit/create replica VMs to the resource group.

• Use the option to not start the VM when the restore is completed so you can check each one individually for corruption.
• Try to restore from the same time frame for each domain controller
IE – restore for Nov 11 2022 8 PM for one, then every other one should be the same or close
• Recommend you start with your PDCE role holder and restore each subsequent domain controller to a time or date after the PDCE

Step 4 – Verifying Azure Cleanroom VMs

The restored VMs can be booted up but may require additional modifications to function in the different vnet. This can be set on the vnet object or each network interface object.

• Update the Network Interface for each VM
o IP configurations – set to Static and note the IP address
o DNS Servers – add the static IP assigned to each VM as DNS servers
o Update the root IP A address of the domain, test pinging to ensure it resolves to the correct address
Use Bastion to connect to each domain controller. Note that your username should not be your pre-2000 (DOMAIN\username) but rather your modern username (username@internaldomain.loc)

• Suggest you open Active Directory Sites and Services and add the SNET-CACENTRAL-AD-Cleanroom subnet and link to the site name of the original domain controller

Testing AD

In this environment I was able to connect to each domain controller and immediately execute commands to verify AD replication. 

• Start Elevated CMD
• Execute: repadmin /syncall
• Execute: repadmin /showrepl
• Execute: repadmin /replsummary

I also created a bogus DNS entry under our domain and confirmed that it did replicate between controllers. Once confirmed be sure to remove this entry.

There was an issue with NETLOGON/SYSVOL replication with DFS Replication. In DNS the root A host record of the local domain is the original IP address of the PDCE and doesn’t automatically update. We resolved this issue in our test environment by adjusting this IP from the original to the new IP of the PDCE. Immediately all servers showed the NETLOGON and SYSVOL folders, and policies were being replicated between the domain controllers.

After moving back to the production environment make sure this DNS record is updated to its original value.

Step 5 – Migration back to Production Environment

The process to migrate the VMs to the Production Environment is simplified by only migrating the managed disks. To move the virtual disks the easiest way is to snapshot then create a replica from the snapshot in the target Resource Group.

• Shut down the VMs in the Clean Room
• Click on the disk to replicate in the Clean Room
• Click on Create Snapshot
• Give the snapshot a name but leave it in the same Resource group
• Set the networking to ‘Disable public and private access’
• Review + create
• Create

Now click on the snapshot in the Resource group

• Click on Create Disk
• Give the disk a name that identifies it as the replacement OS disk for the DC being restored
• Select the Resource group where the DC is being restored
• Select an appropriate size for the disk
• Set the networking to ‘Disable public and private access’
• Review + create
• Create

Go back to your production Resource group and ensure the servers you are about to restore are shut down

• Start with your PDCE Virtual machine
• Click on Disks
• In the OS Disk section, click on Swap OS Disk

Pay attention to the disk name, size, and which resource group it is in, to ensure correct selection.

Note:
You must confirm the OS Disk Swap by typing in the name of the VM

Click on OK at the bottom of the screen and wait for deployment to complete.

You will need to repeat this process for each VM being restored.

Final Steps

Once you have restored all the domain controllers in this fashion, start them up one at a time starting with your PDCE, and execute your AD tests again.

Because you are only swapping disks and not recreating the VMs you do not need to worry about configuration changes to the VM itself. All that information is retained. The analogy here would be that you have restored to a new drive to an earlier point in time, removed the corrupted or locked drive, and replaced with the new drive.

Tech Notes:

• I found it can sometimes take Azure a little while for the disks to appear in the dropdowns (sometimes a few minutes)
• I strongly recommend that you test this process biannually up to the point of restoring the disks to the production VMs.
• Bastion also has its own costs so if you aren’t going to be using it regularly then you can add when needed and remove when your recovery process has been concluded.  Keeping the Clean Room Resource Group doesn’t really cost anything but it’s so easy to recreate you might as well remove it as well.
• After recovery you should ensure your backups are continuing as expected – check your backup logs and run new backups manually as a test

The post Azure Repair and Recovery for Active Directory appeared first on Wiredwolf Canada.

The production environment:

• One AD server is already in Azure
• One AD server is hosted on-premises in a VMware hypervisor
• Backup is done on both servers with CommVault

We created a Resource Group and populated with the following:

• Virtual Network separate from our production network with 2 subnets with no Public IP
  • Cleanroom Subnet
  • AzureBastionSubnet
• Bastion
• StorageAccount

Using CommVault both servers were restored to the vnet subnet for the Cleanroom.

The Azure server came up right away with no issues.  AD DS was healthy.

The VMware hosted server did not come up – it could not be booted and gave a BSOD of 0xc00002e2 which is Directory Services could not be started.  Uh oh – this is a problem.

Azure can be a little difficult when it comes to a BSOD on a server.  There is no ‘console’ per-say so there’s no way to reboot the server and start pounding on the F8 key to start in Directory Services Recovery Mode.  Even after an exhaustive Google search session, there wasn’t a lot to be found.  Lots of articles on how to enable DSRM on a failed domain controller if the option wasn’t available, but very little on how to actually get a VM to boot into DSRM.

The good news is it is doable, but it’s quite complex and involved.  Here are the steps I performed to achieve my goal:

Step 1 – create a recovery VM

This is done by utilizing Azure CLI commands.

Repair a Windows VM by using the Azure Virtual Machine repair commands – Virtual Machines | Microsoft Learn

I used the CLI option in the top right of Azure to open a CLI shell into Azure.  This is the simplest way to get a shell without having to install a whole bunch of Azure modules into PowerShell and navigate MFA authentication, etc.

az extension add -n vm-repair

This installs the extension so you can run the repair and create the repair environment

If you’ve already installed the extension at some point you may need to update it:

az extension update -n vm-repair

You will be prompted to enable a Public IP – if you aren’t using Bastion you might as well indicate Y – you will need it to access your recovery VM.

Now you run some simple commands 

az vm repair create -g MyResourceGroup -n myVM –repair-username username –repair-password ‘password!234’ –enable-nested –verbose

• where MyResourceGroup is the resource group the failed VM is currently residing
• where myVM is the name of the failed VM
• where repair-username is the is admin user you want to use for your recovery VM 
• where repair-password is the admin password you want to use for your recovery VM (I suggest you use a strong password)
• where –enable-nested sets up your recovery VM (a 2016 server) with Hyper-V installed

Once these commands complete, you’ll have a new Resource Group called repair-myVM–timestamp which has your bootable 2016 recovery VM and a copy of your failed drives – the non-bootable AD controller system drives and whatever other disks were attached to the failed VM from the restore.

Boot up the myVM recovery VM if it isn’t already and use RDP (or Bastion if you have it) to connect to the server using the repair-username and repair-password you specified in the previous command.

• Your Windows 2016 Server should have Hpyer-V installed to it.  If not, install it.
• Open Computer Management -> Disk Management and make sure the drive(s) you need to use for Hyper-V VM are set to “Offline”
• Open Hyper-V and create a VM with appropriate vCPU cores and memory (acknowledging the limits of your Server 2016 recovery instance)
  • Do not create a drive – choose to add a drive later
• Edit your new Hyper-V VM
  • Click on IDE Controller 1
  • Click on Add Hard Drive
  • Select the physical disk available
  • Click on Apply then OK
• Right click on the VM and click on Connect
• Start the VM
• When it starts booting up start tapping F8 to get into the boot menu
• Select Directory Services Recovery Mode
• Log in with your DSRM (Administrator) password

You can now perform the steps to correct whatever issues you’re having with AD.

When completed and you can boot the VM up and log in with your domain credentials, shut down the VM, then shut down your Recovery VM.

Go back to your Azure CLI 

az vm repair restore -g MyResourceGroup -n MyVM –verbose

Obviously, you will again substitute your values for MyResourceGroup and MyVM (same as above).

This final command will completely remove the recovery environment including the repair-ResourceGroup that was created from your Azure tenant.  

You should now be able to boot your recovered AD Domain Controller in Azure.

If your issue was not with Active Directory or you’re having issues recovering Active Directory due to disk issues:

Troubleshoot Windows stop error – directory service initialization failure – Virtual Machines | Microsoft Learn

I found this to be an extremely helpful article on fixing disks with NTDS installed. 

• You can use the 2016 Server to run DISKPART on the attached drive to set the active partition (Gen 1 VMs)
• You can use REG QUERY to find the location of the NTDS (typically C:\Windows\NTDS but it can be moved such as I found in my case which complicated things considerably)
• You can run BCDEDIT commands to force the VM to boot to DSRM (safeboot dsrepair) so you can re-attach the disk to your VM in Azure and not worry about not being able to hit F8 on a console
  • Note – you will have to also reverse this or the BCD will always boot to DS Recovery Mode
  • Note – you still have to have your DSRM password to log in

Other Helpful Resources

I used all of these online resources to solve my problem.

Blue screen errors when booting an Azure VM – Virtual Machines | Microsoft Learn

Troubleshoot a Windows VM in the Azure portal – Virtual Machines | Microsoft Learn

How to reset the Directory Services Restore Mode administrator account password – Windows Server | Microsoft Learn

How to fix Error 0xc00002e2 after rebooting Windows Domain Controller – Hostway Help Center

Azure Serial Console for Windows – Virtual Machines | Microsoft Learn

Fixing non-bootable Azure Virtual Machine (cloud-aware.net)

How to boot Windows Server in Directory Services Restore Mode (DSRM) (kapilarya.com)

Recovering the Active Directory database in Windows Server 2012 R2 | Dell Canada

How to Deploy Hyper-V Nested Virtualization on Azure: Full Overview (nakivo.com)

Attach a managed data disk to a Windows VM – Azure – Azure Virtual Machines | Microsoft Learn

The post Recover a Domain Controller in Azure appeared first on Wiredwolf Canada.