What to do when things go wrong with Citrix Command Center?


, , ,

There are many things you can do to mess up the Citrix Command Center and I will keep updating this document as I come across solutions to things I mess up. 🙂

Problem: Changing the default port for the web interface to 80 or 443.
Solution: Reverting the default port to 8443 or something unique.

The moment you make this change to the port and restart the server or the Citrix Command Center service the service will no longer start. Well, if you are lucky and if there were no bindings whatsoever to port 80 and 443 it will work. Otherwise you will be stranded with no way to change the configuration since the web interface is unavailable. Here’s what you need to do when this happens so you can get things going back again.

  1. Find the wrapper.conf file in the directory “C:\Program Files (x86)\Citrix\Citrix Command Center\conf\”. This is where the port configurations for the Citrix Command Center are stored.
  2. Find the attributes,
    • wrapper.java.additional.3=-Dwebcontainer.port
    • wrapper.java.additional.4=-Dresource_check
    • wrapper.java.additional.5=-Dwebserver.port
    • wrapper.app.parameter.5 –> refers to WEBSERVER_PORT
    • wrapper.app.parameter.7 –> refers to WEBCONTAINER_PORT
    • wrapper.java.additional.21=-Dssl.port
  3. Replace the port 80 or 443 with 8443 or anything that is unique.
  4. Save the changes and restart the service “Citrix Command Center Service”.

The service will start successfully and you will be able to access the web interface as usual.

Problem: LDAP or RADIUS authentications fail and the default root account is disabled preventing logons to the Command Center.
Solution: Enable the root account.

The statuses of the user accounts in the Command Center are stored in the table “UserConfTable” in SQL database connected to it. We need to go into the table and set the root account’s status to “enabled” so that we can use it.

  1. Run the Microsoft SQL Management Studio and connect to the Citrix Command Center database.
  2. Execute the SQL command UPDATE UserConfTable SET STATUS = 'enabled' WHERE USERNAME = 'root';

Now you should be able to use the root account from the web interface.

These instructions apply to Citrix Command Center 5.2. They have not been tested with other versions.

Preparing for 70-743 Yourself!


, , , ,

I thought of numerous times to write this to help you folks prepare for the exam 70-743 (Upgrading Your Skills to MCSA: Windows Server 2016) but never had the time to do it. Hope this still finds you well in time!

When I received the invitation for the beta exam last year I immediately accepted it and scheduled it for August. This was my first beta exam and I never really understood how much effort I had to put into the exam until it was a week or two before the exam day. I usually start with instructor-led video tutorials and end with cross checking that I have covered everything that is mentioned in the exam blueprint. Man, this was brand new, in beta, and I could hardly find anything to get going except the blueprint. I like challenges and I don’t usually give up; wasn’t planning to start with this one either. I jotted down the blueprint and started mapping the online content the best I could to expand each of those key components, which ended up being my guide, followed by a small lab environment built over Hyper-V and also a few years of experience with Windows Server and PowerShell under the belt.

I thought of sharing this sooner but I wanted to wait until I see that I have passed the exam myself before I send others down a path of unknown fate. I’m ready to share it now and hopefully this helps you in getting through the exam. I am certifying the quality of this guide with my fresh “Charter” status on the certification. 🙂

Fair warning, I prepared this guide for the beta version of the exam and the blueprint may have changed since then. Make sure you are always up to date with the exam blueprint that you can find here. Also, this exam has more weight on knowing how to do most of the things via PowerShell and working with containers and Docker.

I wish you all the very best in your certification attempts and if this helped you pass I would be really happy to know about it! Share the good news with me if time permits.

70-743 Self Study Exam Guide

You can find this guide in a well formatted manner in the PDF:  70-743 Self Study Guide

Install Windows Servers in host and compute environments

  • Install, upgrade, and migrate servers and workloads
    • Determine Windows Server 2016 installation requirements; determine appropriate Windows Server 2016 editions per workloads; install Windows Server 2016; install Windows Server 2016 features and roles; install and configure Windows Server Core; manage Windows Server Core installations using Windows PowerShell, command line, and remote management capabilities; implement Windows PowerShell Desired State Configuration (DSC) to install and maintain integrity of installed environments; perform upgrades and migrations of servers and core workloads from Windows Server 2008 and Windows Server 2012 to Windows Server 2016; determine the appropriate activation model for server installation, such as Automatic Virtual Machine Activation (AVMA), Key Management Service (KMS), and Active Directory-based Activation
  • For TP 5
    • Windows Server Edition Family (Datacenter): 6XBNX-4JQGW-QX6QG-74P76-72V67
    • Windows Server Edition Family (Standard): MFY9F-XBN2F-TYFMP-CCV49-RMYVH
    • Windows Server Essentials: NYK9H-Y2FDB-2XKGC-F2XHK-WTT88

Implement storage solutions


Implement Hyper-V


  • Install and configure Hyper-V
    • Determine hardware and compatibility requirements for installing Hyper-V, install Hyper-V, install management tools, upgrade from existing versions of Hyper-V, delegate virtual machine management, perform remote management of Hyper-V hosts, configure virtual machines using Windows PowerShell Direct, implement nested virtualization
        • Disable DM: Set-VMMemory -VMName <VMName> -DynamicMemoryEnabled $false
        • 2 packet routing options:
  • MAC Spoofing at the virtual switch of the physical host.
  • Use new NAT switch at the “middle VM”.
  1. Aggregated: IOPS limits are shared across all assignee VHD’s.
  2. Dedicated: IOPS are dedicated to individual assignee VHD’s.
        • Create policy:
          • New-StorageQosPolicy -Name Desktop -PolicyType Dedicated -MinimumIops 100 -MaximumIops 200 [-PolicyId 24735331-0a1b-4a69-a973-dd6dd74ea07f -MaximumIOBandwidth 10MB]
        • Set policy:
          • Get-VM -Name “VM Name” | Get-VMHardDiskDrive | Set-VMHardDiskDrive -QoSPolicyID $storagePolicy.PolicyId
        • Confirm that the policies are applied.
          • Get-StorageQoSFlow
            • Uses an average of each counter using a 5-minute sliding window.
        • Remove policy:
          • Get-VM -Name “VM Name” | Get-VMHardDiskDrive | Set-VMHardDiskDrive -QoSPolicyID $null
      • sQoS Normalization
        • Get IOPS normalization size:
          • Get-QoSPolicyStore
        • Set IOPS normalization size:
          • Set-QoSPolicyStore -IOPSNormalizationSize 32KB
      • sQoS Monitoring
        • Get-StorageSubSystem -FriendlyName “FriendlyName” | Debug-StorageSubSystem
  • Configure Hyper-V networking
    • Add and remove virtual network interface cards (vNICs), configure Hyper-V virtual switches, optimize network performance, configure MAC addresses, configure network isolation, configure synthetic and legacy virtual network adapters, configure NIC teaming in VMs, configure virtual machine queue (VMQ), enable Remote Direct Memory Access (RDMA) on network adapters bound to a Hyper-V virtual switch using Switch Embedded Teaming (SET), configure Bandwidth Management

Implement Windows containers


  • Deploy Windows containers
    • Determine installation requirements and appropriate scenarios for Windows containers, install and configure Windows Server container host in physical or virtualized environments, install and configure Windows Server container host to Windows Server Core or Nano Server in a physical or virtualized environment, install Docker on Windows Server and Nano Server, configure Docker daemon start-up options, configure Windows PowerShell for use with containers, install a base operating system, tag an image, uninstall an operating system image, create Windows Server containers, create Hyper-V containers
  • Manage Windows containers

Implement high availability


  • Implement high availability and disaster recovery options in Hyper-V
  • Implement failover clustering
    • Implement Workgroup, Single, and Multi Domain clusters; configure quorum; configure cluster networking; restore single node or cluster configuration; configure cluster storage; implement Cluster-Aware Updating; implement Cluster Operating System Rolling Upgrade; configure and optimize clustered shared volumes (CSVs); configure clusters without network names; implement Scale-Out File Server (SoFS); determine different scenarios for the use of SoFS versus clustered File Server; determine usage scenarios for implementing guest clustering; implement a Clustered Storage Spaces solution using Shared SAS storage enclosures; implement Storage Replica; implement Cloud Witness; implement VM resiliency; implement shared VHDX as a storage solution for guest clusters
  • Implement Storage Spaces Direct
    • Determine scenario requirements for implementing Storage Spaces Direct, enable Storage Spaces direct using Windows PowerShell, implement a disaggregated Storage Spaces Direct scenario in a cluster, implement a hyper-converged Storage Spaces Direct scenario in a cluster
  • Manage failover clustering
    • Configure role-specific settings, including continuously available shares; configure VM monitoring; configure failover and preference settings; implement stretch and site-aware failover clusters; enable and configure node fairness
  • Manage VM movement in clustered nodes
    • Perform live migration; perform quick migration; perform storage migration; import, export, and copy VMs; configure VM network health protection; configure drain on shutdown

Implement Domain Name System (DNS)

  • Install and configure DNS servers
    • Determine DNS installation requirements, determine supported DNS deployment scenarios on Nano Server, install DNS, configure forwarders, configure Root Hints, configure delegation, implement DNS policies, implement DNS global settings using Windows PowerShell, configure Domain Name System Security Extensions (DNSSEC), configure DNS Socket Pool, configure cache locking, enable Response Rate Limiting; configure DNS-based Authentication of Named Entities (DANE), configure DNS logging, configure delegated administration, configure recursion settings, implement DNS performance tuning, configure global settings using Windows PowerShell

Implement IP Address Management (IPAM)

ALL: https://technet.microsoft.com/en-us/windows-server-docs/networking/technologies/ipam/ip-address-management–ipam-

  • Install and configure IPAM
    • Provision IPAM manually or by using Group Policy, configure server discovery, create and manage IP blocks and ranges, monitor utilization of IP address space, migrate existing workloads to IPAM, configure IPAM database storage using SQL Server, determine scenarios for using IPAM with System Center Virtual Machine Manager for physical and virtual IP address space management
  • Manage DNS and DHCP using IPAM
    • Manage DHCP server properties using IPAM, configure DHCP scopes and options, configure DHCP policies and failover, manage DNS server properties using IPAM, manage DNS zones and records, manage DNS and DHCP servers in multiple Active Directory forests, delegate administration for DNS and DHCP using role-based access control (RBAC)

Implement network connectivity and remote access solutions

Implement an advanced network infrastructure

  • Implement high performance network solutions
    • Implement NIC Teaming or the Switch Embedded Teaming (SET) solution and identify when to use each, enable and configure Receive Side Scaling (RSS), enable and configure network Quality of Service (QoS) with Data Center Bridging (DCB), enable and configure SMB Direct on Remote Direct Memory Access (RDMA) enabled network adapters, enable and configure SMB Multichannel, enable and configure virtual Receive Side Scaling (vRSS) on a Virtual Machine Queue (VMQ) capable network adapter, enable and configure Virtual Machine Multi-Queue (VMMQ), enable and configure Single-Root I/O Virtualization (SR-IOV) on a supported network adapter
      • VMQ:https://blogs.technet.microsoft.com/networking/2016/01/04/virtual-machine-queue-vmq-cpu-assignment-tips-and-tricks/
      • RDMA&SET:https://technet.microsoft.com/en-us/library/mt403349.aspx
      • RSS or VMQ can be used at a single time on a NIC.
      • “RssBaseProcNumber” represents the CPU core a NIC should use for processing and should always be set to a value so that it steers the network processing from the core 0 that is mostly used by the system processing.
      • SET load balancing
        • Hyper-V port: best when VMQ is used.
        • Dynamic: based on “flowlets” and egress flow is rebalanced in real time.
      • DCB enables QoS and flow control.
      • QoS for SMB Direct
        • Install-WindowsFeature Data-Center-Bridging
        • New-NetQosPolicy “SMB” -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
        • Enable-NetQosFlowControl -Priority 3
        • Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7
        • Enable-NetAdapterQos -InterfaceAlias “SLOT 2*”
        • New-NetQosTrafficClass “SMB” -Priority 3 -BandwidthPercentage 30 -Algorithm ETS
      • New RDMA vNIC for SMB Direct
        • New-VMSwitch -Name RDMAswitch -NetAdapterName “SLOT 2”
        • Add-VMNetworkAdapter -SwitchName RDMAswitch -Name SMB_1
        • Enable-NetAdapterRDMA “vEthernet (SMB_1)”
        • Verify RDMA capabilities: Get-NetAdapterRdma
  • Determine scenarios and requirements for implementing software-defined networking (SDN)
    • Determine deployment scenarios and network requirements for deploying SDN; determine requirements and scenarios for implementing Hyper-V Network Virtualization (HNV) using Network Virtualization Generic Route Encapsulation (NVGRE) encapsulation or Virtual Extensible LAN (VXLAN) encapsulation; determine scenarios for implementation of Software Load Balancer (SLB) for North-South and East-West load balancing; determine implementation scenarios for various types of Windows Server Gateways, including L3, GRE, and S2S, and their use; determine requirements and scenarios for distributed firewall policies and network security groups

Install and configure Active Directory Domain Services (AD DS)

  • Install and configure domain controllers
    • Install a new forest, add or remove a domain controller from a domain, upgrade a domain controller, install AD DS on a Server Core installation, install a domain controller from Install from Media (IFM), resolve DNS SRV record registration issues, configure a global catalog server, transfer and seize operations master roles, install and configure a read-only domain controller (RODC), configure domain controller cloning

Implement identity federation and access solutions


Additional Resources


MCSA – Windows Server 2016 – Charter Member


I always wanted to have the Charter status since I first started with Microsoft certifications 8 years back and after attempting my first beta exam with 70-743 to upgrade my MCSA was able to achieve it. 🙂

Microsoft’s explanation of the Charter status is as follows and it is different from what it used to be a couple of years back.
“Certification that was achieved within six months following the retail release date of the certification. Charter Members are recognized by being given the Charter version of the certificate acknowledging their early adoption of the technology solution.”

Fixing the Last Run Result 0x667 of a Scheduled Task

So today was another day with another cryptic error from the Windows Server 2008 R2 Task Scheduler. Interestingly (or as usual may be 🙂 ), the MSDN article on Task Scheduler Error and Success Constants is missing the error code. A further look into the task’s history revealed a message saying “Task Scheduler successfully completed task “\Restart Service” , instance “{ca1e5236-51d1-483f-9c39-40832f015120}” ,action “C:\Windows\System32\sc.exe” with return code 1639.” with a return code of 1639.

1639 in the MSDN article “System Error Codes (1300-1699)” referred to “ERROR_INVALID_COMMAND_LINE” which indeed it was because I was trying to use “;” as the command separator with PowerShell in mind. Replacing “;” with “&&” fixed the problem.

Note – This was a fast publish article so that the information is available to everyone without delay. Fast publish articles of mine may not be very well formatted initially.

Fixing the Last Run Result 0xFFFD0000 of a Scheduled Task



So I found out that one of my auditing scripts that were scheduled to run in Windows Task Scheduler was not working as expected and the “Last Run Result” of it was shown as 0xFFFD0000 which had no documentation on what it was trying to say. Luckily for me Brian at briantist.com had come across this issue before and had some pointers on what to look for. The problem was that recently we renamed the folder structure in which my script was and once the path was corrected the task started to work fine.


Sources: https://www.briantist.com/errors/scheduled-task-powershell-0xfffd0000/

Configuring a Windows Server 2012 R2 PHP Web Server


, , , , , , , ,

     $r = shell_exec ("powershell -File $tools\create-aduser.ps1 -InputFormat none > NULL");

See something interesting above? That *.ps1 part. Yeah, that’s why I love PHP! PHP is advanced, full of features, capable of integrating with different systems, and open source. That’s why I “like” it but the reason for me to “love” it is it’s ability to execute PowerShell scripts. Imagine a set of complex PowerShell scripts being backed by a powerful yet simple PHP powered web portal. Then imagine people with no deep technical knowledge performing technically complex tasks from a web page in a controlled manner without any knowledge of PowerShell. How cool is that!? If you’re like me then you might be dreaming of all the new web pages that are going to make your life easier right now.

PHP can be easily configured in Unix environment. I have run PHP on Apache on a Ubuntu box and it was very easy to configure it initially. However since I’m focusing on PowerShell and administering Windows based systems Microsoft IIS Server seems to be the ideal web server but it’s a pain to configure it initially and get everything started. Once you configure everything so that you can view the output of a *.php web page with no errors you are good to go and play PHP however you want.

The difficulty in configuration exists in the correct registration of PHP in IIS. For Windows Server 2008 R2 you can use the tool PHP Manager for IIS that is designed for IIS 7 and IIS 7.5 at the time of this writing for the initial configuration. This tool is a life saver! Once you provide the path of the PHP executable to it and select “Register new PHP version” it goes ahead and takes care of all the necessary registrations. It’s 2015 and 2008 R2 is way old! We need to play with the new stuff. So when you start doing the configuration in Windows Server 2012 R2, well the PHP Manager for IIS is not supported and you have to look for a different option. I have been pulling my hair while reading the enormous number of forum posts for quite some time about the issues that we have to face when we try to configure everything ourselves. Something always falls through the cracks and it never works! But last week I think I cracked it and I know the exact recipe that it needs to work in a single streak.

…and everything starts at Build a PHP Website on IIS on TechNet! This TechNet article is a great guide with detailed steps. Therefore I’m not going to repeat the things in here. Instead I will fill in the gaps so you have everything to get this working.

Microsoft Web Platform Installer – The new web guy in town!

Microsoft Web PI

Similar to PHP manager for IIS the Web PI will take care of the necessary PHP registrations in IIS and some of its dependencies such as WinCache extension when you install PHP through it. As I usually say, if you want to burn a few more calories, you can do it manually as instructed in the document.

THE HTTP Error 500.0 – Internal Server Error

Once the download is done and you go for the first test to see if PHP is working, if you are lucky and privileged like the most of us IT folks you’ll be presented with a BIG RED HTTP Error 500 web page! 🙂 This is where we start to pull our hair and start messing with everything including NTFS permissions.

…but the first, the very FIRST thing to check is the Visual C++ packages that are installed in the Windows server. Let’s say this in quotes to emphasize it!

For PHP to work on a Microsoft IIS Server each PHP version needs its matching Visual C++ package installed!

So there’s that! Since there are many PHP versions and many Visual C++ packages available I don’t know all the matching pairs at the moment but I’ll try to update this post with them when I come across them. The latest PHP version that is available through the Web PI, at this writing is version 5.6. The matching Visual C++ package for PHP v5.6 is Visual C++ Redistributable for Visual Studio 2012 Update 4. Once this package is installed everything should be working fine.

THE 500.0 persists even after the matching Visual C++ package is installed

If the HTTP Error 500 is still there then it needs further troubleshooting. You can start by checking the IIS server-level FastCGI settings and the Handler Mappings, and continue troubleshooting along the steps mentioned in the HTTP Error 500.0 web page.

IIS server-level FastCGI settings and Handler Mappings configuration of a working PHP host

FastCGI Settings

  • Full path : Path to php-cgi.exe (e.g. C:\Program Files(x86)\PHP\v5.6\php-cgi.exe)
  • Arguments : default
  • Max. Instances : default
  • Instance Max. Requests : default

Handler Mappings

  • Path : *.php
  • State : Enabled
  • Path Type : File or Folder
  • Handler : FastCgiModule
  • Entry Type : Local

Let’s make those wonderful PowerShell powered web sites!

Update: 2016-02-17

The new PHP 7.0.0 package in the Web Platform Installer 5.0 seems to resolve these dependencies automatically. Yesterday I installed PHP 7.0.0 (x64) on a Windows Server 2012 R2 machine from Web PI and I was surprised to see PHP working right away after the installation. When I checked the Programs and Features I could see the matching Visual C++ packages installed automatically.


PHP package in Web PI 5.0


Automatically installed Microsoft Visual C++ package


Enabling WinRM on Domain Controllers


, , , , ,

WinRM is a really cool feature when scripting tasks and managing Windows servers remotely. I’ve been struggling to enable it on the domain controllers in our globally-distributed environment, following almost every guide I could find online but haven’t had any success until yesterday. So I thought about having it documented here in case someone else comes across similar problems in future.

Configuring the listeners

I’m going to use Kerberos authentication for WinRM so the configuration is quite simple. If you are planning to use a different type of authentication such as basic authentication or CredSSP then you’ll need a few additional steps which I won’t be discussing here.

The first step is to configure the WinRM listeners for IPv4 and IPv6. I’m going to allow the WinRM listeners to listen on all IP addresses of the domain controllers in this case. If you want to restrict it to listen only on certain IP addresses follow the instructions given with the settings. So I’m configuring the GPO setting Computer Configuration / Policies / Administrative Templates / Windows Components Windows Remote Management (WinRM) / WinRM Service to listen on all IP addresses as follows.


Configuring the Windows service

Now since we have the listeners configured we need the WinRM service get going for the configurations to take effect. Let’s make sure that this service is configured to run automatically using group policy. Of course you can do this manually on all the domain controllers if you feel like burning a few more calories. 😉

I’m adding a new service in the GPO setting Computer Configuration / Preferences / Control Panel Settings / Services for WinRM and setting the “Startup” to “Automatic (Delayed Start)” and “Service action” to “Start service”.

winrm-svc winrm-svc-1

Once the service is running you should be good to go. However if you have the firewall turned on then you need to make sure the necessary ports are allowed through it. I won’t cover the firewall configuration in here.

Exceptional Cases

“The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.”


I’m not completely sure of the exact reasons for this error but I’ll explain my experience with it. Our domain is one of those where the user accounts are member of a large number of Active Directory security groups. With systems that use Windows authentication (or Kerberos authentication) with HTTP we usually come across problems with authentication due to the large size of the Kerberos tokens. This error tries to explain this problem (in its own way I guess :-P). The solution however is to configure the HTTP service to process larger HTTP headers. We can do this by configuring the two registry keys MaxRequestBytes and MaxFieldLength for HTTP service parameters. Let’s do this via group policy as well so we have everything at a single place and applied consistently across all the domain controllers.

We will configure the registry settings under Computer Configurations / Preferences / Windows Settings / Registry as follows.



The following image shows the error given when trying to connect to a domain controller through WinRM before performing this configuration and the successful connection after applying the new settings.



The Ultimate RODC Q&A


, , , , , ,

During the past couple of months many people have asked me a lot of questions regarding the functionality of RODCs. It’s a relatively new concept and some information are not easily available as well. So I thought of recording them in this post so the information is easily available at a single place for everyone. The purpose of this post is to answer the questions to which the answers are not readily available out there. Therefore, the basic stuff are obviously not covered here except for a few of them like the first two. 🙂

What is a RODC?

A Windows domain controller that holds complete, read-only copies of the partitions of the Active Directory database and a read-only copy of the SYSVOL folder and its contents.

Why a RODC?

By selectively caching credentials, RODCs address some of the challenges that enterprises can encounter in branch office locations where physical security of the domain controllers cannot be guaranteed or where other applications must run on the domain controller and be maintained by a server administrator that is not a domain administrator.

Is RODC DNS different?

Yes. The DNS of a RODC is also read-only and it refers to a writable DNS server for any write requests that are directed to it. Afterwards the new information is replicated back to itself.

How does dynamic DNS work with RODC?

When a client attempts a dynamic update, it sends a Start of Authority (SOA) query to its preferred DNS server. Typically, clients are configured to use the DNS server in their branch site as their preferred DNS server. The RODC reads its SOA record and at best effort return a writable Windows Server 2008 domain controller to the client. The client makes the dynamic update at the writable DNS server returned. The RODC waits a time that is a minimum of 30 seconds and a maximum of 210 seconds and attempts to replicate the updated DNS record object in Active Directory from the DNS server that it referred the client to through a Replicate Single Object (RSO) operation back to itself.

How are the minimum and maximum waiting times calculated?

The SOA query triggers the DNS server on the RODC to put an entry in remotePollList, which is an internal queue on each DNS server. The entry includes the following:

  • The object to be replicated
  • The source domain controller to replicate from
  • A time stamp

The time stamp is set to a time in the future that is equal to the current time plus a replication delay. The replication delay is controlled by a registry setting named DsRemoteReplicationDelay. By default, the value of this setting is 30 seconds. The internal queue (remotePollList) is processed at regular intervals. The queue-processing interval is controlled by a registry setting named DSPollingInterval. By default, the value of the interval is 3 minutes (180 seconds).

When the DNS server processes the queue, it attempts to replicate only objects whose time stamp is less than current time. Therefore, the delay between the time that the RODC refers the client to an authoritative DNS server and then attempts to replicate in is determined by the following:

  • The next time that the DNS server processes the queue
  • Whether the remote replication delay that is set on the entry in the queue has elapsed

If the default values for the registry settings are used then the amount of time before the RODC attempts to perform a RSO operation is a minimum of 30 seconds where the DsRemoteReplicationDelay has exceeded and a polling operation is executed, and a maximum of 210 (180+30) seconds where the DsRemoteReplicationDelay has exceeded and a polling operation is yet to be executed.

Can these values be changed for faster replication?

Yes. You can modify the values of these registry settings to reduce the amount of time before the RODC attempts to replicate the DNS update. The minimum value for the DsRemoteReplicationDelay setting is 5 seconds. The minimum value for the DSPollingInterval setting is 30 seconds. When the minimum values are used the amount of time before the RODC attempts to replicate the DNS update is a minimum of 5 seconds and a maximum of 35 seconds.

Will the changes affect other Active Directory operations of the RODC?

Yes. DsPollingInterval controls all Active Directory polling, not just RODC RSO handling. If you change this value, be aware that this change will affect more than just RODC RSO operations. For example, this setting will affect how often the DNS server polls Active Directory for new or updated resource records or DNS zones.

What is a RSO operation?

A RSO operation is an operational attribute named replicateSingleObject that has existed in Active Directory since Windows 2000 and allows replication of a single object by using a LDAP modify operation of the replicateSingleObject attribute. However the replicateSingleObject has been updated in Windows Server 2008 to support replication of secrets to RODCs.

Can a large number of RSO operations overload a domain controller?

No. The maximum number of RSO requests per 5 minutes cycle is limited to 300 to prevent Denial of Service attacks.

Do RODC’s register NS records?

No. RODC’s do not register Name Server (NS) records.

What is the SOA selection model for RODC’s?

  1. Try to select a writable domain controller that is running Windows Server 2008 and is published as a Name Server for the zone.
  2. Pick a random NS from the NS list if there are no Windows Server 2008 writable domain controllers that have published a NS for the zone.

Can a RODC select another RODC as a SOA during the selection process?

No. Since RODCs do not register NS records they are not available for selection.

Are SOAs selected every time a DNS update operation is requested?

No. The current SOA target DC is maintained separately for each zone and re-selected every 20 minutes. The duration is not configurable.

Is the same SOA DC selected during each selection process?

No. The selection algorithm contains a random component to try to spread load between writable domain controllers.

Is there a tool to make the DNS related registry modifications in a RODC safely?

Yes. To modify any of the registry entries that are related to the RSO operations for DNS updates on an RODC, use the Dnscmd.exe command-line tool to set the appropriate parameter.
Example: “dnscmd <server>.<domain>.<com> /Config /DsRemoteReplicationDelay 10”

Can we make changes in the SYSVOL of a RODC?

Yes but the changes will be overwritten with the SYSVOL content from a writable domain controller during the next replication cycle. So it would be a useless task.

Do RODCs store any credentials at all?

Yes. It stores the credentials for its own computer account and the krbtgt account that is unique to it but nothing else.

Do RODCs use the normal krbtgt account of the domain for authentication services?

No. During the promotion of a RODC a separate krbtgt account is created on the domain for the use of that particular RODC only. This account is used for all authentication operations performed at this RODC.

Are RODCs advertised as Key Distribution Centers (KDCs)?

Yes but only for its own site.

If the credentials are not replicated then how can a user or computer authenticate for the first time with a RODC?

The first time an account attempts to authenticate to an RODC, the RODC sends the request to a writable domain controller. If the authentication is successful, the RODC also requests a copy of the appropriate credentials. The writable domain controller recognizes that the request is coming from an RODC and consults the Password Replication Policy that is in effect for that RODC to check if the requested credentials are allowed to be cached.

What is a “Password Replication Policy”?

The Password Replication Policy determines if a set of credentials are allowed to be replicated and stored on a particular RODC. If allowed, a writable domain controller sends the credentials to the RODC, and the RODC caches them. After the credentials are cached the next time that user or computer attempts to log on, the request is directly serviced by the RODC.

Do RODCs cache credentials for eternity?

No. The cached credentials are flushed once they are changed.

Can a non-domain administrator be made an administrator of a RODC?

Yes. This is known as “Administrative Role Separation”. A user can be given administrative access to a single RODC without giving administrative access to any other domain controllers.

What explicit access the “RODC admins” have on Active Directory?


Do RODCs support DFS-n and DFS-R?


How does DFS replication work in a RODC?

As for ADDS, the DFS replication of a RODC is also unidirectional in the inbound direction.

A RODC is a domain controller. Can a corruption or a potentially harmful change at a RODC damage the whole domain?

No. No changes at a RODC are replicated outbound.

Can a non-domain administrator promote a RODC?

Yes but the new RODC’s domain controller account should be created prior to the promotion by a domain administrator. Check Steps for Deploying a RODC for more information.

How do RODC’s handle password changes?

When a password change is requested by a user or computer in a RODC site the request is forwarded to a writable domain controller that runs Windows Server 2008 or later. The next steps are the same as would occur if the password change happened directly on the writable domain controller.

Can a cached password be cleared/flushed from a RODC?

No. There is no mechanism to erase passwords after they are cached on a RODC. If you want to clear a password that is stored on a RODC, the password should be reset at a writable domain controller.

Are there any advantages of hosting RODCs on Microsoft Azure?

Yes. Microsoft Azure does not charge for inbound network traffic. Since RODCs only perform inbound replications there will be no chargeback for network usage. Usage of other resources such as compute and storage will still be charged.

What will fail at a RODC site if a writable domain controller is not available?

  • Access to Windows Management Instrumentation (WMI) filters by Group Policy.
    • Failure to access WMI filters may prevent affected clients from applying intended Group Policy or cause those clients to improperly apply Group Policy.
  • Application of Internet Protocol Security (IPsec) policies by IPsec clients.
  • Time synchronization in Windows XP and Windows Server 2003.
    • The Windows Time service (W32time) in Windows XP and Windows Server 2003 does not recognize an RODC.
  • Domain joins.
  • Password changes.
  • Retrieval and creation of public key certificates.
    • The Data Protection Application Programming Interface (DPAPI) on client computers that only have access to an RODC cannot decrypt master keys unless they have previously contacted a writable domain controller and retrieved a public key certificate. Clients that only have access to an RODC cannot decrypt master keys.
  • Successful publishing of printers in ADDS.
    • If an RODC services a client request to publish a printer, it forwards the request to a writable domain controller. The spooler attempts to read from the RODC immediately after the write. Since the information has not yet been replicated to the RODC, and spooler fails the publish operation. All spooler internal structures are updated, and the printer is marked as unpublished.
  • Retrieval of printers published in ADDS.
  • Active Directory Service Interfaces (ADSI) calls.


Delegating User/Computer Delegation to Non-domain Admins


, , , , ,

Last week I came across a situation where a user needed to set up Microsoft SQL Server Reporting Services and was having difficulties in integrating Kerberos to it. He needed to change the options related to account delegation continuously on several user accounts and computer accounts that he was using for this purpose. Since he had no experience with Active Directory I was helping him with his queries and changing these options when needed. After some time, my schedule started to conflict with his troubleshooting. Therefore I decided to give him a quick training on Active Directory basics and delegate the necessary permissions for him so he can continue with his work at his own pace.

From a quick glance, trusting user/computer account for delegation is an option related to each user or computer account. So granting Full control to the target account for the user should work fine. But this was not the case! After the user was granted permissions, he was receiving the error message “The following Active Directory Domain Services error occurred: Access is denied.” Now this is a “super helpful” error message with a lot of information in it, that could direct us to a solution – “Thanks Microsoft”! 😀

Before the user was granted Full control - all options are greyed out due to lack of permissions.

Before the user was granted Full control – all options are greyed out due to lack of permissions.

The options are available after the user was granted Full control for this computer account.

The options are available after the user was granted Full control for this computer account.

The "super helpful" error message received when trying to save delegation options.

The “super helpful” error message received when trying to save delegation options.

A day passed. I tried delegating at the OU level – no success! Since my mind was telling me that there should be a solution I decided to seek help from Google! One of the interesting search results was this post and it directed me to a setting in group policy. Who would think to look that far for a simple thing like this?!!

The delegation settings for user accounts and computer accounts belong to the domain controllers, sort of. Therefore a user needs to have this permission on the domain controllers. When you go to User Rights Assignment section in the Default Domain Controllers Policy (Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment), you can find the setting Enable computer and user accounts to be trusted for delegation. By default only the Administrators are granted this permission. Since I did not want to come here and change the GPO every time a user is in need of this permission, I decided to create a security group named “Access.EnableDelegation” and added it to this GPO setting. Once the user was added to this security group, voila! he is now able to change this setting himself and I’ve got a very happy customer. 🙂

The GPO setting

The GPO setting “Enable computer and user accounts to be trusted for delegation”

Adding the new security group to the GPO setting

Adding the new security group to the GPO setting

A few things to note

  • Microsoft cautions on using this permission as it can impersonate clients and use their credentials to gain access to network resources. Therefore, make sure that the users are revoked of this permission once they are done using it.

Misuse of this user right, or of the Trusted for Delegation setting, could make the network vulnerable to sophisticated attacks using Trojan horse programs that impersonate incoming clients and use their credentials to gain access to network resources.

  • It is a best practice to perform custom changes such as this one in a separate GPO rather than in the Default Domain Controllers Policy.
  • Once a user is granted this permission he has access to change delegation options for all user accounts and computer accounts. Be mindful of the scope of this permission. The only workaround for this would be to use “data hiding”. No one can change anything that is not visible to them!