The Definitive Algorithm for Calculating Active Directory Site Link Costs


, ,

If you have worked with Active Directory for some time and you have a lot of branch sites in your organization distributed globally then one thing you probably have come across is the need to make sure that you have your Active Directory Sites and Services (ADSS) configured properly and is maintained regularly. A properly configured ADSS infrastructure provides you with a very efficient Microsoft environment within your organization. The thing with the connections to branch sites is that these connections change over time due to connection upgrades, path changes within the WAN service provider, and etc. When these changes occur the site link costs that were once configured for can be invalid and a link that was the best previously might be the worst now. Thus these costs need to be updated promptly upon these changes so that the user experience is not affected.

What we usually do is, for convenience, to configure cost values based on our “gut feeling”. For example, if we have three datacenters in Sweden, USA, and Sri Lanka, and for our branch site in Australia we would say that the site link cost for Australia-Sri Lanka is 100, Australia-Sweden is 200, and Australia-USA is 300. If you have an idea about the network then this works for the most part. This is also very easy and convenient in day to day administration but things get complicated when you have a large number of site links and many changes are occurring within the network. So you might have had this in the back of your mind now, with almost everything being automated nowadays, this is something that we can handover to a computer to do. However for a computer to do something it needs to be told how exactly to do that and for something like this to be done in a consistent way we need a – yes, an algorithm!

Designing an algorithm is the first step. Automating how to use it is a simple task when we have the algorithm. I will focus on the algorithm in this post. I haven’t had the time yet to automate this but if you get there I’d love to hear your journey!

THE Algorithm

ROUND ((1024 / LOG10 (Bandwidth)) x LOG10 (Latency))

The syntax is designed for Microsoft Excel and I will explain this so you can use and translate the logic behind it however you want in your applications.

  • Bandwidth: The minimum bandwidth of the connection (between the two sites in a site link) in kilobits per second (Kbps).
  • Latency: The average round trip time in the connection in milliseconds (ms).
  • LOG10: Common logarithm or logarithm to the base 10.
  • ROUND: The nearest whole number when rounded off.

Yeah, it’s simple as that!

In essence, this will generate low costs for high bandwidth low latency connections and generate high costs for low bandwidth high latency connections making the endpoints prefer the former over the latter for network traffic direction.

I’m writing this from Microsoft Ignite 2017, Orlando FL enjoying the beautiful climate with the sun blazing at 30 C. 🙂

Microsoft Ignite 2017, Orlando FL


What to do when things go wrong with Citrix Command Center?


, , ,

There are many things you can do to mess up the Citrix Command Center and I will keep updating this document as I come across solutions to things I mess up. 🙂

Problem: Changing the default port for the web interface to 80 or 443.
Solution: Reverting the default port to 8443 or something unique.

The moment you make this change to the port and restart the server or the Citrix Command Center service the service will no longer start. Well, if you are lucky and if there were no bindings whatsoever to port 80 and 443 it will work. Otherwise you will be stranded with no way to change the configuration since the web interface is unavailable. Here’s what you need to do when this happens so you can get things going back again.

  1. Find the wrapper.conf file in the directory “C:\Program Files (x86)\Citrix\Citrix Command Center\conf\”. This is where the port configurations for the Citrix Command Center are stored.
  2. Find the attributes,
    • –> refers to WEBSERVER_PORT
    • –> refers to WEBCONTAINER_PORT
  3. Replace the port 80 or 443 with 8443 or anything that is unique.
  4. Save the changes and restart the service “Citrix Command Center Service”.

The service will start successfully and you will be able to access the web interface as usual.

Problem: LDAP or RADIUS authentications fail and the default root account is disabled preventing logons to the Command Center.
Solution: Enable the root account.

The statuses of the user accounts in the Command Center are stored in the table “UserConfTable” in SQL database connected to it. We need to go into the table and set the root account’s status to “enabled” so that we can use it.

  1. Run the Microsoft SQL Management Studio and connect to the Citrix Command Center database.
  2. Execute the SQL command UPDATE UserConfTable SET STATUS = 'enabled' WHERE USERNAME = 'root';

Now you should be able to use the root account from the web interface.

These instructions apply to Citrix Command Center 5.2. They have not been tested with other versions.

MCSA – Windows Server 2016 – Charter Member


I always wanted to have the Charter status since I first started with Microsoft certifications 8 years back and after attempting my first beta exam with 70-743 to upgrade my MCSA was able to achieve it. 🙂

Microsoft’s explanation of the Charter status is as follows and it is different from what it used to be a couple of years back.
“Certification that was achieved within six months following the retail release date of the certification. Charter Members are recognized by being given the Charter version of the certificate acknowledging their early adoption of the technology solution.”

Fixing the Last Run Result 0x667 of a Scheduled Task

So today was another day with another cryptic error from the Windows Server 2008 R2 Task Scheduler. Interestingly (or as usual may be 🙂 ), the MSDN article on Task Scheduler Error and Success Constants is missing the error code. A further look into the task’s history revealed a message saying “Task Scheduler successfully completed task “\Restart Service” , instance “{ca1e5236-51d1-483f-9c39-40832f015120}” ,action “C:\Windows\System32\sc.exe” with return code 1639.” with a return code of 1639.

1639 in the MSDN article “System Error Codes (1300-1699)” referred to “ERROR_INVALID_COMMAND_LINE” which indeed it was because I was trying to use “;” as the command separator with PowerShell in mind. Replacing “;” with “&&” fixed the problem.

Note – This was a fast publish article so that the information is available to everyone without delay. Fast publish articles of mine may not be very well formatted initially.

Fixing the Last Run Result 0xFFFD0000 of a Scheduled Task



So I found out that one of my auditing scripts that were scheduled to run in Windows Task Scheduler was not working as expected and the “Last Run Result” of it was shown as 0xFFFD0000 which had no documentation on what it was trying to say. Luckily for me Brian at had come across this issue before and had some pointers on what to look for. The problem was that recently we renamed the folder structure in which my script was and once the path was corrected the task started to work fine.



Configuring a Windows Server 2012 R2 PHP Web Server


, , , , , , , ,

     $r = shell_exec ("powershell -File $tools\create-aduser.ps1 -InputFormat none > NULL");

See something interesting above? That *.ps1 part. Yeah, that’s why I love PHP! PHP is advanced, full of features, capable of integrating with different systems, and open source. That’s why I “like” it but the reason for me to “love” it is it’s ability to execute PowerShell scripts. Imagine a set of complex PowerShell scripts being backed by a powerful yet simple PHP powered web portal. Then imagine people with no deep technical knowledge performing technically complex tasks from a web page in a controlled manner without any knowledge of PowerShell. How cool is that!? If you’re like me then you might be dreaming of all the new web pages that are going to make your life easier right now.

PHP can be easily configured in Unix environment. I have run PHP on Apache on a Ubuntu box and it was very easy to configure it initially. However since I’m focusing on PowerShell and administering Windows based systems Microsoft IIS Server seems to be the ideal web server but it’s a pain to configure it initially and get everything started. Once you configure everything so that you can view the output of a *.php web page with no errors you are good to go and play PHP however you want.

The difficulty in configuration exists in the correct registration of PHP in IIS. For Windows Server 2008 R2 you can use the tool PHP Manager for IIS that is designed for IIS 7 and IIS 7.5 at the time of this writing for the initial configuration. This tool is a life saver! Once you provide the path of the PHP executable to it and select “Register new PHP version” it goes ahead and takes care of all the necessary registrations. It’s 2015 and 2008 R2 is way old! We need to play with the new stuff. So when you start doing the configuration in Windows Server 2012 R2, well the PHP Manager for IIS is not supported and you have to look for a different option. I have been pulling my hair while reading the enormous number of forum posts for quite some time about the issues that we have to face when we try to configure everything ourselves. Something always falls through the cracks and it never works! But last week I think I cracked it and I know the exact recipe that it needs to work in a single streak.

…and everything starts at Build a PHP Website on IIS on TechNet! This TechNet article is a great guide with detailed steps. Therefore I’m not going to repeat the things in here. Instead I will fill in the gaps so you have everything to get this working.

Microsoft Web Platform Installer – The new web guy in town!

Microsoft Web PI

Similar to PHP manager for IIS the Web PI will take care of the necessary PHP registrations in IIS and some of its dependencies such as WinCache extension when you install PHP through it. As I usually say, if you want to burn a few more calories, you can do it manually as instructed in the document.

THE HTTP Error 500.0 – Internal Server Error

Once the download is done and you go for the first test to see if PHP is working, if you are lucky and privileged like the most of us IT folks you’ll be presented with a BIG RED HTTP Error 500 web page! 🙂 This is where we start to pull our hair and start messing with everything including NTFS permissions.

…but the first, the very FIRST thing to check is the Visual C++ packages that are installed in the Windows server. Let’s say this in quotes to emphasize it!

For PHP to work on a Microsoft IIS Server each PHP version needs its matching Visual C++ package installed!

So there’s that! Since there are many PHP versions and many Visual C++ packages available I don’t know all the matching pairs at the moment but I’ll try to update this post with them when I come across them. The latest PHP version that is available through the Web PI, at this writing is version 5.6. The matching Visual C++ package for PHP v5.6 is Visual C++ Redistributable for Visual Studio 2012 Update 4. Once this package is installed everything should be working fine.

THE 500.0 persists even after the matching Visual C++ package is installed

If the HTTP Error 500 is still there then it needs further troubleshooting. You can start by checking the IIS server-level FastCGI settings and the Handler Mappings, and continue troubleshooting along the steps mentioned in the HTTP Error 500.0 web page.

IIS server-level FastCGI settings and Handler Mappings configuration of a working PHP host

FastCGI Settings

  • Full path : Path to php-cgi.exe (e.g. C:\Program Files(x86)\PHP\v5.6\php-cgi.exe)
  • Arguments : default
  • Max. Instances : default
  • Instance Max. Requests : default

Handler Mappings

  • Path : *.php
  • State : Enabled
  • Path Type : File or Folder
  • Handler : FastCgiModule
  • Entry Type : Local

Let’s make those wonderful PowerShell powered web sites!

Update: 2016-02-17

The new PHP 7.0.0 package in the Web Platform Installer 5.0 seems to resolve these dependencies automatically. Yesterday I installed PHP 7.0.0 (x64) on a Windows Server 2012 R2 machine from Web PI and I was surprised to see PHP working right away after the installation. When I checked the Programs and Features I could see the matching Visual C++ packages installed automatically.


PHP package in Web PI 5.0


Automatically installed Microsoft Visual C++ package


Enabling WinRM on Domain Controllers


, , , , ,

WinRM is a really cool feature when scripting tasks and managing Windows servers remotely. I’ve been struggling to enable it on the domain controllers in our globally-distributed environment, following almost every guide I could find online but haven’t had any success until yesterday. So I thought about having it documented here in case someone else comes across similar problems in future.

Configuring the listeners

I’m going to use Kerberos authentication for WinRM so the configuration is quite simple. If you are planning to use a different type of authentication such as basic authentication or CredSSP then you’ll need a few additional steps which I won’t be discussing here.

The first step is to configure the WinRM listeners for IPv4 and IPv6. I’m going to allow the WinRM listeners to listen on all IP addresses of the domain controllers in this case. If you want to restrict it to listen only on certain IP addresses follow the instructions given with the settings. So I’m configuring the GPO setting Computer Configuration / Policies / Administrative Templates / Windows Components Windows Remote Management (WinRM) / WinRM Service to listen on all IP addresses as follows.


Configuring the Windows service

Now since we have the listeners configured we need the WinRM service get going for the configurations to take effect. Let’s make sure that this service is configured to run automatically using group policy. Of course you can do this manually on all the domain controllers if you feel like burning a few more calories. 😉

I’m adding a new service in the GPO setting Computer Configuration / Preferences / Control Panel Settings / Services for WinRM and setting the “Startup” to “Automatic (Delayed Start)” and “Service action” to “Start service”.

winrm-svc winrm-svc-1

Once the service is running you should be good to go. However if you have the firewall turned on then you need to make sure the necessary ports are allowed through it. I won’t cover the firewall configuration in here.

Exceptional Cases

“The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.”


I’m not completely sure of the exact reasons for this error but I’ll explain my experience with it. Our domain is one of those where the user accounts are member of a large number of Active Directory security groups. With systems that use Windows authentication (or Kerberos authentication) with HTTP we usually come across problems with authentication due to the large size of the Kerberos tokens. This error tries to explain this problem (in its own way I guess :-P). The solution however is to configure the HTTP service to process larger HTTP headers. We can do this by configuring the two registry keys MaxRequestBytes and MaxFieldLength for HTTP service parameters. Let’s do this via group policy as well so we have everything at a single place and applied consistently across all the domain controllers.

We will configure the registry settings under Computer Configurations / Preferences / Windows Settings / Registry as follows.



The following image shows the error given when trying to connect to a domain controller through WinRM before performing this configuration and the successful connection after applying the new settings.



The Ultimate RODC Q&A


, , , , , ,

During the past couple of months many people have asked me a lot of questions regarding the functionality of RODCs. It’s a relatively new concept and some information are not easily available as well. So I thought of recording them in this post so the information is easily available at a single place for everyone. The purpose of this post is to answer the questions to which the answers are not readily available out there. Therefore, the basic stuff are obviously not covered here except for a few of them like the first two. 🙂

What is a RODC?

A Windows domain controller that holds complete, read-only copies of the partitions of the Active Directory database and a read-only copy of the SYSVOL folder and its contents.

Why a RODC?

By selectively caching credentials, RODCs address some of the challenges that enterprises can encounter in branch office locations where physical security of the domain controllers cannot be guaranteed or where other applications must run on the domain controller and be maintained by a server administrator that is not a domain administrator.

Is RODC DNS different?

Yes. The DNS of a RODC is also read-only and it refers to a writable DNS server for any write requests that are directed to it. Afterwards the new information is replicated back to itself.

How does dynamic DNS work with RODC?

When a client attempts a dynamic update, it sends a Start of Authority (SOA) query to its preferred DNS server. Typically, clients are configured to use the DNS server in their branch site as their preferred DNS server. The RODC reads its SOA record and at best effort return a writable Windows Server 2008 domain controller to the client. The client makes the dynamic update at the writable DNS server returned. The RODC waits a time that is a minimum of 30 seconds and a maximum of 210 seconds and attempts to replicate the updated DNS record object in Active Directory from the DNS server that it referred the client to through a Replicate Single Object (RSO) operation back to itself.

How are the minimum and maximum waiting times calculated?

The SOA query triggers the DNS server on the RODC to put an entry in remotePollList, which is an internal queue on each DNS server. The entry includes the following:

  • The object to be replicated
  • The source domain controller to replicate from
  • A time stamp

The time stamp is set to a time in the future that is equal to the current time plus a replication delay. The replication delay is controlled by a registry setting named DsRemoteReplicationDelay. By default, the value of this setting is 30 seconds. The internal queue (remotePollList) is processed at regular intervals. The queue-processing interval is controlled by a registry setting named DSPollingInterval. By default, the value of the interval is 3 minutes (180 seconds).

When the DNS server processes the queue, it attempts to replicate only objects whose time stamp is less than current time. Therefore, the delay between the time that the RODC refers the client to an authoritative DNS server and then attempts to replicate in is determined by the following:

  • The next time that the DNS server processes the queue
  • Whether the remote replication delay that is set on the entry in the queue has elapsed

If the default values for the registry settings are used then the amount of time before the RODC attempts to perform a RSO operation is a minimum of 30 seconds where the DsRemoteReplicationDelay has exceeded and a polling operation is executed, and a maximum of 210 (180+30) seconds where the DsRemoteReplicationDelay has exceeded and a polling operation is yet to be executed.

Can these values be changed for faster replication?

Yes. You can modify the values of these registry settings to reduce the amount of time before the RODC attempts to replicate the DNS update. The minimum value for the DsRemoteReplicationDelay setting is 5 seconds. The minimum value for the DSPollingInterval setting is 30 seconds. When the minimum values are used the amount of time before the RODC attempts to replicate the DNS update is a minimum of 5 seconds and a maximum of 35 seconds.

Will the changes affect other Active Directory operations of the RODC?

Yes. DsPollingInterval controls all Active Directory polling, not just RODC RSO handling. If you change this value, be aware that this change will affect more than just RODC RSO operations. For example, this setting will affect how often the DNS server polls Active Directory for new or updated resource records or DNS zones.

What is a RSO operation?

A RSO operation is an operational attribute named replicateSingleObject that has existed in Active Directory since Windows 2000 and allows replication of a single object by using a LDAP modify operation of the replicateSingleObject attribute. However the replicateSingleObject has been updated in Windows Server 2008 to support replication of secrets to RODCs.

Can a large number of RSO operations overload a domain controller?

No. The maximum number of RSO requests per 5 minutes cycle is limited to 300 to prevent Denial of Service attacks.

Do RODC’s register NS records?

No. RODC’s do not register Name Server (NS) records.

What is the SOA selection model for RODC’s?

  1. Try to select a writable domain controller that is running Windows Server 2008 and is published as a Name Server for the zone.
  2. Pick a random NS from the NS list if there are no Windows Server 2008 writable domain controllers that have published a NS for the zone.

Can a RODC select another RODC as a SOA during the selection process?

No. Since RODCs do not register NS records they are not available for selection.

Are SOAs selected every time a DNS update operation is requested?

No. The current SOA target DC is maintained separately for each zone and re-selected every 20 minutes. The duration is not configurable.

Is the same SOA DC selected during each selection process?

No. The selection algorithm contains a random component to try to spread load between writable domain controllers.

Is there a tool to make the DNS related registry modifications in a RODC safely?

Yes. To modify any of the registry entries that are related to the RSO operations for DNS updates on an RODC, use the Dnscmd.exe command-line tool to set the appropriate parameter.
Example: “dnscmd <server>.<domain>.<com> /Config /DsRemoteReplicationDelay 10”

Can we make changes in the SYSVOL of a RODC?

Yes but the changes will be overwritten with the SYSVOL content from a writable domain controller during the next replication cycle. So it would be a useless task.

Do RODCs store any credentials at all?

Yes. It stores the credentials for its own computer account and the krbtgt account that is unique to it but nothing else.

Do RODCs use the normal krbtgt account of the domain for authentication services?

No. During the promotion of a RODC a separate krbtgt account is created on the domain for the use of that particular RODC only. This account is used for all authentication operations performed at this RODC.

Are RODCs advertised as Key Distribution Centers (KDCs)?

Yes but only for its own site.

If the credentials are not replicated then how can a user or computer authenticate for the first time with a RODC?

The first time an account attempts to authenticate to an RODC, the RODC sends the request to a writable domain controller. If the authentication is successful, the RODC also requests a copy of the appropriate credentials. The writable domain controller recognizes that the request is coming from an RODC and consults the Password Replication Policy that is in effect for that RODC to check if the requested credentials are allowed to be cached.

What is a “Password Replication Policy”?

The Password Replication Policy determines if a set of credentials are allowed to be replicated and stored on a particular RODC. If allowed, a writable domain controller sends the credentials to the RODC, and the RODC caches them. After the credentials are cached the next time that user or computer attempts to log on, the request is directly serviced by the RODC.

Do RODCs cache credentials for eternity?

No. The cached credentials are flushed once they are changed.

Can a non-domain administrator be made an administrator of a RODC?

Yes. This is known as “Administrative Role Separation”. A user can be given administrative access to a single RODC without giving administrative access to any other domain controllers.

What explicit access the “RODC admins” have on Active Directory?


Do RODCs support DFS-n and DFS-R?


How does DFS replication work in a RODC?

As for ADDS, the DFS replication of a RODC is also unidirectional in the inbound direction.

A RODC is a domain controller. Can a corruption or a potentially harmful change at a RODC damage the whole domain?

No. No changes at a RODC are replicated outbound.

Can a non-domain administrator promote a RODC?

Yes but the new RODC’s domain controller account should be created prior to the promotion by a domain administrator. Check Steps for Deploying a RODC for more information.

How do RODC’s handle password changes?

When a password change is requested by a user or computer in a RODC site the request is forwarded to a writable domain controller that runs Windows Server 2008 or later. The next steps are the same as would occur if the password change happened directly on the writable domain controller.

Can a cached password be cleared/flushed from a RODC?

No. There is no mechanism to erase passwords after they are cached on a RODC. If you want to clear a password that is stored on a RODC, the password should be reset at a writable domain controller.

Are there any advantages of hosting RODCs on Microsoft Azure?

Yes. Microsoft Azure does not charge for inbound network traffic. Since RODCs only perform inbound replications there will be no chargeback for network usage. Usage of other resources such as compute and storage will still be charged.

What will fail at a RODC site if a writable domain controller is not available?

  • Access to Windows Management Instrumentation (WMI) filters by Group Policy.
    • Failure to access WMI filters may prevent affected clients from applying intended Group Policy or cause those clients to improperly apply Group Policy.
  • Application of Internet Protocol Security (IPsec) policies by IPsec clients.
  • Time synchronization in Windows XP and Windows Server 2003.
    • The Windows Time service (W32time) in Windows XP and Windows Server 2003 does not recognize an RODC.
  • Domain joins.
  • Password changes.
  • Retrieval and creation of public key certificates.
    • The Data Protection Application Programming Interface (DPAPI) on client computers that only have access to an RODC cannot decrypt master keys unless they have previously contacted a writable domain controller and retrieved a public key certificate. Clients that only have access to an RODC cannot decrypt master keys.
  • Successful publishing of printers in ADDS.
    • If an RODC services a client request to publish a printer, it forwards the request to a writable domain controller. The spooler attempts to read from the RODC immediately after the write. Since the information has not yet been replicated to the RODC, and spooler fails the publish operation. All spooler internal structures are updated, and the printer is marked as unpublished.
  • Retrieval of printers published in ADDS.
  • Active Directory Service Interfaces (ADSI) calls.


Delegating User/Computer Delegation to Non-domain Admins


, , , , ,

Last week I came across a situation where a user needed to set up Microsoft SQL Server Reporting Services and was having difficulties in integrating Kerberos to it. He needed to change the options related to account delegation continuously on several user accounts and computer accounts that he was using for this purpose. Since he had no experience with Active Directory I was helping him with his queries and changing these options when needed. After some time, my schedule started to conflict with his troubleshooting. Therefore I decided to give him a quick training on Active Directory basics and delegate the necessary permissions for him so he can continue with his work at his own pace.

From a quick glance, trusting user/computer account for delegation is an option related to each user or computer account. So granting Full control to the target account for the user should work fine. But this was not the case! After the user was granted permissions, he was receiving the error message “The following Active Directory Domain Services error occurred: Access is denied.” Now this is a “super helpful” error message with a lot of information in it, that could direct us to a solution – “Thanks Microsoft”! 😀

Before the user was granted Full control - all options are greyed out due to lack of permissions.

Before the user was granted Full control – all options are greyed out due to lack of permissions.

The options are available after the user was granted Full control for this computer account.

The options are available after the user was granted Full control for this computer account.

The "super helpful" error message received when trying to save delegation options.

The “super helpful” error message received when trying to save delegation options.

A day passed. I tried delegating at the OU level – no success! Since my mind was telling me that there should be a solution I decided to seek help from Google! One of the interesting search results was this post and it directed me to a setting in group policy. Who would think to look that far for a simple thing like this?!!

The delegation settings for user accounts and computer accounts belong to the domain controllers, sort of. Therefore a user needs to have this permission on the domain controllers. When you go to User Rights Assignment section in the Default Domain Controllers Policy (Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment), you can find the setting Enable computer and user accounts to be trusted for delegation. By default only the Administrators are granted this permission. Since I did not want to come here and change the GPO every time a user is in need of this permission, I decided to create a security group named “Access.EnableDelegation” and added it to this GPO setting. Once the user was added to this security group, voila! he is now able to change this setting himself and I’ve got a very happy customer. 🙂

The GPO setting

The GPO setting “Enable computer and user accounts to be trusted for delegation”

Adding the new security group to the GPO setting

Adding the new security group to the GPO setting

A few things to note

  • Microsoft cautions on using this permission as it can impersonate clients and use their credentials to gain access to network resources. Therefore, make sure that the users are revoked of this permission once they are done using it.

Misuse of this user right, or of the Trusted for Delegation setting, could make the network vulnerable to sophisticated attacks using Trojan horse programs that impersonate incoming clients and use their credentials to gain access to network resources.

  • It is a best practice to perform custom changes such as this one in a separate GPO rather than in the Default Domain Controllers Policy.
  • Once a user is granted this permission he has access to change delegation options for all user accounts and computer accounts. Be mindful of the scope of this permission. The only workaround for this would be to use “data hiding”. No one can change anything that is not visible to them!