TMG is taking more than 16 minutes to start on Windows 2008 R2

Yeah, ok today is not an OpsMgr post but about “Microsoft Forefront Threat Management Gateway server” (TMG).
Because there is almost nothing known about this strange issue I described the steps which I did to solve the case.
I had 6 Windows 2008 R2 server (for TMG) deployed in a ESX environment in which I have installed TMG with a single adapter. I was successful in installing TMG, defining the internal network etc, however when I restart the server for the first time after initial configuration, it takes approx. 16 minutes at the “Applying Computer Settings” which is very slow. After these 16 minutes I can login but the TMG Services are stopped.



- The following service is taking more than 16 minutes to start and may have stopped responding: Microsoft Forefront TMG Control
- The following service is taking more than 16 minutes to start and may have stopped responding: SQL Server Reporting Services (ISARS)
- The Microsoft Forefront TMG Control service terminated with service-specific error The wait operation timed out..
- The SQL Server Reporting Services (ISARS) service hung on starting.
- The Microsoft Forefront TMG Firewall service depends on the Microsoft Forefront TMG Control service which failed to start
- The Microsoft Forefront TMG Managed Control service depends on the Microsoft Forefront TMG Control service which failed to start
- The Microsoft Forefront TMG Job Scheduler service depends on the Microsoft Forefront TMG Control service which failed to start
I noticed that the issue happens circa 70% of the Windows servers; the other ones where functioning ok.
I had the option of manually starting the services after the reboot and everything works 100%. Until a new reboot the issue came back.
After this I applied the following steps;
Checked the following settings:
- IP configuration
- DNS configuration
- Windows Setup
- Ran the BPA; found nothing strange
All settings where ok

Installed the following updates:
- Service pack 1 for TMG 2010
- Software Update 1 for TMG 2010
- Software Update 2 for TMG 2010 (Hotfix)
Rebooted the server but no luck

Server dependencies
Then I checked the service startup dependencies comparing the working servers against the nonworking servers; I found out that the servers where the 16 minutes delay happens the dependencies where not setup properly.
Changed the startup dependencies according to the succesfull servers (default+http+keyiso) with the following command:

sc config isactrl depend= RasMan/SSTPSVC/FwEng/ISASTG/bfe/mpssvc/HTTP/KeyIso

Rebooted the server and Problem solved!
 

Rebooted the machine and it started up within 3 minutes without any errors.
Contacted Microsoft about this and they stated it is a known issue. They did not have an explanation why it does not happen all the time.
They confirmed it was a bug and the command I ran solved the issue.

Cause:
The issue is that while the Control service is starting (has not reported the started state back to SCM), its actions might require to start additional service(s).
This causes a Service Control Manager deadlock due to which services cannot start right on time. This causes the “slow boot” behavior.
Therefore, we added the dependencies, so that the required services are started before the control service. This way we avoid the above problem.

Enabling HTTPS Inspection with Forefront TMG Server

HTTPS inspection, is one of the greatest new features in TMG 2010. Prior to TMG, ISA Server was able to perform Inbound HTTPS inspection ( SSL bridging ), however, since the first generation of ISA Server, Outbound HTTPS Inspection was not available. Finally this feature is introduced with TMG 2010. In this article, I will be showing you how to enable and configure HTTPS Inspection.

By default, HTTPS Inspection is disabled. To enable it, follow the below steps:

1.  From the Left side node, click on Web Access Policy
   
   
   
   
2. Then from the right side node, and under the Tasks tab, click on Configure HTTPS Inspection
   
   
   
   
3. The HTTPS Outbound Inspection window will open, under the General Tab, enable the checkbox Enable HTTPS Inspection
   
   
   
   
4. Select the Use Forefront TMG to generate a certificate and then click on the Generate button to generate a certificate for TMG, which will be as a handshake between the server and its clients to decrypt the encrypted channel and inspect it.
   
   
   
   
5. In the Generate Certificate page, you can change the issue name to whatever you want, for example TMG Server, as for the expiration, its up to you if you want the certificate to be expired on a specific date, or to Never. Keep the issuer statement textbox empty. Then click the Generate Certificate Now
   
   
   
   
6. The generated certificate will be opened, and is currently installed on the TMG Server, don't click on Install Certificate... ,  Click OK
   
   
   
   
7. To configure the deployment options for the generated certificate, click on the HTTPS Inspection Trusted Root CA Certificate Options
   
   
   
   As you can see, there are two options to deploy the generated certificate, either Automatically through Active Directory (Recommended) or Manually on each client computer. If you have an active directory, then keep the default option selected. This will make the certificate be installed by active directory saving administrators the overhead of manual deployment. The certificate has to be installed with a domain administrator privilege.  Click on the Domain Administrator Credentials, type the username then click OK
   
   
   
   
   
   
   
   Else, if you don't have Active Directory, then select the Manually on each client computer, export the certificate, and then go physically to computer by computer and install the exported certificate.
   
   To see the generated certificate, click on the View Certificate Details... button and you will see the certificate that was just created moments ago.
   
   Once you set your options, click OK to go back to the HTTPS Outbound Inspection window.
   
   We have now enabled HTTPS Inspection, created a certificate and have selected the method for the certificate deployment. What remains is, fine tuning it to meet your requirement.

8. Click on Source Exceptions Tab. Under this tab, you can exclude clients from HTTPS inspection, for example, your boss don't wish his https connections to be inspected. So you will need to add his computer IP. Click on the Add... button to add whoever you want to exclude from HTTPS inspection.
   
   
   
9. When done with the Source exceptions, click on the Destination Exceptions, where in this tab, you can exempt destination from being inspected, and by clicking on the Validation button, you can select whether the https certificate for these exempted sites should be validated or not. By default, some Microsoft websites are exempted from being inspected, you can add any of Microsoft website to this Domain Name Set.
   Also you can add your own website to be excluded from being inspected.
   
   
   
10. Under the Certificate Validation Tab, you can set the options for what to do when a certificate is expired, for server certificates that are not yet valid and for server certificate revocation. make any changes, if required.
    
    
    
    
11. As for the Client Notification Tab, its better to enable the option Notify users that their HTTPs traffic is being inspected. Check with your corporate policy, as HTTPS Inspection might have legal implications, and so users should be notified that their traffic is being inspected.
    
    
    
    
12. Once all of this is done, click on the OK button, the HTTPS Outbound Inspection windows will close, make sure that you click the Apply button, so that changes take effect.
    
    
    
    
    as you can see, now HTTPS Inspection is enabled.
    
    
    
    
    

Summary

HTTPS Inspection is one of the greatest new feature in TMG 2010. If you have a active directory domain, then the deployment of the HTTPS Inspection Certificate can be deployed automatically via Active Directory. The options available with HTTPS Inspection, gives the administrator the freedom to exclude either sources or Destinations from HTTPS Inspection.

How to Configure DNS Conditional Forwarders

Conditional Forwarders was a new feature within the Microsoft DNS server for Windows Server 2003.  It was a great addition that allowed you to specify a specific DNS server for clients trying to resolve hosts in a specific domain.  This way you could tell the DNS server to always go to a specific DNS server for specific domain names.
One of the things that you will find different in Windows Server 2008’s DNS is how it displays Conditional Forwarders.  Previously you needed to view the Forwarders tab in the DNS server’s properties. Here is shot of the old way.
 
The new way is in plain site…but it still seems like people miss it.

Now here is a look at what type of options you have for it:

You just place the DNS domain name in the top section and the IP address of the DNS server that is authoritative for that domain below.  Notice you also can store this Conditional Forwarder in Active Directory if you want.  It is not  the default.  Behind that drop down is the amount of time the DNS server will wait before it times out…which is 5 seconds by default.
Hopefully that helps you figure out how DNS Conditional Forwarders are set up in Windows Server 2008

Configuring outbound access rules

We’ll begin by logging on to our TMG console, and browsing down to the firewall policy.

• Right-click Firewall Policy, scroll down to New, and over to Access Rule.
  

• As with most things Microsoft, this will launch a wizard. Our first step is to name our access rule. I like short, descriptive names, and I prefer CamelCase to make things readable and easier to script later, so I will call this AllowAllOutbound and then click Next.
   

• The default action is to deny. As we’re looking to permit all, we need to click Allow, and then click next.
  

• The next screen is where we need to define the traffic for this rule. Again, our choice is pretty easy based on our goal, “All outbound traffic” and then we click Next.
  

• The next step has us decide whether or not to enable malware inspection for this rule. This may be a bit confusing, as we’re dealing with an outgoing rule, but the help file defines this as “Outbound inspection refers to HTTP requests that originate from clients on networks protected by Forefront TMG.” This is not going to inspect the replies. So if you want to be a good citizen of the tubes, and you acknowledge that your client might have picked up the e-flu somewhere along the way, enabling this could protect the rest of the world from an infected client. We’ll tick the Enable option and then click Next.
  

• Now we need to define the source of this outgoing traffic we’re allowing. Assuming you want to cover all your users, click Add, expand Network Sets, and choose All Protected Networks, which will include your TMG server(s), your internal clients, and your VPN clients. Click to add them, then click Close, then click Next.
  

• In this step, we’re defining just where we want to permit this outgoing traffic. Since we’re trying to implement ip any, click Add, expand Network Sets, and choose All Networks (and Local Host.) Add them, click Close, and then click Next.
  

• Since we want this to apply to all users, click Next.
  

configure Forefront TMG to block AD users from accessing internal resources

The secure socket tunneling protocol (SSTP) allows Web users authenticated by the Forefront UAG portal to access the published remote network. You can use Forefront TMG on UAG to configure who has access to what over SSTP VPN. In this example, we’ll block a specific user/group from accessing the entire Internal network on all protocols. You can also select specific protocols to block.
Note: You have to have a working AD with the previously defined users/groups to who you want to deny SSTP services connection.
Procedure:
1. Open the Forefront TMG Management console:
2. Right-click the Firewall Policy mode, select New, select Access Rule.
3. Give the rule a name, such as SSTP Block, and click Next.
4. On the Rule Action page, select Deny, and click Next. When rule conditions are met, access will be denied.
5. On the Protocols page, select All Outbound traffic, and click Next.
Note: This is the point in the procedure where you can choose a more granular approach – specify protocols you want to block.
6. On the Access Rule Sources page, click Add, and from Networks, select VPN Clients.

7. Click Next.
8. On the Access Rule Destinations page, click Add, and select Internal from the list of Networks, and click Next.
9. On the User Sets page shown below we’ll actually configure what we’ve set out to do – block specific users. Select All Users and click Remove.

10. Click Add, and on the Add Users dialog box shown below, select New. This kicks off the New User Set wizard. Name the new user set SSTP Deny.

11. On the Users page, click Add, and select Windows users and groups.

12. Browse to the right user/group, confirm it with the picker dialog, and close all of the dialogs. Click Next and complete the New User Set wizard.
13. Back in the New Access Rule wizard User Sets page, you can select the new user set, click Next and Finish to finalize the rule. You’ll also have to click Apply in the Forefront TMG console.
Remember, Forefront TMG rules are ordered, so if this rule is not near the top and another rule has these conditions and allows access, the request won’t even get to this rule denying access. So make sure this rule is at or near the top. You can read more about Forefront TMG access rules on TechNet.
You're done!

Implementation of Client Security

 

NLB with Forefront

If you plan to load balance internal Web Servers with the Forefront TMG Web Server Farm Load Balancing feature, you should also keep in mind that Forefront TMG Server might be the single point of failure (SPOF) when TMG is not load balanced. Forefront TMG Enterprise uses NLB to load balance TMG Server. It is possible to use NLB in integrated mode, the preferred and recommended mode in Forefront TMG. It is also possible to use NLB with Forefront TMGStandard but this is not officially supported by Microsoft and has some limitations.

Load balancing mechanism

Round-robin
Webserver requests from different IP addresses will be distributed among the Web farm members. The round-robin mechanism ensures that the request of the user to a Web application serviced by a Web farm is distributed evenly among farm members that are online. When failover occurs, servers that are not responding are detected, and the load is distributed among the available servers.
Cookie based affinity
Session (Cookie) based affinity is normally used to publish Outlook Web Access (OWA) from Exchange Server 200x or Microsoft SharePoint services/Servers sites. You should not use Session affinity if you want to publish RPC over HTTP(S) services or Outlook Anywhere in Exchange Server 2007 and higher. RPC over HTTP(S) is used to give Outlook clients full access to Exchange Server from the Internet. RPC traffic will be tunneled through HTTPS. With Outlook it is not possible to use Cookie based affinity.
IP affinity
With IP affinity, the web server traffic is distributed based on IP to all members of the Web farm. If one Server fails to respond, the traffic will be sent to another member of the Web farm.
You should not use IP based affinity, if remote clients are located behind a NAT server, because the web server farm will only see the IP address of the TMG Server. If this is the case you should use Session affinity, if it is possible.
IP affinity is useful in an Exchange RPC over HTTP(S) also called Outlook Anywhere scenario, where session affinity cannot be used or, in Exchange Active Sync publishing scenarios, where the client does not fully understand HTTP 1.1 (which is needed for cookie based affinity).
To create a webserver load balancing publishing rule start the TMG management console and navigate to the Firewall policy node and create a Web Site Publishing rule.

Figure 2: Start the Web publishing wizard
Name the new policy rule and allow the traffic.
Click publish a server farm of load balanced Web servers

Figure 3: Publish a server farm
Because we are publishing an internal Web server without HTTPS, specify the appropriate option.

Figure 4: Use only HTTP
Enter the internal Site name and specify a path if you want to publish the web server only to a specific path.
As the next step create a new Farm, enter the name of the farm and add the internal web servers to the Web Server farm, as you can see in the following screenshot, and specify how Forefront TMG should load balance incoming web requests.

Figure 5: Specify Farm member
Forefront TMG creates a connection verifier to monitor the availability of the farm members. If one server is not reachable an alert will be created. You can customize the alert actions and I will show you later how to do this.

Figure 6: Connection verifier
A new popup window appears and will ask you if you want to activate a system policy rule to allow HTTP requests from Forefront TMG to the published web servers. Click Yes if you want to do this.

Figure 7: System policy rule
The next step is to create the specify Listener which Forefront TMG uses to listen to incoming traffic. Because my article only focuses the Web Server farm load balancing functionality, I will only give you some more information when you publish a web server over HTTP.
Forefront TMG warns the user that the current configuration may be unsecure when authentication requests are sent over HTTP.

Figure 8: Allow a system policy rule
To allow client authentication over HTTP you must allow this in the Advanced Authentication options window in the Listener properties as you can see in the following screenshot.

Figure 9: Allow client Authentication over HTTP
After the Webserver publishing rule has been created, navigate to the properties of the rule and click the Web Farm tab to verify the correct configuration.

Figure 10: Web Farm properties

Monitoring Web Server Farm Status

If you want to know which web server farm member is available or unavailable, Forefront TMG automatically creates connection verifiers when you create the Web Server farm. A connection verifier detects the status of farm member and reports this event to the alert configuration in TMG Server, which creates notifications like e-mail messages, entries in the event log and more.
Servers in a web server farm can have five different states:
Active
This is the normal state of a web server in the farm and indicates that the server is reachable and able to accept requests.
Out-of-service
This state indicates that the web server didn’t respond to the connection verifier within the specified timeout. No requests are sent to this farm member.
Draining
This state indicates that the web server is in the process of being drained. Existing connections will be finished but new requests will not be sent to this server. This feature is useful if you want to place one Server of the Web Server Farm in maintenance mode.
Removed
This state indicates that the web server has been removed from the farm, and is not accepting requests.
Unable to verify
This indicates that the server state cannot be verified.

Web Server maintenance

If you want to place one web server manually into maintenance mode navigate to the Servers tab, select the server and click the Drain button to place the server in maintenance mode so that Forefront TMG knows that this node is not available for load balancing requests. For session based affinity, the server will continue to handle current sessions but will not accept new connections. If you are using IP based affinity a drained server stops receiving requests, but existing connections to that server are still handled.

Figure 11: Servers included in the Farm

Alert actions

To configure alert actions when the Web server farm servers are not available, navigate to the monitoring node and in the task pane select Alerts properties and specify the actions you want to perform when a Server in the Web farm is unavailable.

Figure 12: Web Farm monitoring and alerting