Azure Active Directory Application Proxy installation and troubleshooting

11225654646_7fc9621cc9_bRecently we decided to migrate away from our legacy reverse-proxy product to something that would integrate better with our AD \ Office 365 systems. I’ve wanted to try out Azure AD Application Proxy for a while since seeing it in beta last year so this seemed a good time to get to grips with it. This post outlines a few gotchas to watch out for and some useful background reading.

Let’s start off with the initial Microsoft documentation available here

https://docs.microsoft.com/en-us/azure/active-directory/active-directory-application-proxy-get-started

Education freebies

Although Microsoft’s recent price hikes haven’t come at a good time for us in education we do get a lot of extras thrown into our Microsoft licensing agreement. One of the lesser-known ones is Azure AD Basic, which is the minimum requirement to use Azure AD Application Proxy – see comparison chart at https://www.microsoft.com/en-cy/cloud-platform/azure-active-directory-features for more info

To get your free licenses you’ll need to get in contact with your EES reseller and they’ll get them added to your tenant in a similar way to Office 365.

Applying the Azure AD Basic license is nice and simple, go to your Azure Management portal at https://manage.windowsazure.com, select your Azure AD directory then assign suitable groups to the license. What’s handy is that if you’re using Azure AD Connect to sync from your on-prem directory any new users will get automatically licensed as they come on board.

Installation

Next step in the documentation list is here:

https://docs.microsoft.com/en-us/azure/active-directory/active-directory-application-proxy-enable

I used two dedicated Server 2012 R2 VMs for our install, the connector will be installed on each so we have failover should it be required at some point. Enabling the Application Proxy in Azure is nothing more than one click in the portal

Now in theory the installation should be straightforward, nothing more than downloading the installer from the link, sign in with admin credentials and job done. However if everything went that smoothly this blog wouldn’t exist (!)

Troubleshooting 403 Forbidden errors

At the end of the installation the wizard helpfully offers to run a troubleshooter to check all is well but in fact all was far from well…

Checking Event Viewer threw up the following errors:

  • Event ID 32012
    The Connector update using the update service failed: ‘The remote server returned an error: (403) Forbidden.’. Check your firewall settings.
  • Event ID 12020
    The Connector was unable to connect to the service due to networking issues. The Connector tried to access the following URL: ‘https://***GUID***.bootstrap.msappproxy.net:8080/’

Outbound firewall settings were already configured to allow all the ports that were asked for in the documentation, proxy was disabled in Connection Settings and the firewall didn’t register any outbound traffic being blocked so what’s going on here? The mystery deepens…

Although the wizard only offers to run the troubleshooter once you can run it again manually by launching it from:

C:\Program Files\Microsoft AAD App Proxy Connector\ConnectorTroubleshooterLauncher.exe

Troubleshooting the troubleshooter

Although there’s a fair bit of documentation in the troubleshooting section on Microsoft’s pages none of it referred to this particular error. Google didn’t have much to go on either but did throw up some useful and detailed slides from the Ignite conference that are well worth a read:

Ref: https://channel9.msdn.com/Events/Ignite/2015/BRK3864
Ref: https://techcommunity.microsoft.com/t5/Microsoft-Ignite-Content/BRK3139-Throw-away-your-DMZ-Azure-Active-Directory-Application/td-p/10675

The second link references another useful document aimed purely at troubleshooting:

Ref: http://aka.ms/proxytshootpaper

Whilst searching I stumbled across an email contact for the Microsoft Azure AD Application Proxy team

aadapfeedback@microsoft.com 

so I dropped them a message with the errors I was encountering. The team replied almost instantly and initially suggested ensuring that the following updates were applied on the server:

https://support.microsoft.com/en-us/kb/2973337
https://support.microsoft.com/en-us/kb/2975719

Proxy proxy proxy!

However still no joy even with everything present as it should be. The next recommendation was to check if I was using a proxy server for outbound connections. We do have one but it’s not used for server VLANs and is the first thing I disable on a new VM build.

However I did get intrigued to check the traffic going out via TCPView… lo and behold there was the proxy server trying to take the outbound connections and failing miserably. It seems that despite everything in the operating system suggesting traffic should be going out directly the Connector was still trying to use the proxy route instead.

Ref: https://blogs.technet.microsoft.com/applicationproxyblog/2016/03/07/working-with-existing-on-prem-proxy-servers-configuration-considerations-for-your-connectors/

The solution is in this document under the section “Bypassing outbound proxies”, which basically involves adding these lines to the .config files for both Connector and Updater services

<system.net>

<defaultProxy enabled="false"></defaultProxy>

</system.net>

Checking Event Viewer and the Azure Portal afterwards showed success, my Connectors were now up and running with nice green icons, much better 🙂

Note: even though this fix resolves the issue the current version of the Troubleshooter doesn’t seem to follow the settings in the .config files and will still report connection failures. The Azure AD Application Proxy team are aware of this and are aiming to have a new version out soon.

Additional considerations

There’s a few other points to bear in mind when you’re completing the configuration of the application proxy. None of them are major issues but good to have everything ready before you start…

Certificates

Once the Connectors are up and running the rest of the process went smoothly, although note you will need a wildcard certificate if you want to publish your applications via a “vanity” URL i.e. your own domain rather than “msappproxy.net”

Using the vanity domain and some DNS CNAME records means that if you use Office 365 SharePoint for your Intranet your internal applications can work from the same URL both inside and outside.

Setting SPNs for Kerberos SSO

Even better, those internal apps can SSO based on the Office 365 initial sign-on for a suitably slick user experience! This does require a bit more configuration with Kerberos delegation but it’s not too bad.

When setting the SPN records I remembered the gotcha from when I worked on Dynamics CRM to type the command in manually… bizarre as it is the same still applies!

Using the -S switch worked well for me:

setspn -s HTTP/yourserver yourserver

Ref: https://blogs.msdn.microsoft.com/saurabh_singh/2009/01/08/new-features-in-setspn-exe-on-windows-server-2008/

Nested groups

Finally, bear in mind if you’re using groups created natively in Azure AD you can’t nest memberships when creating application assignments, which is a shame. As a workaround create any nested ones in your local AD instead and sync them up via Azure AD Connect or just create flat groups in Azure AD if you prefer to work solely up there.

Ref: https://docs.microsoft.com/en-us/azure/active-directory/active-directory-accessmanagement-manage-groups

Application links

You can either publish your application links via your Intranet or users can browse them via the portal (I’ve linked to the new makeover version as it looks much better than the previous one in my opinion)

https://account.activedirectory.windowsazure.com/r#/applications

image credit Rainer Stropek 

Advertisements

Attack of the Chromes: a Google Apps adventure begins

dell-chromebook-11
I’ve been watching the growth of Google’s Chrome OS for some time now – scarily about 4 years have gone by since they first appeared on the radar.

Recently I had the chance to get some in to work with first-hand as part of our STEM Centre project. It’s a new modern learning space and part of that vision involves the effective use of mobile devices.

Dell Chromebook 11 ready for action

Why Chromebooks?

There’s no denying the price point and simplicity of the Chromebook model so even though we’re an Office 365 site at present it would be foolish not to try the Google platform, especially with many of our courses already using online resources via Moodle and similar web platforms. This post covers our first steps along the way and a couple of tips and tricks to get you started if you’re in a similar position 🙂

The Chromebook is an interesting product for education and one that’s been discussed at length over at the Edugeek forums. First revisions of the platform weren’t quite there but looking at Chrome OS now it’s a lot more mature in terms of both concept and implementation. The hardware available has also moved up a level in terms of performance and quality as manufacturers have perhaps shown more faith in the Chrome OS platform.

One interesting point from the past that still hasn’t quite been resolved is Android vs Chrome OS, as it stands still two separate products but with some interesting convergence ideas showing through. On one head there’s ARC Welder allowing Android apps to run in Chrome and then there’s the Microsoft Surface-inspired Pixel C hybrid that could be an effective vehicle for either platform. Just to add another option into the mix you can also now get touch-screen Chromebooks (!)

Which device?

After a video conference with our Google Account Manager to discuss the platform in more detail we decided to get a couple of different devices in on trial. This included the HP Chromebook 11 and 14, plus the Dell Chromebook 11. The latter is particularly interesting as it’s built with the education market in mind and should be a bit more robust in the long term, as proven by this teardown article that pitches the Dell device against the equivalent Acer model:

Ref: http://www.edlisten.com/2014/04/dell-11-vs-acer-720-chromebook.html

One word of advice that we were given is to pay the little bit extra for a 4GB RAM model to avoid performance issues when browsing media-rich sites and \ or using multiple tabs. Thus far the Dell Chromebook 11’s haven’t skipped a beat in use thus far so I’d agree with this recommendation.

Ref: http://thewirecutter.com/reviews/best-chromebook/

The HP G3 14″ is an interesting device for its larger screen size, although from reading around I had some performance doubts related to the ARM Tegra processor. We’re not yet sure if 11″ is enough screen estate for students to work comfortably but the pilot projects will give us some feedback on that front. A 13″ device would be ideal, Dell are launching one but it looks somewhat more expensive than the 11″ version and aimed more at business customers.

In the end we bought in 6 Dell Chromebook 11’s from Haptic Networks to use in the STEM centre alongside a parallel trial of Microsoft Surface 3 tablets (more on that another time).

Thus far the people I’ve shown them to have been impressed by their lightweight sturdy build and solid keyboard (something that’s not quite up to the same standard on the HP devices). Battery life also looks very promising.

The only gripe I have with the Dell units is the fact someone, in their infinite wisdom decided to place a grinning lizard as a non-configurable logon screen wallpaper. There’s currently no way to change it from the Chrome admin console and it seems Dell aren’t too bothered about providing a solution either. I’m just glad the logon box covers up most of the image but even so… a lizard… why?!

Getting started tips and tricks

Although Chromebooks are pretty simple to get up and running using the online management portal there’s a couple of tips I’ll share from my initial experiences

Update in Guest Mode before doing anything else

Although our batch of Dells arrived in one shipment they all had different versions of Chrome OS installed where they’d been produced at different times. The visual differences are subtle but noticeable when all running side by side.

The quickest way I’ve found to get them up to date is to log in using the Guest mode option, preferably on a direct Internet connection then navigate to this URL in the browser:

chrome://chrome

Of course you can do this through the menu but this is so much quicker than pointing and clicking 😉
On two of our devices with the oldest out-the-box OS versions the first update run didn’t get them up to the newest Chrome OS so you may need to repeat the process.

Retrieving device MAC addresses

You may need to retrieve the MAC address of the devices for your Wi-Fi system or asset management records. The GUI way of doing this is a bit click-heavy and requires you to be connected to a network first. Alternatively you can do it a quicker way:

  1. in the browser navigate to chrome://system
  2. do a CTRL+F on the page and search for ifconfig
  3. the MAC address is listed under HWaddr

Resetting a device ready for enrolment

If you receive your Chromebooks before having completed your Google Apps registration it can be tempting to sign into the Chromebook with a consumer Google Account to try them out. This works OK until you then try to enrol the Chromebook as a managed device, at which point it promptly fails as per Google’s documentation below. The KB article also explains how to completely wipe the device using Developer Mode so you can repeat the out-of-box setup process.

Ref: https://support.google.com/chrome/a/answer/1360642?hl=en

Enrolling a device is simple using a keyboard shortcut that will soon become muscle memory CTRL + ALT + E

Ref: https://support.google.com/chrome/a/answer/1360534

Proxy problems

unnamedA page about cloud services wouldn’t be complete without a proxy-related caveat and the Chromebooks are no exception. Initially none of them connected after they switched networks from the direct-connected (NAT) temporary SSID I’d used for setup to the proxied one defined in the policy manager.

My first thought was to switch from Auto Detection using WPAD to a manually-specified proxy address. That change worked wonders almost immediately and I soon had login screens instead of connection failed errors… apart from two devices…

After putting all 6 Chromebooks in a line and rebooting them at the same time I soon found the same two devices failed every time. As far as I knew everything had been updated so it didn’t initially make sense why two were behaving differently from the rest.

After double-checking the Chrome OS versions it then became apparent two hadn’t fully updated and were sitting on Chrome OS 45.x instead of the new 46.x release. After moving them back onto the direct connection for another round of updates they then started behaving.

Moral of the story comes back to my first bit of advice: update, update, update!

It’s also worth running the Chrome Connectivity Diagnostics app on your devices if you suspect any network-related issues:

Ref: https://chrome.google.com/webstore/detail/chrome-connectivity-diagn/eemlkeanncmjljgehlbplemhmdmalhdc?hl=en

Next steps

Now the devices are up and running we need to start provisioning users into the Google Apps tenancy. For that we’ll need to install and configure GADS and GAPS to sync users and passwords from Active Directory. For now Gmail has been turned off until a decision is made about where email lives in future (as it’s not something we’d change part-way through an academic year).

Now just a matter of waiting for some initial user feedback to see how they get on with the Chromebooks and in what contexts they become an effective learning tool 🙂

Random error of the day – Office 365 OWA URL change

7612.Office-365-logo_thumb_58DAF1E4

This post is another heads-up for Office 365 admins behind proxy servers as we had an odd issue last week that’s an easy fix but wasn’t so obvious to spot…

OWA loading error

In Chrome some of our users were reporting that Chrome returned an ERR_CONNECTION_RESET page when trying to load Outlook from the Office 365 apps tray. It only affected a couple of users and the link was working OK in Internet Explorer.

After a bit of playing with various ways to get to OWA I found that if we removed the end of the URL and simplified it to just the “outlook” subdomain everything loaded OK… strange.

We logged a call with our Bloxx proxy support and they responded swiftly, taking some Wireshark captures on both proxy and client. Their analysis helped pinpoint the cause of the issue…

The captures showed that the client did initially talk to the proxy but then decided to continue the next part of the connection directly, which was then blocked by outbound rules on our firewall. In our WPAD file we have a wildcard entry towards the end that returns DIRECT for any in-house servers and it seemed that the “realm=havering-college.ac.uk” part of the OWA URL in the image above was matching the rule and thus trying to bypass the proxy.

What’s changed?

Although the theory made sense on one hand it didn’t on another as we’d never had these issues before. Looking around for what’s changed in our environment put a few culprits in the frame; Google Chrome updates and the Office 365 platform itself.

At this point I went back on my VM, which was now exhibiting the same issue. At one point I had the WPAD file open and the failed web page side by side… then the lightbulb moment!

On one screen I had OWA open with a domain starting https://outlook.office365.com but on my VM it was trying to load https://outlook.office.com – that’s new!

Going back into WPAD I then added another explicit traffic entry:

   /* Send Outlook OWA traffic via proxy here*/
        if (shExpMatch(url, "*outlook.office.com*"))
      { return "PROXY X.X.X.X:8080";}

Immediately OWA started working! It’s bizarre how some applications seem to need these rules in order to route traffic using Auto Detect but it seems to consistently fix issues when they pop up. As for the OWA URL, it seems to be a bit random at present which one gets used, although both are listed in the Office 365 URLs and IP ranges KB article so maybe Microsoft are in the middle of changing them over?

Cloud stories: Groupwise to Office 365 (part 1)

office 365 cloud

I’ve been meaning to start this series of posts for a while, documenting our journey as we move various legacy on-premise systems to Office 365 and all the fun and games we’ve had along the way. Like many email was the first item on the migration hit-list and indeed was a key driver for investigating moving services into the cloud.

Unlike many organisations we weren’t moving from Exchange, our on-premise system was actually Novell Groupwise. The large increase in storage space was the biggest draw for us to go cloud but mobile access also played a part. Novell were very short sighted in not providing a (free) mobile app for Groupwise as the majority of users weren’t willing to mess around with manual IMAP settings nor were they particularly enamoured with having to pay for a 3rd party app to read email on their phones, which these days is a bread-and-butter feature requirement for any enterprise-grade product (imo).

Although moving to a cloud-based service is often marketed as a click-and-forget experience it’s not entirely accurate unless you’re setting up from scratch. This post covers a few lessons we learnt and some tips to help you avoid them…

Firewall Rules

We use a combination of standard firewall and proxy web filter to direct traffic in and out to the Internet at large, a pretty common setup for an organisation of our size with ~2500 workstations.

1401412783_firewallHowever we soon found out that this wasn’t as simple as first thought. Although our firewall can identify traffic based on application some of the login processes for the Outlook client were only identified as vanilla SSL, I guess some of the Autodiscover processes may be the reason for this.

At first we thought the solution might be fairly simple, at my last place I used Microsoft’s Forefront Online Protection for Exchange (FOPE) which gave a defined list of IP ranges to allow mail from so was hoping Office 365 would be similar… we were wrong!

It started well enough, Microsoft gives what looks like a helpful list of addresses to allow here…

http://onlinehelp.microsoft.com/en-gb/office365-enterprises/hh373144.aspx

We dutifully added all of them to the firewall then watched as Outlook played it’s own game of Russian Roulette with logins, some would work and others would fail miserably with messages like the example below about an encrypted connection being unavailable. Trying to continue with the unencrypted version also failed.

outlookwarning01
The next logical step was to check the firewall logs and see what was going on, lo and behold there was a bunch of blocked traffic from the affected machine, type listed as SSL (in addition to the Office 365 traffic that was allowed by our rule). At first glance the listed IPs looked pretty similar to some of the ones on the Microsoft link so I punched them into the very helpful CentralOps domain dossier to do a bit of detective work into what they really were.

Surprise! The IPs were owned by Microsoft, One Redmond way. Over the course of a couple of days I’d collected (and kept adding) these new address ranges but new ones kept coming. After a while another surprise was that some were listed as being owned by Akamai, which is the CDN Microsoft use to deliver content for the Office 365 web interface. We expected that for OWA but not for the desktop client but there it was clear as day in the logs.

In the end we decided that there was no way we were going to be able to keep up with how rapidly the addresses were changing and give our users a decent experience (our pilot users didn’t have a fun time getting set up with Outlook) so we were forced to push all Office 365 traffic trough our proxy server instead. We didn’t want to do this if we could help it as our firewall is much faster at processing large amounts of traffic than the proxy – no choice for now but thus far it’s stood up to the load pretty well.

A well-written TechNet article explains how Microsoft have built the CDN network for Office 365 and why they’re only supporting wildcard URL rules for filtering…

http://blogs.technet.com/b/exchange/archive/2013/12/02/office-365-url-based-filtering-is-just-better-and-easier-to-sustain.aspx

…which is great but until the firewall vendors catch up and put it in their core feature sets many network admins won’t be happy. The issue crops up more when using the desktop Outlook client but that’s a fundamental reason for choosing Microsoft over Google so a workaround had to be found to give a reliable first-run experience.

Proxy issues with Outlook

Now that we had the basic traffic issues on the way we thought Outlook would kick into life quite easily but there was one more sting in the tail. Autodiscover still wasn’t working reliably despite having proxy authentication disabled for all Office 365 domains (seems a common recommendation from various proxy suppliers) so we needed to dig a bit deeper.

We ran the Outlook first-run wizard with TCPView running alongside and noticed something odd…

Our proxy settings were pushed out via a GPO setting, along with a manual exceptions list for various IP ranges and internal sites. Because our internal domain is the same as our external one we added autodiscover.havering-college.ac.uk to the list of proxy exceptions (and created an internal DNS record) so we expected to see connections going out via the proxy server address… except we didn’t.

Bizarrely it seemed Outlook was ignoring the proxy server completely and trying to connect directly – which on a standard user’s machine won’t work. Cue some furious Googling which showed that it wasn’t just us experiencing the same problem…

http://social.technet.microsoft.com/Forums/ie/en-US/e696adc8-e05d-42a0-bc95-f3c48b04bef3/bypass-proxy-for-local-addresses-office-365-connection-outlook-2010?forum=ieitprocurrentver

The only solution was to create a proxy auto-configuration file that explicitly says to use the proxy for the Office 365 URLs. As soon as we did this and switched the GPO over to Auto-Detect Settings Outlook kicked into life. I’m not sure if this issue has been sorted in Office 2013 SP1 but it’s not a listed fix and recent posts in the thread above suggest it’s still an unresolved issue.

Next-time
Next time

Next post in this mini-series will cover some decisions you’ll need to make about what version of Office you use on your machines and how it can affect your 365 experience.

Dropbox and WPAD auto proxy detection

dropbox-1We recently changed our proxy configuration method from manual settings to auto detect to fix some issues we were having with an Outlook bug affecting Office 365 Autodiscover, unfortunately around the same time Dropbox seemed to stop working.

After doing some testing I found that I could do either of the following to get it to work using version 2.6.2 from the Dropbox website:

  1. Set proxy manually in Windows, Dropbox set to auto
  2. Leave proxy as auto detect in Windows, set Dropbox manually

In the latter situation a valid login has to be made after the proxy configuration otherwise Dropbox will reset itself back to default (auto) settings and any further logons will fail.

A bit of research led me to the Dropbox forums where it turns out there’s another build available with new features that haven’t yet made it to the stable branch. Version 2.7.38 contains WPAD support. The Dropbox staff on there are helpful and replied quickly which is always good to see if you have any other queries.

I downloaded the slightly updated version 2.7.39 and deployed it via ZENWorks using the /S switch to make the installation (mostly) silent. Users still have to accept a UAC prompt which is a pain but can’t be helped. Logging in with the updated version brings up a login screen straight away with Windows and Dropbox proxy set to auto and we’re away 🙂

We also add an explicit entry in our WPAD file to ensure Dropbox goes via the proxy, it might not be absolutely necessary but removes all doubt about where the traffic is going which can never be a bad thing (replace X.X.X.X with your proxy server IP address)

/* Exclude Dropbox from proxy here */
	if (shExpMatch(url, "*dropbox.com*"))
	{ return "PROXY X.X.X.X:8080";}