Save yourself from insanity… Google and Outlook contacts on Android

2000px-Android_dance.svgRecently I had to factory reset my HTC One M8 whilst it was in for a repair (thanks to a stray bottle of soy sauce landing square on the screen, ouch!) but since reinstalling all my apps I noticed my contacts sync wasn’t working correctly.

Although my Google account had synced contacts when first setting things up the People app would’t let me add a new contact to my Google account. Rather it would default to SIM instead. Very strange I thought, it’s never done that before and I could still see everything else that was already there. Oddly the filter menu wouldn’t list “Google” as an option either.

Initial thoughts

First I thought maybe the app permissions after the Android M update may have gone wonky so checked those, no problems there (People app had access to Contacts permission).

Next… maybe the Google Account sync had Contacts sync turned off but after checking it’s there and working fine.
As another test I created a new contact online via Google Contacts and then forced a sync on the phone… contact didn’t appear. Very odd.

Tried a few other ideas like clearing App caches, also cleared the Android cache partition via Recovery as I’d been having some issues with the HTC Camera app as well but no joy there either (although Camera app now seems to have sorted itself out so a bit of a bonus there).

Solution – turn off Outlook contacts!

Finally I came across this…

http://forums.androidcentral.com/google-nexus-5/350303-phone-contacts-not-syncing-google-account-contacts-2.html

Credit to “haneyman” for this…

Confirmed, you cannot have Outlook sync contacts and expect Google contacts to sync. As soon as I unlinked the Outlook account on my phone, the Google contacts appeared.

So it seems the Outlook app is the culprit. To confirm I went into the sync options for my Outlook.com account and sure enough contacts sync was enabled. Turned that off, cleared my running apps then sure enough on next load the People app was letting me create and sync Google contacts again.

Maybe having accounts on both Google and Microsoft is a bit unusual but definitely one to watch out for if you have a foot in both camps and use an Android smartphone.

Field notes: OST cache, shared mailboxes and SSD drives

As we’ve been running all our staff and students on Office 365 for the best part of a year now we’ve found a few tweaks that may be of interest to a wider audience. Here’s one of them from experiences earlier in the week…

SSD vs HDD

hdd-154463_1280Like many of you out there all our recent machines have been specced with SSD drives as the performance difference is incredible (can you imagine going back to HDD now?!) but the downside being that the size of drive isn’t as large.

This has become less of an issue on newer builds as 120GB drives have dropped right down in price now but for the first-gen machines with 60GB drives we have hit some space issues, mainly due to…

Shared mailbox caching behaviour

In a previous post I’ve mentioned the hybrid cache in Outlook 2013 that makes working with large mailboxes much easier; however what’s hidden away in the small print is that the hybrid cache doesn’t apply to shared mailboxes or other peoples’ mailboxes that you have access to.

That has some knock-on effects that aren’t immediately apparent but have began to manifest themselves recently in a couple of ways:

Low disk space

We’ve had a few calls coming in recently with machines running out of disk space, an issue we’d pretty much consigned to the history books after being spoiled with 160GB+ drives being more than spacious for most generic desktops.

Upon running the old but incredibly useful WinDirStat tool we could see where the space had gone… OST files! The worst case thus far was 35GB in a single file but other machines have had numerous ~7GB files (multiply that by a factor of 3-4 on a machine used by multiple staff and you can soon see where the space goes)

Calendar entries not syncing

Another recent call involved staff responsible for managing other users’ calendars not seeing updates when new entries were added or moved, yet when viewing on another machine or via OWA there was no such problem. What seems to happen is as the OST grows it corrupts and eventually the sync behaviour becomes a bit erratic.

Emails stuck in Outbox

Similar to the calendar scenario above some users have had random emails getting stuck in their Outbox and refusing to send. This particular issue has occurred on smaller OSTs as well as the huge ones above so it seems to be a corruption issue that can pop up from time to time.

Resolution steps

The quickest way to fix a corrupted cache is just to delete it. If, however you don’t want to do that for some reason (slow connection, user doesn’t want to wait for cache to reload or has unsynchronised items) you can run the scanpst tool that’s included within Office. It’s not something you’ll find in the Start Menu so run it manually from:

C:\Program Files (x86)\Microsoft Office\Office15\scanpst.exe

Ref: https://support.office.com/en-za/article/Repair-Outlook-Data-Files-pst-and-ost-25663bc3-11ec-4412-86c4-60458afc5253

In the scenario above where emails were getting stuck in the Outbox scanpst resolved the issue without needing to delete and repopulate the cache so it’s worth a shot as a quick fix

Disable caching for additional mailboxes

Some of our users need to have 10+ additional mailboxes open, others have shared mailboxes with many attachments and have tended to be the ones hit hardest by the caching and disk space issues. If money was no object we could just get them all 250GB+ SSDs but seeing as that’s not the case we need plan b)

Ref: https://support.microsoft.com/en-us/kb/982697

The solution is to enable an Office 2013 GPO setting under User Configuration > Administrative Templates:

Outlook 2013 > Outlook Options > Delegates > Disable shared mail folder caching

Once this applies you’ll notice a change in the status bar along the bottom of Outlook, non-cached mailboxes will show up as “Online” (confirmed at the end of the KB article above)

status bar

The slight downside is a bit of lag when first opening the folders. For a secondary mailbox that isn’t used as regularly it’s an acceptable compromise, given the issues that users were experiencing with the oversized caches so we’ve rolled it out across the board.

The only thing we’ll have to wait and see is whether the large OSTs reduce in size or if they need deleting to remove the cache that was previously stored for the shared mailboxes.

 

Cloud stories: Groupwise to Office 365 (part 3)

office 365 cloudIt’s been a while since part 2 of our Groupwise to Office 365 story but without further ado here’s the promised part 3 of our migration story…

How much to migrate?

Before figuring out how to move the data the first thing was to decide what we wanted to migrate. Starting from the basics the first question was “do we need to take anything?”

Starting from ground zero and providing users with an empty Office 365 mailbox would’ve been the easiest method. However it’s also the least helpful for staff as they need access to historical data so that idea was soon dismissed.

The next question was whether we could take a subset of the most recent data and keep the rest in an archive. In the end we decided that would be just as much work as moving everything and it was soon obvious we’d need to migrate complete mailboxes. Even the largest ones were a fraction of the 50GB per user we get on Office 365 so no worries on the space front, just a matter of deciding how to do it.

Tools of the trade

My colleague investigated various migration tools but not many work with GroupWise, which narrowed down the field pretty quickly. We also didn’t want to go to the trouble of doing double migration going from GroupWise > an in-house Exchange server > Office 365 as some Microsoft documentation suggested as an option. Quest produce a migration product that seemed to be the cream of the crop (and also recommended by Microsoft) but it also had a price tag to match. As one of the reasons for moving away from GroupWise was to reduce costs paying through the nose for a migration tool seemed counter-intuitive, so we went for the next best option…

Kernel Office 365 Migrator for GroupWise

kernel logo

http://www.nucleustechnologies.com/groupwise-to-office365-migration.html

This tool made by Nucleus Technologies offered us the core functionality of moving GroupWise data directly into a fresh Office 365 mailbox but at a fraction of the cost of Quest, just what we needed! However it did need a bit of ingenuity to do the end-t0-end migration process.

The tool itself was pretty simple to use and worked well with our migration method of going team-by-team, 20 or so users at a time in batch mode (usually running overnight). Batch processing mode allowed us to set off a list of users to be migrated overnight and worked well. We did hit a couple of snags but soon ironed them out:

Missing items

Initially we found some issues with a couple of mailboxes where email and calendar items were missing. We soon narrowed this down to inconsistencies in the mailboxes themselves and found that running GWCHECK before the migration process ensured a complete migration of all data.

Slow migration speeds

Although our 100Mb JANET connection had plenty of spare capacity we found the upload speed to Office 365 was quite slow. Reading the Technet forums suggests Microsoft limit upload speed into Exchange Online so bear this in mind and allow plenty of time for mailboxes to migrate, esepcially for very large (2GB+) mailboxes. Most of our migrations were started at 5pm so they’d complete for the following morning. There’s some more detailed information on the throttle rates at http://technet.microsoft.com/library/dn592150(v=exchg.150).aspx

Close the old mailbox

After the migration is complete ensure that the GroupWise mailbox is closed so the user can no longer login to it. Although users were notified that their accounts were being migrated and to use Outlook at next logon some still used the old GroupWise client by mistake.

Coexistence and the need for customisation

With nearly 1000 mailboxes to migrate, an Office upgrade to deploy and hundreds of users to train there was no way that a “big bang” changeover would be feasible. As a result we needed to run Office 365 and GroupWise side-by-side for a couple of months while users were moved over.

Our workaround for this was to use a forwarding rule on migrated mailboxes, which allowed GroupWise to continue serving mail for users still on the existing system whilst sending migrated users’ mail to Office 365.

The only problem with this plan was that GroupWise doesn’t allow rules to be configured on the server side, in fact they have to be done from the client GUI. Doing this manually for each user was going to take a lot of time and effort, as well as opening up additional risk due to human input error. We needed a better way…

AutoIt to the rescue

autoitI’d done some work with AutoIt in the past to automate a couple of particularly awkward software installations – in this case its ability to script window controls made it the perfect fit.

Along with the main installer also grab the enhanced version of the SciTE editor (I personally find it nicer to work with).

Previously I’d used AutoIt in quite a basic way using the recorder tool to grab a series of keypresses. Although this method often works I wanted to make the process a bit more robust if possible.

Squirreled away in the AutoIt program folder is the very useful window info tool that can be used to find the exact ID of particular buttons in dialog boxes, as well as window titles (more on that in a moment)

Run it from C:\Program Files (x86)\AutoIt3\Au3Info.exe and drag the finder tool onto an open window.

I created the tool to accept two command line arguments for the username and password of the GroupWise user (our GroupWise accounts weren’t linked to eDirectory and wouldn’t be used again after migration so no problem resetting passwords on those).We also needed to reset the GroupWise client settings to ensure the GUI was in its original state for the AutoIt code to run as expected.

Ref: https://www.autoitscript.com/autoit3/docs/intro/running.htm

$CmdLine[0] ; Contains the total number of items in the array.
$CmdLine[1] ; The first parameter.
$CmdLine[2] ; The second parameter.
$username = $CmdLine[1]
$password = $CmdLine[2]

We also need to include the Function for the WinWaitActivate command. The AutoIt recorder tool sets this up itself so I created a blank script and grabbed the function from it.

Func _WinWaitActivate($title,$text,$timeout=0)
 WinWait($title,$text,$timeout)
 If Not WinActive($title,$text) Then WinActivate($title,$text)
 WinWaitActive($title,$text,$timeout)
EndFunc

I also set up some variables to make the GWIA email forwarding command that my colleague had researched:

$rulept1 = "web-mail.gwia:"
$rulept2 = "@ourdomain.onmicrosoft.com"

The next stepwas to run the GroupWise client (we built a dedicated Windows 7 32-bit VM so I knew the path would be OK). The last parameter ensures the client forgets the previously logged in user and always asks for credentials:

Run('"C:\Program Files\Novell\GroupWise\grpwise.exe" /@u-?')

Interestingly I still needed to add a mandatory 5 second load delay or the WinWaitActivate command didn’t work consistently. Reading back now using the WinWaitActivate function’s timeout option may have been a neater way of achieving the same result but either way it works as-is. A longer timeout value may be required on slower machines.

Sleep (5000)
_WinWaitActivate("Novell GroupWise Startup","")

Next part used the window info tool to find the IDs of the username and password fields then press Enter to login

ControlSetText("Novell GroupWise Startup","",1002,$username)
ControlSetText("Novell GroupWise Startup","",1003,$password)
Send("{ENTER}")

Once logged in use keystrokes simulating the keyboard shortcuts of the client to open the Rules page (Tools menu, Rules) then create a new rule called Forward to Office 365

_WinWaitActivate("Novell GroupWise - Mailbox","")
Sleep (10000)
Send("{ALTDOWN}t{ALTUP}")
_WinWaitActivate("Open the Address Book","")
Send("r")
_WinWaitActivate("Rules","")
Send("{ALTDOWN}n{ALTUP}")
_WinWaitActivate("New Rule","")
Send("{SHIFTDOWN}f{SHIFTUP}orward{SPACE}to{SPACE}{SHIFTDOWN}o{SHIFTUP}ffice{SPACE}365")
Send("{ALTDOWN}i{ALTUP}{ALTDOWN}a{ALTUP}{SPACE}m")
Send("{ENTER}")
_WinWaitActivate("Forward","")
Sleep (2000)

The code above creates a “Forward as Attachment” action for Received mail (use this to ensure the email gets to Office 365 in its original form).

Ref: https://www.novell.com/support/kb/doc.php?id=7002601

I then push the forwarding rule (made up of the variables outlined earlier) into the command text box, with the username parameter sandwiched in between then press enter.

Send($rulept1)
Send($username)
Send($rulept2)
Send("{ENTER}")

At this point we soon realised that many users had created their own rules, however our forwarding rule always needs to be at the top to be effective.

First we save the rule from earlier by clicking OK in a couple of dialog boxes:

ControlClick ( "Forward", "", 1)
_WinWaitActivate("New Rule","")
ControlClick ( "New Rule", "", 1)
_WinWaitActivate("Rules","")

The next step might make some programmers cry but a quick and easy solution was to press the “Up” button in the Rules GUI an arbitrary number of times (I chose 10) which forces the newly created rule up to the top. Repeat these two lines another 9 times 😉

Sleep (500)
Send("{ALTDOWN}u{ALTUP}")

When done exit the Rules window and exit GroupWise

ControlClick ( "Rules", "", 2)
_WinWaitActivate("Novell GroupWise - Mailbox","")
Send("{ALTDOWN}f{ALTUP}x")

At that point I compiled the code into an EXE then it was a case of either launching the tool from a command line e.g.

C:\forwardtool\forward.exe auser password123

forwardtool
Or for batch processing put the login details in a CSV file and launch from PowerShell. Using PowerShell for the CSV side saves re-inventing the wheel in AutoIt to do the same thing and lets me use the command in two different modes.

$list = Import-Csv "C:\inputfile.csv"
foreach($entry in $list) {
    $username= $entry.username
    $password = $entry.password
    
    $exe = "C:\forwardv4.exe"
    $params = $username, $password
    $process = [Diagnostics.Process]::Start($exe,$params)
    $process.WaitForExit()
}

I then sat back, very pleased and slightly amazed as it chewed through the accounts, saving us loads of manual effort and avoiding typo-related issues too 😀

Cloud stories: Groupwise to Office 365 (part 2)

office 365 cloudHaving sorted out the initial connectivity issues the next stage of the process was to make a decision on what version of the Outlook client (and therefore Office) to deploy to users’ machines. Until now the college had been running Office 2007 and there would need to be a strong reason to change staff machines midway through the academic year.

Outlook, Office 365 and cached profiles

In my previous experiences with Exchange I’d always disabled Cached Exchange Mode as it always seemed to cause more trouble than it was worth, especially with users that frequently moved between PCs. However due to the fact the Exchange server is now on a shared platform and more “distant” from the local network it becomes necessary to look at it again. Initially we tried running with Cached Mode disabled but performance wasn’t at an acceptable level; moving between folders caused a noticeable delay, as did searches and listing the contents of a large inbox.

With that in mind we knew Cached Mode was going to be mandatory, this is where things get interesting. In previous on-premise scenarios it mailbox limits are generally an order of magnitude smaller than the standard offering on Office 365, however with the standard offering now 50GB per user the effects on OST files could potentially be rather problematic. If a user moves machine the time taken to build the cache again could lead to a pretty painful user experience too!

All Outlook versions prior to 2013 are pretty much a blunt instrument in terms of how they deal with the cache. Fortunately Office 2013 comes to the rescue with its new Hybrid Cache feature. More information can be found at http://technet.microsoft.com/en-us/library/cc179067(v=office.15).aspx

cached mode
configure this either manually in the Outlook profile or via GPO \ OCT

What it basically does is to cache a smaller amount of email (from 3 months upwards) then offers the user a small “There are more items in this folder on the server” link for anything older, at which point Outlook grabs the required mail seamlessly from the Office 365 servers. It’s a best-of-both-worlds scenario and, in my opinion is a key reason to upgrade if using Office 365 with the desktop Outlook client.

That said even at the 3 months setting a reasonably sized mailbox can still need 1GB+ of data to be downloaded before Outlook is ready to use. As a result we’ve recommended our users to access email via OWA if they’re on a machine they’re not likely to use again any time soon.

Self-service Office upgrade

The cache alone pretty much made up our minds to go with Office 2013, the additional integration with OneDrive for Business \ SharePoint was also another factor that will become more apparent in the coming months once our SharePoint Intranet goes online too.

With the decision made we needed to find the simplest method to deploy the new version of Office to staff. With nearly 1000 staff machines all in use at different times it wasn’t something we could push out overnight. The team-by-team migration plan also meant that we couldn’t switch in a “big bang” method as Outlook and GroupWise don’t play nicely if both are active and fighting for control of MAPI profiles.

nalwin office upgradeThe solution was to make a self-service process that users could initiate at a time convenient to them. Generally it tended to be a lunch break on the day of their migration but equally it could be done at the end of the day etc. We used ZENWorks to push out a Bundle named Office 2013 upgrade which contained a customised MSP created with the Office Customisation Tool (OCT).

If you’ve not used it before basically you run setup.exe /admin then generate an MSP file which you place in the Updates folder within the Office 2013 installation media folder structure.

More info available at http://technet.microsoft.com/en-us/library/cc179097(v=office.15).aspx

Tip: remember to include the AUTO_ACTIVATE property while configuring your OCT deployment file as avoids users seeing any pesky pop-ups asking them to activate Office when they run it for the first time

Progress indicator

We also hit a bit of a problem of our own making due to the effects our Novell environment has on Folder Redirection. Our (until now) lack of Active Directory meant we had to use some unofficial local policy ADM templates to achieve a similar effect to the native Windows GPOs. The downside of this was that we had to map to a drive letter rather than the supported method of using UNC path. As a result Office setup bombed out when run as the logged-on user, even if using the Dynamic Local Admin option in ZENWorks.

office_HTA
the HTA loads full-screen and disables CTRL+ALT+DEL so setup doesn’t get interrupted

The workaround was to use the SYSTEM account to run the setup executable, however it meant that we lost any form of progress indicator to let the user know something was actually happening. I knocked up a quick HTA that effectively locks the workstation with a full-screen splash page informing the user to wait for the automatic reboot, along with some quick tips on how to set up Outlook on next logon.

Fixing setup errors

In testing I found that the upgrade would sometimes fail for no apparent reason. In line with some of the gems I’ve had from Windows 8 the error messages were about as much use as a chocolate teapot… “Microsoft Office Professional Plus 2013 encountered an error during setup” doesn’t really help much.

chcoloate teapot office error
unhelpful error message mandatory, teapot optional…

Digging through temporary files yields a setup log file that gives a more detailed insight into what went wrong, although in this case the failure was listed as a fairly non-specific “1603” error. The workaround listed on the Microsoft forums that recommends deleting a couple of folders from ProgramData seemed to work so I’ve included the folder delete \ rename actions as the first steps in the Bundle to be sure.

delete directory

Bundle requirements

Recently we noticed a few machines were failing the upgrade but initially couldn’t think why as they were all built from the same base Windows 7 image and Office 2007 installation. The error we kept getting was a 1605 which means “out of disk space”… oops! Turns out the affected machines were our 1st-gen SSD PCs that only had 60GB drives. Between the ZCM cache, Office install directory and other detritus on the local drive there wasn’t enough room to install Office.

Disk cleanup was the easy fix, along with a couple of requirements on the Bundle to check for disk space, as well as to only run the installer if Office 2013 wasn’t already installed

Tip: use HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\15.0 to check this

requirements

Outlook profile \ Launch actions

We experienced some issues when GroupWise and newer versions of Outlook were installed on the same machine, basically both programs were fighting for control of the default MAPI profile which made Outlook rather upset.

To get around this I added some additional Launch actions to our Outlook desktop shortcut to check for (and remove) any GroupWise MAPI profiles first. In older versions of Office profile information was stored in a hard-to-find location in the registry, fortunately in 2013 that’s changed for the better and you just need to look in:

HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Profiles\[profile name]

We used OCT to define that all new Outlook profiles are called Office365 by default, with that in mind I use the registry location above to check if a legitimate profile exists. If not we import a PRF file with the accounts section removed to ensure the first-run wizard appears as expected, the command to do this looks like

Command: ${ProgramFiles32}\Microsoft Office\Office15\OUTLOOK.EXE
Command Line Parameters: /importprf Z:\Office365.prf

Next-time
Next time

Next post in this mini-series will cover how we migrated 400GB+ of mail data from GroupWise to Office 365 with the help of some rather nifty scripting 🙂