Microsoft Flow first impressios

It's been an interesting week of follow-ups after last week's "The Future of SharePoint" event held at San Francisco. The world is excited about the many new (really new) functionalities and tools that were presented. Today I tested one of them - Microsoft Flow, which is still in Preview and is available to Office 365 FR tenants (or select FR people only).

Flow is similar in concept to IFTTT ('If This Then That'), but aimed at organizations rather than the individual, which means you can't get to it if you don't have a work or school account (Office 365).

Similar to Planner, the tool is not yet integrated into SharePoint, but feels like standing a little bit aside from it with its own domain: https://flow.microsoft.com.
It does, however suggest future tighter integration with Office 365 and SharePoint Online, as today I saw the Flow tile in the App launcher.

Going to the Flow website, very similar to Power BI and Planner, you need to Sign Up.

Once signed in, you can either browse the available Templates ("recipies ready to cook") which will allow you to connect about 35 services or you can start a new flow from scratch. In my example, I'd like to be emailed every time someone creates a new project in Project Online (which is the cloud brother of the newly released Project Server 2016.

Here you'd need to provide the URL for your PWA site collection, then you have just two basic options - "Add a condition" or "Add an Action" - just like the old school workflows for WSS 3.0.

That's understandable as Flow is designed to be simple in its core. So it's definitely not a full-blown Business Process Automation solution (like Nintex, for example) and the flows here are per user, so no centralized store and management of those.

When you choose "Add an action", you see the available hardcoded actions, sorted by the service they're related to. Send Email is under Office 365 Outlook. You'd need to sign in.

You can then configure the recipients, subject and body of the email - the basics. There are some predefined lookups to the current item that you can insert like Project Name, type, etc. That's cool, you just click on them and add them where appropriate.

Then all you need is a name for this new simple flow, you click Create Flow and Done.

On the next Project that gets created in Project Online, I get this:

Looks simple and clean to me. What are your thoughts? Have you signed up for Flow and tested it yet?

Migrating to Exchange Online (Part 1)

Yep, you got that right.

I volunteered and got involved  in a migration to Exchange Online these days and I thought it would be quick, nice and easy job to help the IT team in a mid-sized organization. Not even close!

The scope is the following:

3 IMAP e-mail solutions (5 domains, 3 geo-locations of the servers) -> Exchange Online

No migration tools and no external consultants will be involved in the migration.

The first thing you'd need to know when you plan and estimate your migration is speed. Depending on how important this is for you, you can choose the different options ahead.

Option 1: PST Import service 

Caveat: If your e-mail servers can't export a PST, then you need a 3rd party tool to do that.

Limitation: You can't import PST files larger than 10 GB or containing more than 1 000 000 items. So you might as well need a tool to split your largest mailboxes into multiple PST files.

1.1. Copy your PSTs to a secure Azure BLOB storage and then import them through the Admin Panel (you need to be a Global Admin)

We were considering to use the new Office 365 Import service launched last year (and still free, MS are planning to charge for it if you ship your drives) and we were very frustrated to find out the capping of 0.5 GB / hour when you import your PSTs. There's an additional prerequisite step to that, first you'd need to copy your PSTs in an Azure Blob, which for us worked as a charm - over 300 Mbps - and that's when the speed traffic police comes in and your happy face is gone :)

By speed traffic police I meant the Office 365 User Throttling - I still dunno how the PST import of 500 mailboxes, for instance could be considered user import. It's purely the migration team's job to export all PSTs, prepare PST mapping files and all that. I totally don't see it the same as if you'd import your data through Outlook - this is where there should the throttling apply, and not if these actions are associated from the Office 365 interface, but anyway... you do get the perks like calendars, tasks, contacts. It's never perfect.

Tip: If you run multiple PST import jobs at the same time - you get 0.5 GB / h for each of them.

1,2. Ship your hard drives to the nearest Microsoft data center and then import through the Admin Panel (still, you need to be a Global Admin)

This is usually the preferred method if you have slow bandwidth to Azure. On our tests, 300 Mbps were more than satisfactory and we don't use an ExpressRoute to Azure, This means that we can transfer 1 TB for ~13 hours. So in our particular scenario, shipping the hard drives would definitely take more than the copying to Azure, but if you have 10 TB of e-mail and slower connection to Azure, your choices are different.

If you still decide to go that dirt road (PST import in general), the official MS Import guide is here.

Option 2: IMAP migration

Obviously the only other option (I'm excluding user-handled migration through Outlook and all that, of course) is the IMAP migration. It comes in two flavors.

2.1. DIY e.g. get one of the many free 3rd party scripts out there and run it on your own environment - why would you do that?

This will get the source and target mailboxes in sync, but a big trouble with that is you need your source e-mail account passwords, and the destination (Office 365) passwords for the users as well. Not a big deal, unless your users are already in Office 365 and using stuff like SharePoint, OneDrive, Yammer, etc. A no-brainer to reset or ask them for their passwords.

2.2. Initiate the IMAP migration from the Exchange Admin Center

Before you start, a good survival guide about it is found here. The most obvious limitations:

• Only items in a user's inbox or other mail folders can be migrated. You can’t migrate contacts, calendar items, or tasks.
• A maximum of 500,000 items can be migrated from a user’s mailbox.
• The maximum message size that can be migrated is 35 MB.

Provided this works for you, let's imagine you've already tried the PST import and you want to clear all stuff that you've imported and start anew. The most intuitive thing will be to delete the users - but they will be retained for about 30 days. A workaround is to rename and delete them :) 

Or... as a best practice empty those mailboxes through PowerShell.

Use the Search-Mailbox cmdlet and pipe it with DeleteContent.

Example: Search-Mailbox -Identity "John Doe" -DeleteContent

It's a bit slow, but maybe faster than opening the user's mailbox through Outlook (and for that you'd need to grant yourself Full Access to that mailbox first - lame and unprofessional approach).

So... the trouble with IMAP migration. If you run a single mailbox - the speed is still not much better - between 0.5 and 1 GB per hour. A disaster I'd say.

The key is concurrency - you'd need to run at least 20 mailboxes per batch so you can get your migration done in some realistic schedules that you can present to the business.

With the concurrency, we've easily reached speed of 10 GB per hour with 15 concurrent mailboxes. This totally corresponds with the Microsoft provided data, so you can bet on it. Of course, this excludes a lot of other factors which might block you from reaching such speeds - your ISP, your firewall, your current e-mail server bandwidth and so on and so forth.

In the process of setting this up we've found a nice little calc that helps you do the math.

So once you have your estimations and you prepare your CSV file for the IMAP migration, you start your sync. Everything goes well over the weekend, the Spring flowers are blooming and etc. NO!

There are many reasons why the IMAP sync can fail - wrong CSV format, wrong passwords, your e-mail server goes down, Microsoft Exchange Online goes down...but all of those you can either fix or just hope for Microsoft to fix as quickly as possible. But what to do when the sync fails and you want to start afresh?

Well, first the sync will not always stop if some of the above problems occur. It will continue running even if it runs fine for 1 mailbox and all the rest fail. So you need to stop it manually. From the GUI - I've noticed that stopping it manually leaves it in "Removing" state for more than 12 hours - dunno if it's a bug on our tenant or not, but then your only way is to kill it with PowerShell. The Remove-MigrationBatch command comes handy in this case. If you miss this part, your new sync with all the correct data will NOT run as the same mailboxes are already part of another sync which is not totally gone. Use the "-force" parameter, otherwise there is no result (at least in our tests).

Once you clear the migration job, you go to the GUI in EAC, you see it gone and you import your same CSV file and hope to get this running. No! 2 reasons:

If you import the same CSV file, EAC tries to be smart and doesn't detect the mailboxes which were part of the previous synchronizations. So rename your CSV file.

Your next attempt would also fail. Even if your mailboxes get detected and they go into the batch, they will fail. Don't ignore the error report, but download it and read the following:

The user "dimitar.miriyski@abc.xyz" already exists, but the migration batch that includes it couldn't be found.

Does that make sense? Yes, of course. Your Office 365 user is not deleted. But the migration batch is. So what's the problem with that? There's a history of all those batches... which you need to sweep off manually with the Remove-MigrationUser command. Hopefully you haven't messed with too many users in your failed batch :) Ideally it should never have failed, but things like that happen.

Another trick when running the migration batches is not to use CAPS in the batch title.

Option 3: A mix of 1 and 2.

It seems as the perfect cocktail - you do the bulk stuff with IMAP sync to gain the speed advantage and then you do the calendars, tasks and contacts through PST export and import. Not tested yet.

Conclusion

So... from the short time of research and testing this, I'd say there are a lot of things you'd need to consider and Exchange Online is not a toy. No wonder the Exchange Administrators are still out there, but in the next post from this series I'll see if I can prove that a SharePoint guy with some Office 365 knowledge and experience can do the trick for a small to medium sized company.

Office 365 down in Europe

Today Office 365 went down in Europe and it's down since more than 3 hours at the time of this post.

OK, Microsoft, it's technology - we get it, and you'd still make the numbers in the SLA report due to the high number of users you've got.

But there's something more worrying in today's scenario - communication!

Wouldn't it be nice to put up something on the http://status.office365.com page rather than the bogus message below? And not asking people to login to their tenants and check their dashboards? When in fact they CAN'T login at the time of calling you...

People are relying on this platform for their daily jobs, and a good communication would have been key in user satisfaction with the products. The communications today is absolutely ridiculous.

I even got a call from one of your support reps asking me to do a remote session and see one of our other issues we're experiencing, so I kindly had to inform him everything is DOWN in Europe!

Meanwhile, we had to find the hard way what's working and what's not, e.g;

- OneDrive sync is working
- OneNote sync is working
- Yammer mobile app is fine (if you haven't logged out)
- Everything else is FINE if you were logged in and haven't logged our prior to 11:00 this morning.

I'd say this is a complete communication failure on your end today...

And while you've put this nice little "warning" message on the Azure portal status, it's still unclear and doesn't mention Office 365 (you need to be aware it's dependent on Azure) and it's saying North and West Europe...while NO ONE in Europe can't go to the Login Portal.

1 TB space in SharePont Online! /not OneDrive/

Update 18/02. The space has reappeared (this time as 1 TB - expected) although the feature on the roadmap is still "In Development".

Update: Microsoft has reclaimed this space and when talking to them, they said this was "a test 2 TB rollout to select tenants", and ours is not a First Release one. Shame.

Anyway, they said by the end of January all tenants (first they claimed Enterprise only) should get upgraded to 1 TB.

On this foggy November morning, I've logged in to the admin panel in one SPO tenant and saw this!

Prior to Friday afternoon, that space was ~130 GB, based on the well-known formula 10 GB + 0.5 GB per licensed user. 

The O365 roadmap says Office 365 tenants will now get 1TB + (500MB * # of E/A/G users) but this feature is still in development, and our tenant is not a FR one.

Have you already observed the storage increases in your tenants?

By the way, thanks Microsoft! Really generous :)

Injecting JavaScript in a SharePoint Online site collection the proper way

With recent changes in Office 365 branding guidelines, it has become significantly more difficult to insert small bits and pieces of functionality like a 10-line JavaScript, for example. I had to do this in order to bring the same global navigatoin that we use in our main site collection into the Search Center site collection (the requirement is even more difficult if you use Managed Navigation, but that's not the topic of this post).

By default, the navigation is missing from the Search Center in SPO and SharePoint 2013.

A great way to get it back is described in this post by MVP Elio Struyf. But how would you inject the JS in the whole site collection? By default, we have 7 different result pages in this site, and we plan to create more... so here are our options:

1) Insert the script in a CEWP or a Script Editor webpart - manually in each of the pages... time consuming and the need for manual adjustments arises each time you create a new results page.

2) Use custom master page for the site collection - a big No-No for SharePoint Online since a few months.

3) Build a provider-hosted app that can inject your customization :) Great video by the Office 365 PnP team, but it sounds to me like killing the fly with a hammer! 

4) Register it through another .js and forget about supportability and removal with no code... Quick & Dirty approach by Tim Ferro that I'd only ever use as a PoC.. (tested it and does the job by the way).

5) Using PowerShell! Wait, what!? Yep. After some head-banging last Friday afternoon, my friend and PowerShell guru Ivan Yankulov has reminded me that there was a pretty good cmdlet from the PnP-PowerShell project. 

All you need to do is... 

1) Download the project locally

2) Have WiX toolset installed (at least 3.10 if you use VS2015)

3) (Optional) Have Windows Management Framework 4.0 installed if you want to generate the cmdlet Help

4) Install the binaries (\Binaries\PnPPowerShellCommands16.msi) for SharePoint Online

5) Run PowerShell 3.0 as an admin...

6) Have a look at the AddSPOJavaScriptLink documentation.

7) Connect to your SPOnline tenant (Connect-SPOnline –Url https://yoursite.sharepoint.com –Credentials (Get-Credential)

8) Execute the command...

Let's assume I want to inject something called elementshide.js to my whole site collection. So whenever someone creates a site or a page inside, they'll get the JavaScript loaded automatically for consistent site collection-wide user experience. The key parameter is like an identifier for your script  - take a note of this if you want to remove the script reference later on :) The url is basically the path to the script (relative URL is always recommended) and then the scope could be Site for a site collection, or web for a subsite in SharePoint Online. 

 Add-SPOJavaScriptLink -key ElementsHide -url "~sitecollection/SiteAssets/elementshide.js" -Scope Site -Web https://dimitarm.sharepoint.com

9) Verify. Go to your page and view the source code... you'll have the following piece if the injection went through:

if (scriptsSrc.indexOf('/SiteAssets/elementshide.js') === -1) {

var newScript = document.createElement('script');

newScript.type = 'text/javascript';

newScript.src = '/SiteAssets/elementshide.js'; 

headID.appendChild(newScript);

scriptsSrc.push('/SiteAssets/elementshide.js');

}(function(){ 

So this is it... in a little bit less than 10 steps we've managed to make our Search Center navigation show up...and we're ready for a lot more customization done this way. Hopefully the guidelines by the O365 team do not change to something even more complicated in the next months...

Automating group assignment for external users in SharePoint Online

I have this requirement as part of a site that will be built specifically for external users in SharePoint Online.

It's basically an automation need for users to be granted access to certain areas of the site by adding them to appropriate SharePoint Online groups. Users should also be automatically created by a workflow when their details are picked up from a list.

I've done an attempt to do this in SharePoint Designer, but it turns out there's no such action in there.

A couple of alternatives exist - Nintex Workflow which I have in place and Plumsail Workflow Action Pack which I don't have in place but a quick overview can be found on their website.

With Nintex, there is the handy action "Office 365 add user to group", which is usually working well. For external users, however, I found it impossible to resolve them like any Office 365 user.

I understand this might be due to the licensing part of the whole picture (external users can't be assigned a SharePoint Online license), but I am hoping that Nintex would address this issue in any of the future releases.

The only alternative I found so far is to create this user in the Office 365 tenant, assign a license to it (all automated through the workflow) and then add it to the group. Unfortunately, my requirements involve no licensing for such users, so I'll have to find another possible automation approach.

Has any of you managed to accomplish this? Would be happy to hear oppinions.

SharePoint Online slowness in document libraries when using Managed Metadata

I've had a very strange complaint from a customer that has been setup on Office 365 quite soon and everything worked like in terms of their SharePoint Online for a few months. They're a very small company with less than 50 users. The scenario is the following:

- A couple of subsites under a root site collection in SharePoint Online
- A few document libraries underneath each site
- Some document libraries having 4-5 Managed Metadata columns in the default view
- Those same document libraries take about 30 seconds to load, compared to 1-3 seconds for the rest.
- None of the document libraries exceeds even a thousand documents.
- That issue appeared out of the blue, it was working fine for months.

I've done some tests to remove the Managed Metadata columns and prior to that, reviewed the term store to see if something is exceeding any limit or is in a contradiction with best practices.

After removing the Managed Metadata columns, the libraries started to load in 10x less time. That was very weird, as the columns have just a few choices, nothing that would potentially load the backend too much. Anyway I've decided to raise this with Microsoft and they resolved it in a few days. The first engineer just tried to replicate the issue on another Office365 test tenant, but he couldn't see such high load times. The escalation engineer, however solved it by working on the backend with their internal tools.

The explanation was the following:

"Using backend tools I was able to look up ECMPermissions table for the specified environment and delete any users that were not found in SPODS (user data base). After this action the issue was solved"

In plain English, that means that if you delete some users from an Office 365 tenant, sometimes there are orphaned records in this ECMPermissions table, and that has to be cleaned up manually. I haven't asked if they plan a hotfix or not, but after they intervened, the Managed Metadata is useful again.

2015 will be the Office365 year

First of all, Happy New Year everyone! Let it be more successful and more rewarding in every aspect for each and every one of you in personal and professional terms.

 

After reading Joel Oleson's SharePoint and Office 365 2015 Predictions I can't agree more with the power of Office 365 and its growing ascendancy over on-prem versions of SharePoint, Exchange and Lync. We can definitely see that Microsoft is pushing that really hard, especially to IT Pros with the certification track changes. Just a few months ago, you had to learn Windows Server 2012 and sit three exams related with it in order to proceed to the MCSE levels for SharePoint and Exchange.

Today, you have to know Office 365, no matter if you are an Exchange or SharePoint expert.

Both terms are already becoming too narrow and expanded skillset will be required in order for us to be more competitive on the market and provide even higher added value to the organizations we work with.

I sat the 70-347: Enabling Office 365 Services just before Christmas and I can say it was a close one with a passing score of just 775. The reason is SharePoint being just some 25 % of the exam and all the rest belonging to other Office 365 features like Exchange and Lync Online. Configuring Exchange Online and Lync Online is definitely not one of my specialties, SharePoint is though and next (and final) on the list to MCSE is the 70-332: Advanced Solutions for Microsoft SharePoint Server 2013 exam which I'll sit this month. 

Microsoft seem to have a plan to unite all the IT Pros working with their stack by hosting the Ignite Conferene, replacing all others like the world-famous SharePoint Conference.

2015 will be intesting with vNext coming out as well, I am sure that it will be a vigorous thing.

Exam 70-346: Managing Office 365 Identities and Requirements passed

I went for this exam yesterday morning and passed it with a result of 866. It's been about 10 months since this exam was published, but I needed some hands-on experience with the technologies involved before being able to pass it. I am saying technologies, as it is totally not just about SharePoint, but instead has a lot of quesions about Active Directory Federation Servers, Directory Synchronization, Exchange and even Lync. Overall, if you've setup a few Office 365 tenants you should be able to cover the general questions like how to setup a new domain, how to plan and perform migrations and what ports would you need to open on the on-prem firewalls, how to troubleshoot Office 365 issues, etc.

I found the questions on Lync and Exchange the most challenging, the good thing is that we've done quite a big hybrid project recently and I was closely involved with the consultants that did those bits of the scope. Not much questions on SharePoint Online. Lots of PowerShell commandlets which are strictly from the MSOnline module. I've used it a couple of times when setting up Hybrid Search.

So... here's what my result looks like, weakest part is Cloud Identities, well... I kind of prefer another 3rd party identity management provider, rather than setting up SSO through ADFS, but I'll blog about that in one of my next posts. Next exam on the list: 70-347: Enabling Office 365 Services