Grouper performance update

On Monday, the Grouper UI was, at times, unusably slow. This, understandably, was a great inconvenience to several people. For that, I apologise. It was unforseen but we now understand the reasons and have formulated a plan of action.

Firstly, the reason – why did this happen? Well, the eagle-eyed amongst you may have spotted that Monday was the first day of teaching for the majority of our students. OK, I hear you say, but what does that have to do with Grouper? Well, it’s actually down to the phenomenal popularity of the Newcastle University mobile app. The way the app is currently architected, there is a web service call to the Grouper API every time a student logs in to the app or they refresh their news feed.

Chart showing a large spike in load on the mobile app at start of term.

Chart showing a large spike in logins to the mobile app at start of term

The chart above shows the spike on Monday, when new students downloaded the app for the first time and returning students logged in to see their timetables. The resulting spike in calls to the Grouper API were too much for the poor little server to handle, with the database process maxing out the CPU.

So, what are we doing about it? Well, we’re not just going to stand here and do nothing but apologise. Our approach is three-pronged (rather like a fancy dessert fork):

  1. We’re going to add more CPU to our Grouper server.
  2. We’re working with Mike, Mike, Andy and Marc to redesign the data architecture behind the app, using purpose-built RESTful web services from IDFS and removing Grouper from the equation.
  3. We’re continuing with our Grouper upgrade plans.

Grouper performance (issues)

One of the main benefits of upgrading Grouper last year was the introduction of “real-time” provisioning of groups and memberships into AD. Previously, AD syncing had occurred four times a day which was good enough for most scenarios but not perfect for everyone.

Since upgrading, the Grouper PSP technology, which handles “real-time” AD provisioning, has coped nicely with everything that’s been thrown at it (averaging around 50,000 changes per month). This chart, showing monthly group membership changes by stem, gives an indication of what it’s handled from March to August 2016. You can see it was busy in August and there’s a peak in April.

Chart showing monthly updates to Grouper groups, by stem, from March to August 2016

Chart showing monthly updates to Grouper groups, by stem, from March to August 2016

It had coped nicely, that is, until the end of the academic year. Now, if we add September 2016 into the chart, it provides a nice visualisation of why PSP has been suffering for the last fortnight.

Chart showing monthly updates to Grouper groups, by stem, from March to September 2016

Chart showing monthly updates to Grouper groups, by stem, from March to September 2016

So, since 1st September we’ve not had any “real-time” provisionsing and the situation has been far worse than it was prior to our upgrade last year, with some changes having to wait well over a week before being reflected in AD. We’ve offered some workarounds to alleviate the impact of the most urgent cases but this service failure still weighs heavily upon me.

As I write this, I’m hopeful that the provisioning service will finally catch up with itself overnight tonight and we’ll return to the happy state of “real-time” provisioning tomorrow.

So, whilst I’m quite content that PSP can cope with our needs for the majority of the year, the service since the start of September has not been satisfactory. We’ve now started making the necessary plans to replace PSP so that we won’t have to suffer like this again next year.

Shibboleth Overview Session

Last Friday, I gave a quick overview of the Shibboleth Single Sign-On service

The session was varied, aimed at everyone from sys-admins, to developers, to end-users but hopefully there was something for most people to take away.

The slides are available for download and the session was recorded using ReCap, if you missed it or want to watch again.

The interest was high and questions (during, and after, the presentation) were wide ranging; it was useful to see from a user perspective what people think/are doing with the service.

Getting to grips with Grouper

Yesterday, Phil and I delivered a short overview session on Grouper, covering what it is, what it’s for, why you might like to use it and some tips and lesser known features.

The session was aimed at both new and existing Grouper users at Newcastle University (anyone who doesn’t consider themself to be an expert). We were very pleased with the turnout (about double what we expected) and the feedback.

If you missed the fun (or if you were there and want to hold on to the experience for a bit longer) you can now enjoy the full session on ReCap and peruse a copy of the slides at your leisure.

By the way, I’d like to offer a belated apology for the Star Wars references; I’ve recently watched the original trilogy with my daughter (her first time) so I’m afraid it was on my mind. I should’ve pointed out that Grouper is not quite as powerful as the Death Star.

Getting started with group types

The Grouper structure at Newcastle University has an implied data flow through the use of four distinct group types. Administrators are able to use Corporate Data Groups to build custom User Groups, a combination of which can then be used to build Application and Non-AD groups.

  • Corporate Data
    These groups are populated from our corporate data source systems (principally SAP), and will automatically update membership as individual circumstances change.
    These groups are not editable.
  • User Groups
    These groups are generally created by computing officers to manage groups which are not reflected in our corporate data, e.g. research groups. User groups may contain a combination of Corporate Data groups and directly assigned users.
    These groups are editable by delegated administrators.
  • Application Groups
    All groups created in this branch will be automatically provisioned to the Active Directory. These groups are used to control access to applications and resources such as websites (via shibboleth) and filestores etc. Application groups may include a combination of Corporate Data groups, User Groups and directly assigned users.
    These groups are editable by delegated administrators.
  • Non-AD Access Control
    These groups are identical to Application Groups other than that they are not provisioned to the Active Directory. Non-AD Access Control groups are used to control access to non-AD integrated applications such as Chubb.
    These groups are editable by delegated administrators.

Why use Grouper?

To many computing officers who have grown up around the Windows stack, you may be wondering why we use an additional group management system such as Grouper when Active Directory has inbuilt group functionality. Simply put, AD is only a very small part of our overall landscape, and Grouper gives us the ability to centralise access control across many systems.

Furthermore, Grouper allows us to call upon the vast repository of data the University holds to automatically provision groups and keep membership up to date, while you sit back and do more interesting stuff.

Speaking of sitting back, Grouper offers delegated group management functionality through a convenient web interface. This allows you to devolve responsibility for maintaining your groups to non-technical office administrative staff or members of a research group who you would normally not want to let loose near AD.

So you want a group that contains only Postgraduate Research students in the SAgE faculty studying on their second year, excluding those from the school of Chemical Engineering, that can be used to populate your door access control system?…Grouper can do that.

Or, you want a group that can auto provision all staff in your school to have access to a file share and internal school website?…Grouper can do that too.

Grouper takes away the pain of managing access control to many systems, ensuring that when individuals move between roles or courses, their effective memberships are automatically propagated to connected systems.

Handy hints for viewing group names

When dealing with long group names in the new Grouper UI you’ll notice that the column isn’t wide enough to display the full name.
Scrolling Group Names

Thankfully there’s a few easy ways you can overcome this annoyance…

Clicking on the folder name will display the entire list of groups contained within:
Scrolling Group Names

Scrolling to the bottom of the white column reveals a scrollbar allowing you to scroll across to display the group names:
Scrolling Group Names

Pressing the scroll wheel on your mouse will reveal directional arrows. Continuing to hold the wheel down you can move your mouse left and right to pan across the group names:
Scrolling Group Names

Alternatively, you can also revert to the ‘old’ Grouper interface without the additional tree view by selecting the Admin UI option in the menu.

Principles, tips and good practice for Grouper administrators at Newcastle University

As I’ve watched the uptake of Grouper grow within the computing officer community over the last year (with many new and interesting use cases), I’ve seen many people face the same problems and struggle with the same dilemmas. The more I’ve helped people tackle these challenges, the clearer the solutions have become in my mind and the more I’ve realised I should try to document some of it. So, here it is, my list of principles, tips and good practices for Grouper administrators …

  1. Use a ‘User Group’ to determine who has admin privileges on all of your groups and folders.
  2. Whenever possible, use ‘Corporate Data’ groups to control memberships of your groups.
  3. Where adequate source data doesn’t exist to define a ‘Corporate Data’ group, create a ‘User Group’ if you’re likely to use the same set of members in more than one place.
  4. Only create ‘Applications’ groups if you need them to be provisioned to AD or available as Shibboleth attributes.
  5. Adhere to the naming convention for ‘Applications’ groups.
  6. Make ‘Applications’ groups specific to a service or purpose.
  7. If you are delegating control of a group’s membership, give out the ‘update’ privilege, not the ‘admin’ privilege.

I realise it’d be useful to supplement this post with a higher-level overview of how Grouper works and what I mean by ‘Corporate Data’, ‘User Groups’ and ‘Applications’ groups. We’ll endeavour to get that written soon.

Give out the ‘update’ privilege, not the ‘admin’ privilege

Our seventh principle of Grouper good practice is:

  • If you are delegating control of a group’s membership, give out the ‘update’ privilege, not the ‘admin’ privilege.

One of the strengths of Grouper is that it enables you to delegate privileges to others to maintain membership of their own groups. If you’re doing this, you should assign the update privilege, which allows the updating of group memberships.

You should not assign admin privileges lightly. As well as allowing for the updating of group memberships, the admin privilege also allows the user to rename, move or delete the group. Or they could even remove your admin privileges, leaving you in a bit of a hole if you need to fix something.

Make ‘Applications’ groups specific

Our sixth principle of Grouper good practice is:

  • Make ‘Applications’ groups specific to a service or purpose.

This should ensure that each group provisioned to AD has a clear purpose and avoid any unforeseen consequences when a group is deleted, for example.

Additionally, requirements can change. Initially you might want to control access to a number of services with the same group but, over time, the people you want to grant access to might diverge from the original group. This scenario is best served by having specific ‘Applications’ groups all containing the same ‘User Groups’ group as a member. This gives the flexibility to allow the memberships to diverge, if required.