Technology Blog

Intro

When you’re building a consumer-facing product, you’re very quickly going to need a way to communicate updates and changes to those users outside the product itself. At SimplyDo, we send notifications for a variety of things - updates to ideas, challenges, groups, and more. We also notify users for relevant account updates, such as password changes.

Communicating with users is a powerful part of SimplyDo’s customer success process, and a key method of driving engagement and re-engagement. Many users first interaction with SimplyDo is via a notification - so ensuring these are successful is of paramount importance to us.

We started out with simple email notifications, but this quickly grew over time as we added features and expanded our scope. Here is a narrative of how communicating with our users has evolved over time, and the unseen complications that come with it.

Email

Email is the obvious first step for communicating with your users. After all, if you’re operating a service that requires creating an account, you’re highly likely going to require users to provide emails on sign-up - and you need a way to verify those emails, allow them to reset password, and send any important account notifications.

We operate our own mail server in our cloud, but it’s also possible to use a third-party service such as SendGrid or Mailgun. Whether you run your own server or use a third-party one, they will all act similarly - they act as a SMTP mail relay, and you send emails to them, and they send them on to the recipient. They can also handle bounces, and incoming mail, which is something we’ll come back to later.

Step 1: Sending the first messages

Regardless of the approach you use for sending emails, your immediate concern will be how the emails look. After all, it’s highly unlikely any professional product will be sending plain-text emails. You’ll want to send HTML emails, and you’ll want them to look good.

Because our emails are coming from our services (e.g. our Python backend), the design is done in code. We use Jinja templates for our email design; we have a global base template that all emails use, and then each email has its own template that extends the base template. This allows us to have a consistent look and feel across all emails, and also allows us to easily change the design of all emails by changing the base template. From the Jinja template we customise the main text, any actions (e.g. buttons) and any other dynamic content. The templates in-code are converted to HTML content, which is then sent to the mail server. Python’s smtplib library is used to send the content to the specified mail server, with the appropriate headers and addresses.

Step 2: User preferences

Once you have the ability to send emails, you’ll quickly realise that you need to give users the ability to control what emails they receive. This is a legal requirement in many countries, and is also just good practice to avoid spamming your users with unwanted emails, and you don’t want to be sending emails to users who are not interested.

As soon as your emailing expands beyond account specific emails, you will need to give users controls over what they receive. For example, if you’re sending notifications for new ideas, you’ll need to give users the ability to opt-out of those notifications. You may also want to batch communicate in some way, to avoid spamming users with too many emails. We added a number of “digest” emails to our product, which summarise activity in daily, weekly or monthly chunks.

Our emails have two methods for opting out.

1. Account Preferences. We initially allowed users to adjust their email preferences from their user account page. We provision all new user accounts with a default set of email subscriptions, and they can adjust these from their account settings. This was an important initial step, but we encountered problems where users who haven’t logged in for a while, or have forgotten their password, were still receiving emails. We needed a way to opt-out of emails without requiring a login.
2. One-click Unsubscribe. GDPR requirements state that your email communication must have an obvious way to unsubscribe from them. All of our non-account emails have a one-click unsubscribe link that includes a token linked to their account and the type of email being sent. Even if users are not logged in, clicking this link verifies the token with our API, and if valid, will unsubscribe them from this type of email without them having to authenticate themselves.

Step 3: Avoiding rejection

SimplyDo as a platform houses “organisations”, which have users contained within them. This means that organisation admins have some control over communicating with users within their organisation. They are also able to send “announcements”, which are effectively custom emails sent to any/all users in SimplyDo that use our email service. This has become very important to our clients, but also means that administators could potentially spam users with emails, even if this is not their intention. We can also encounter the issue of bounced emails - if an email address is no longer valid, or the recipient’s email server rejects the email, we need to handle this.

One of the main issues with spam for us is that it can potentially result in our email server being blacklisted by common email providers. This would mean that emails sent from our server would be rejected by the recipient’s email provider, and would not be delivered. This is a problem we desperately want to avoid as it would totally disrupt the service for our users. Bounced emails can also cause blacklisting.

Bounced emails

As the platform matured, we added robust email tracking to keep an eye on email bouncing. We are notified when an email bounces, and we keep records of unsent emails to aid investigation. We can use these logs and notifications to quickly identify and resolve any issues with our email service, or any user accounts that may be causing problems.

Enforcing email verification can also help with bounced emails, as we immediately know if an email address is invalid before sending a plethora of communication to that address.

Spam emails

Spam is much harder to avoid given we allow users to send custom emails to other users. Our foremost method of avoiding this is communication & trust - We let users know that it is prohibited to send spam emails, letting them know of the consequences of having emails marked as spam. Given that our clients are equally invested in the success of our platform and communicating with users, they tend to be careful about this.

You could theoretically also implement automatic anti-spam measures, ranging from time limits to more sophisticated content analysis. We have not yet had to implement any of these measures, but it is something we may consider in the future.

We also need to be careful to avoid spam with our automated messages for things like new ideas, challenges, etc. If we send too many that aren’t relevant to a given user, they may mark them as spam, which dings our email reputation. We attempt to avoid this by offering users to receive updates in digest form, and by default only sending the minimal amount of emails required for the platform to succeed.

Push Notifications

Beyond our web app, we also offer a native mobile app for Android and iOS devices. A side-effect of this is that we can also send push notifications as a companion to our emails. They allow us to deliver updates directly to users’ lock screens, provide more immediacy than emails, and also tend to be seen more reliably. They also give us another avenue to reach users who are in a position where accessing their emails isn’t a common occurrence.

Implementation

Our mobile app was developed using React Native and Expo; Expo helpfully provides a Push Notifications API that allows us to send push notifications to our users. We then utilise their Python helper library to implement a simple API that allows us to send push notifications from our backend. We tend to link push notifications to our email setup - many of our emails have an equivalent push notification, so they can be sent in parallel for the same event. We also implemented the same method for opting out of push notifications as we did for emails - users can opt out from their account settings.

Another advantage of push notifications is how closely they integrate with the mobile app itself. Using deep linking, push notifications can open the app directly to a specific page; an idea push notification can open the app directly to that idea, for example. This allows us to drive engagement with the app in a way that emails cannot. While emails remain as our primary method of communication, push notifications are a useful companion to them.

Web Push Notifications

Beyond emails and push notifications, we have begun experimenting with other ways to communicate with our users. Some of these are in the pipeline, but one we have currently started to use is web push notifications. These are very similair to mobile push notifications, but appear in a user’s browser on their desktop or mobile device instead. They allow us to provide the immediacy of push notifications without a user having to install our mobile app.

Implementation

Due to the fact we already had a mobile app notification infrastructure setup, it was relatively trivial for us to implement web push on our backend. We can use the same text and images as we do for mobile push notifications, and the same API in our product stack.

Then on our web application, we utilise the JavaScript Firebase Cloud Messaging API to register users for web push notifications. We send the registered token to our backend to be linked with the current user account, so whenever a mobile push notification would be sent, a web push notification is also sent to the same user through Firebase.

Difficulties

While we find web push notifications to be powerful when available, the availability became an issue for us. Web push notifications are only available on relatively modern browsers, and we have many clients running older browsers that do not support the web push APIs. It also requires users to give their permission for web push notifications, when many users may be accustomed to denying them due to the prevalence of spammy web push notifications on many websites.

Outro

We have previously, and continue to, experiment with as many methods of communicating with our users as possible. Finding ways to reach users in increasingly crowded inboxes allows us to increase our own signals against the existing email noise, and provide timely updates and notifications that they need.

We still find email to be by far our most reliable and successful form of communication. Despite the advantages offered by push notifications, the ubiquity of email and widespread understanding causes it to be the most important tenet of our communication strategy.

This article focuses on the narrative of how our communication evolved over time, but skips out on the heavier details of a successful email strategy. The technical implementation of providing a consistently available email service has been challenging over time, and the more theoretical ideas behind adding effective Call To Actions (CTAs) to emails are a whole other topic. This article aims to demonstrate how complicated user communication can be beyond the initial remit of “sending a password reset email”, and how it can grow over time.

Intro

At SimplyDo, we are constantly looking for ways to extend the accessibility of our end-user facing platform. We expect that our end-users (the idea creators) will access SimplyDo in a wide variety of hardware and platforms. One of the biggest hurdles we face is supporting this diversity in addition to expanding the usefulness of our product.

One of our biggest challenges is, consistently, the initial step of getting users into the platform for the first time. Providing the motivation and simplicity to entice unique users is a compelling challenge, and a problem shared by practically any software platform. One of our potential solutions to this was to create a version of our platform that operates solely as a Microsoft Teams app; something that can be pre-installed by organisation admins and doesn’t require users to familiarise them with something new. We hoped that emulating existing patterns and visuals from Microsoft’s provided UI framework, Northstar, would ease users into using SimplyDo, reducing the complexity of learning an entirely new platform.

There are a number of details and intricacies to this development process I won’t go into here. This article focuses on what we perceive to be the biggest advantage of this entire undertaking – removing the hurdle of user sign-up/login by utilising silent authentication. That is, leveraging the fact the current user is already logged into Microsoft Teams, then going through a token exchange process with our servers to provision a SimplyDo user account.

Unfortunately for us, this was a relatively novel process within Microsoft Teams, and documentation for this process was disparate. Microsoft have some surface level blog posts, but following our own trial-and-error, we want to share an overview of the process for anyone else who wants to include a similar procedure in their Microsoft Teams application stack.

The following information assumes usage of a React/JavaScript frontend application, a Python and/or JavaScript backend, the JavaScript package @microsoft/teams-js, and the JavaScript or Python Azure package msal. Some basic knowledge of Microsoft architecture is also assumed; for example, the idea of a tenant ID (unique to the “organisation” using your application) and a client ID (unique to your application).

Step 0: Providing Admin Consent

We hit this wall after developing our initial silent authentication process, but in retrospect, it makes total sense. Microsoft do not want any random service to access a user’s Microsoft account login tokens, for obvious security reasons. Therefore, before any silent authentication can take place, an organisation admin must provide permission for your platform to access this.

Fortunately, due to the aforementioned @microsoft/teams-js library, this is straightforward, if a little cumbersome.

Assuming your platform requires a user to login, the main restriction with this is that your application is unusable from an initial installation until an organisation admin provides consent for users to your platform to authenticate with Microsoft. Additionally, Microsoft does not provide a way for a frontend application to check whether admin consent has been granted until any given users authentication fails, neither can we identify whether the current user is an admin or not. Therefore we need to keep track ourselves (on the SimplyDo end) when admin consent has been granted. Hopefully in the future Microsoft will provide a way to retrieve this information from them, as currently if admin consent is revoked for whatever reason, you will have a data mismatch; your application will assume admin consent has still been granted.

Firstly, and this applies to any instance where you will be using the Microsoft authentication libraries, we must initiate the authentication.

  await microsoftTeams.authentication.initialize();

Next, we initiate an authentication flow, which is again a pattern you will see throughout this process. The authentication flow opens a new secure window to access Microsoft’s identity provider; a new window is required because this content is blocked from appearing in an iframe. At this stage, your application is required to be aware of the current organisation’s Microsoft Tenant ID. There are a number of ways of doing this, I won’t go into them here.

  // Open the authentication flow window
  microsoftTeams.authentication.authenticate({
    url: window.location.origin + "/yourapp/adminConsent?tid=" + organisationTenantId,
  })
    .then((result) => {
      // On successful admin consent granting, store this somewhere
    })
    .catch((error) => {
      // Display an error message
    })

The previous snippet opens an authentication window to a page in our application. Following from that, we will redirect the user to Microsoft’s identity provider.

  const provideAdminConsent = useCallback(async () => {
    if (tenantId) {
      const queryParams = {
        client_id: "Your app's client ID",
        redirect_uri: window.location.origin + "/yourapp/adminConsentEnd",
        scope: ".default"
      };
      const consentEndpoint = "https://login.microsoftonline.com/" + tenantId + "/v2.0/adminconsent?" + util.toQueryString(queryParams);
      window.location.assign(consentEndpoint);
    }
  }, [tenantId]);

  useEffect(() => {
    provideAdminConsent();
  }, [provideAdminConsent]);

This will bring the current user through the traditional Microsoft authentication flow, after which they will be asked to provide admin consent for users to authenticate with your application. Agreeing to provide admin consent will trigger a success response; this automatically closes the popup window and runs the .then(() => {}) from the original microsoftTeams.authentication.authenticate() call. This is where your application should record that admin consent has been granted, so the admin consent option isn’t shown to users going forward. From Microsoft’s perspective, your application now has permission to access their identity service, which is required for the rest of the silent authentication process.

Step 1: Acquiring the auth token

Now that we have permission to access users’ authentication tokens in our application, the step of acquiring the token is luckily extremely simple. The key to this is the silent part of the silent authentication process; the authentication occurs without the users’ knowledge, requiring no input from them.

  microsoftTeams.authentication.getAuthToken()
    .then((authToken) => {
      // Handle success
    })
    .catch(() => {
      // Handle failure
    });

This snippet is all that is required to access the users authentication token. Nonetheless, Microsoft recommend that your application provides a manual option (e.g. username/password) in case of failure when getting the token.

Step 2: Using the auth token

This step will demonstrate how we use the token we acquired from the user in Microsoft Teams. From here, we move from our React Teams application to our backend. In our case, our authentication service is using Node with the @azure/msal-node library - We use this to get information about the user from Azure Active Directory, which in turn we will use to provision a user in our application.

  const msalClient = new msal.ConfidentialClientApplication({
    auth: {
      clientId: your_microsoft_client_id,
      clientSecret: your_microsoft_client_secret
    }
  });

  const result = await msalClient.acquireTokenOnBehalfOf({
    oboAssertion: your_user_teams_token_here,
    skipCache: true,
    authority: `https://login.microsoftonline.com/${your tenant id}`,
  });

In the first part of the code snippet, we construct an msal client to allow our authentication service to communicate with Azure Active Directory. Your clientId and clientSecret are found in your Microsoft Application Portal, and are unique to your application.

In the second part of the code snippet, we exchange the Teams authentication token for a users Azure token - allowing us to access the Azure API on behalf of the user that provided the token. We will use the Azure API to get information about the user.

The following snippet is highly generalised and will depend massively on the specifics of your organisation. The result.access_token acquired from the acquireTokenOnBehalfOf allows you to access user profile data from Azure Active Directory, depending on your app client’s scope. We use some of this information to construct a SimplyDo user account on behalf of this user; we will demonstrate this in the following code snippet, but there are a number of potential options from this point and I recommend consulting the @azure/msal-node documentation.

  request.get(`https://graph.microsoft.com/v1.0/me?$select=${fieldsToFetch.join(',')}`, { auth: { bearer: result.access_token } }

We use the /me endpoint on behalf of the user to request their user data – we then construct a SimplyDo user using their name, email address, job title and job department, where applicable. The user data we requested is dependent on the scope requested of that given user, e.g. User.Read.

Step 3: Finishing up

We have finished the Microsoft specific parts of the authentication process; we have used the user’s Microsoft token to create a SimplyDo (or your application) user on their behalf. At least until we need to reauthenticate with Microsoft at some point, we have completed the silent authentication process. From here your next step is very likely to produce an API token, which can then be used by the Teams application to interact with the backend henceforth.

Outro

Our hope is that this article distils the most important steps of Teams silent authentication. We collectively spent a fair amount of time digging through documentation for what ended up as quite a small amount of code to achieve the main objectives of silent authentication. I recommend reading up on what each function does to understand the implications of what each does, ensuring you don’t misuse your users’ data in any way.

I have personally been motivated for some time to “give back” a tutorial having utilised so many Medium articles in my process to becoming a software developer – this process finally gave me the opportunity to provide some unique insight that I personally would’ve found useful when I was developing it. If this helps you or you have any questions, please feel free to contact me at niall@simplydo.co.uk.