Making Your Own Messaging Service For Fun and Profit

Google Hangouts (at least as we know it) is going away. The company’s largest messaging app, which has been around for 15 years and was tightly integrated into Google Talk and GMail, will slowly lose features over the coming months before being killed off entirely.

As this venerated, popular, way to keep in touch slowly dies, you might be wondering, “what if I could run a messaging service? That way I control how long it lasts. And I could ensure it was safe and secure, with no advertisers (or other people) reading my messages.

Well, here’s the thing: you can! I made a private XMPP (Jabber) server for my husband and I, and you can too!

But Why?

Before I talk about how I did this, and the pitfalls you may face, I want to address a question you’re probably asking: Why? There are plenty of other messaging products out there (after they kill of Hangouts, Google alone will have four other chat services). And there are privacy focused messaging services: WhatsApp and Signal are both end-to-end encrypted, preventing the messaging operators from reading your messages. So why bother with any of this?

In a word: control. Running your own messaging service lets you define exactly what you want. While I’ll be setting up XMPP, a pretty simple instant messaging protocol, you can instead deploy something like Slack and Discord, offering channels for group discussions and direct messages (Mattermost is the free, open source alternative to Slack that seems most popular) or, if you wanted to go in the other direction, simple Internet Relay Chat is still alive and kicking (the classic internet chatroom, created in 1988, where you only see messages when you’re logged in).

Regardless of what you’re going for, all these options let you define your own security and rules – you’re not bound by message size limits, restrictions on how many people can be in your chat or how much history you can save. It is your private messaging system – you can do what you want!

Plus, you’ll get hacker cred. And I know you, dear reader, want that.

And Now, A Warning

So I’ve said why you might want to do this (or at least why I did). Now let me tell you why you might not.

Running servers on the public internet is a responsibility. If you’re acting as your own server admin, it’s up to you to make sure that server is secure, and not getting hacked and attacking vulnerable sites.

It’s not hard to setup a server online – there are excellent tutorials to help you get started, and the initial setup will probably take 15-20 minutes if you’ve used Linux with a command line before. And even if you haven’t, the basic aren’t too difficult to learn.

But once your system is up and running, you do need to monitor it. That means logging in every so often (at least every month or so) and checking if there are security updates, making sure the SSL certificates are up to date, etc… While you can automate much of this with tools like unattended-upgrades and certbot, it’s still a thing you have to check up on. While it shouldn’t take more effort than a basic houseplant, it’s still a thing you need to care about.

If that doesn’t appeal to you – that’s totally fine! Signal is an excellent, privacy focused messenger that works beautifully! Mattermost will give you a private instance, for free, for up to ten people, with unlimited history. Discord is readily available.

But you will be beholden to their policies and rules. That’s the tradeoff.

And there’s something freeing about building your own thing!

XMPP and The Joys of Decentralized Platforms

As I mentioned, I’ll be talking about setting up an XMPP server. The Extendable Messaging and Presence Protocol (also known as Jabber) has been around since the late 90s, and coexisted with other popular IM systems like AOL Instant Messenger (AIM) and MSN Messenger back in the day.

While it’s old, it’s still quite capable, and has provided the backend for other messaging platforms, including Facebook Chat and Google Talk.

Unlike other IM systems even today, XMPP works on a decentralized platform. Instead of everyone needing to be on the same service (like Facebook Messenger, where there’s only one place you can get an account), XMPP is like email – users are username@server.com. And they can talk to each other freely – so I, at sean@seanzwrites.com can happily chat with you at awesome_reader@coolplace.net.

There’s something beautiful about this model (and is something I’ll talk more about if I ever do a write up of decentralized social networks like Mastadon); by having a decentralized network, rather than a monolithic service, each server operator can set their own policies, and users can shift between them while still being able to talk to their friends. Want to send bigger files than your server allows? Switch servers. Want private chatrooms but your server doesn’t offer them? Switch servers.

Similarly, the clients for XMPP were not bound to any network (in the same way most desktop email clients can talk to any mail provider), so you also could choose which client you liked the most!

This decentralization and freedom comes at a cost though – complexity. Not every client implemented every feature, and even if they did have a feature, it might not work when chatting with someone using a different program. This was one of the things that bit me in this journey. But more on that later.

Let’s Build This Thing!

While there’s nothing stopping you from running your chat server on a $30 raspberry pi under your desk (that’s actually what I did for my husband and I), I’ll be discussing creating a public server, using a public cloud provider.

While all this might seem daunting at first, but the basic order of operations is:

  1. Get a server and setup your admin account
  2. Get a domain (and point it to the server)
  3. Get SSL Certificates
  4. Setup the XMPP software on the server (and setup SSL certificates as part of that step).

This should take about 30 minutes for someone who has setup a server before, but can take a few hours or more if you run into trouble.

I do think it’s good to learn about computers and networks by doing, and I do think you’ll learn quite a bit by giving this a shot, but, again, no shame if this isn’t for you.

Step 1: Setting Up Our Server

Your server is the heart and soul of this project – the physical thing that connects to the internet, and allows you to be awesome_reader@coolplace.net. There are quite a few companies that will sell you servers, including the “big” cloud infrastructure providers (Amazon Web Service, or Microsoft Azure). That said, if you’ve never done this before, I’d recommend grabbing one from Digital Ocean – their least expensive “droplet” costs $5/month, but their documentation and support is solid, and their systems are turn-key, requiring very little configuration or special knowledge.1

Setting Up SSH Keys

Before we login and hit that “Create” button we’re need to prepare our system. We’re going to want to use SSH Keys to connect instead of plain old passwords. You can think of these keys as special files that will enable you to login (instead of using passwords). This is far more secure, and Digital Ocean can pre-setup a system to use key-based logins. You just have provide the keys, so… let’s create some keys!

Since this is going to depend on if you’re on Mac, Windows, or Linux (though, if you’re already on Linux, I suspect you know the drill), I’m instead going to link to you Digital Ocean’s own documentation here on Creating Keys and Adding them To Your Account. Just note that you’re adding keys to your account, not a specific droplet, at this stage.

Creating Our Droplet

Once you have your keys setup and uploaded into Digital Ocean, you’re ready. Go ahead and hit that Create button in the upper right of the Digital Ocean website, and then hit ‘Droplet’!

You’ll be asked a few questions about what type of Droplet you want – here’s the quick reference:

  • Distribution: Ubuntu 20.04 LTS x64 (well supported, reliable, and will receive updates until 2025)
  • Plan: Basic, $5/mo (you shouldn’t need more capacity than the $5 tier, even if you have quite a few users)
  • Block Storage: None (don’t do anything here)
  • Datacenter: Pick the one closest to you
  • VPC Network: Leave this alone
  • Authentication: SSH Keys (and check the key you uploaded)
  • How Many Droplets: 1
  • Hostname: This can be anything and distinguishes this from other droplets you may have, so the only real requirement is to make it descriptive, like chatserver
  • Backups: Not required, but up to you. They do cost a small amount extra, though.

It will take a minute or two once you hit the final button, but once it’s done you’ll get an IP address for your new droplet, that you can login to via SSH. You can follow the instructions here on how to login.

Note that since you’re using SSH keys – you’ll need to login from a computer with the SSH keys. You can move them between computers – but remember you’ll need them to access your server.

At this stage we want to setup a regular user on the server (it’s bad practice to always login as the admin), and setup things like the basic firewall. Thankfully, there’s a guide for this too – just follow along here!

At this point, you should have a server online, ready, and secured, that we can use. Now we need to a domain to point to it.

Step 2: Getting a Domain

If you want to be user@someplace.net (for example) – you need to own someplace.net. Namecheap is a popular domain registrar that will allow you to buy a domain, though there are others like GoDaddy.

Some quick terms: the domain registrar allows you to buy the name, and then the name servers (usually run by the registrar) directs someone who goes to someplace.net over to your server.

Now, technically you don’t have to use domain. There’s nothing stopping you from just using IP addresses. But, that’s both a pain for users (who wants to remember 185.232.112.15), and will complicate the certificate setup, so… I’d strongly recommend getting a domain.

Aside: If you’re really opposed to spending money, there are free services like DuckDNS that allow you to get a subdomain (myaccount.duckdns.org) for free. The only requirement here is that you will need the ability make subdomains (chatrooms.mywebsite.org). Thankfully, this is standard when you buy a top level domain (mywebsite.org), and some subdomain providers (like DuckDNS) automatically route all subdomains (like chatrooms.myaccount.duckdns.org) to your primary subdomain (so, in this example chatrooms.myaccount.duckdns.org would automatically route to myaccount.duckdns.org).

Anyway, once you register your domain (which may take up to 72 hours to process), you’ll want to add two ‘A’ records to point to your server. Assuming you’ve registered someplace.net, you’ll need:

  • An ‘A’ record for your ‘root domain’ (someplace.net) pointing to your server’s IP address (which you got from Digital Ocean)

  • Another ‘A’ record, called a wildcard record, that also points to the server’s IP address.

You need that second record because we’re going to use ‘components’ in XMPP. These components power things like multi-user chatrooms, image and file transfer, and more. However, they need their own address (like upload.someplace.net). Instead of creating a record for each one, we’re creating a wildcard record that says “if we haven’t specified otherwise, just route it to the main server.”

How you’re going to insert these records varies from registrar to registrar, unfortunately. However, for Namecheap, the procedure is documented here.

Note: if your registrar/name service doesn’t allow you add wildcard records, you’ll need to create subdomain records for each service manually. To do that, create three additional ‘A’ records, one for each of the components: groups.someplace.net, upload.someplace.net, and proxy.someplace.net.

At this point, you should be able to SSH into your server, specifying the domain name instead of the IP address (ssh sean@someplace.net, for example). Remember, it can take a while after setting these records for them to be reflected across the internet, so don’t worry if it doesn’t work immediately after adding the records.

Step 3: SSL Certificates

This used to be the hard part.

You’re creating a public service that is online and accessible on the internet, which means you’re going to want SSL.2 The SSL certificate does two things – it enables encrypted connections to your site, but also proves that your server is actually the one responding to the request (the SSL connection you have with this site confirms to your computer that you’re really connected to seanzwrites.com, and not someone who just jumped in the middle and hijacked the connection).

You used to have to pay money for certificates and installing them could be painful. Thankfully, this is no longer the case, and you can get certificates for free, automatically, from within your server itself, thanks to Let’s Encrypt.

We’ll cover this in the next section.

Step 4: XMPP Server Software

At this point, you’ve setup a server on the internet. Congrats! It could do so many different things – it could host a website, a private social media instance, your own file store, a private VPN for you and your friends to game… You have options!

But we’re making a XMPP server, so let’s install some XMPP software.

While the configuration isn’t hard, I’m actually going to route you to another tutorial, which will walk you through the process of getting SSL certificates, setting up the XMPP server, creating your first XMPP user, and setting up those pesky certificates to auto-renew.

Here’s that tutorial from the homebrewserver.club site.

Note – you’ll need to use the command line on your server to add accounts manually, unless you want people to freely register accounts on your server (you probably don’t). Users can change their passwords on their own, though, so you just need to give them a starter-password.

Step 5: Connecting

You’ve built a server, you’ve installed software, all that’s left is to login.

For this, you’ll need an XMPP program on your PC. I’d recommend Gajim (Windows) or Monal (Mac). Like an email client, just enter your information, and hit connect. You’ll be greeted with a list of contacts (which is likely empty), but still – you’ve done it!

At this point, you’re set. Create accounts for your friends, grab a client for your phone (Xabber, ChatSecure and Conversations are well reviewed), and enjoy the fact that you’re free. You control your own instant messaging destiny!

Coda: Why Doesn’t File/Image Transfer Work in XMPP?

If you’re following the tutorial from HomeBrewServer that I linked above, you there’s a good chance you won’t run into component problems. But complexity around components and clients can often lead to some frustrating problems. So, even if you followed the tutorial (or if you didn’t, and you somehow ended up here because you were wondering why you couldn’t send an image in XMPP), let’s talk about components and clients.

Think of components as services that provide additional functionality on top of the basic instant messaging system, like providing multi-user chatrooms, or mediating direct file transfer connections.

Some components require their own subdomains (like upload.myserver.net for the HTTP upload component). It’s important that you both configure this domain in the DNS/name server record (or have a wildcard), and include it in your SSL setup (again, you were instructed to do this in the earlier tutorial, so you’ve likely already done this — don’t worry).

I bring this up because these components, in modern instant messaging, are less optional and more expected. For example, here’s something that you’d probably assume was incredibly simple: sending someone an image file in chat.

Because of XMPP’s age, and its need to work across servers, it does not allow you to just send a big file as part of a message itself (email also has this problem). The way this was solved in most XMPP clients and servers was to simply transfer the file to the server – when you want to share a file, your client uploads it to the server with your account, and then the server forwards a link to the file to the other person. That’s what that HTTP upload component is doing – allowing your client to shove the file somewhere that is accessible for another user. It’s a pretty simple solution.

The problem is not every XMPP program does this. Pidgin, for example, is a very popular instant messaging client doesn’t support this type of file upload in XMPP. However, it does support a different way of transferring images – sending them directly to the other person using the server as a mediator (supported by, you guessed it, a different component – proxy65).

When looking at the webpage for a messaging client, it will tell you all the features it supports, like sending images, and it probably does all the things it advertises. What might not be obvious is how it’s handling that, since XMPP’s age and flexibility allows for more than one way to do something.

So if you’re wondering why something doesn’t work with one friend who’s using a different program, or with one feature, look at the XEPs – the XMPP enhancements — supported by the client. XMPP clients often have a page in their documentation where they list which XEPs they support (here’s the list for Gajim). Usually XEPs directly correlate to features (HTTP upload, for example, is XEP-0363). If your client doesn’t support that specific XEP, it probably won’t play nice with other clients doing that same task.

  1. There are cheaper providers of turn-key servers (Vultr is $3.50/mo for a slightly less powerful system), and there are companies that will give you servers for free (Oracle and Google Cloud Platform both have free tiers that include servers). Digital Ocean was chosen here because it’s easy, well documented, and popular enough that users should have no trouble finding answers to common questions. ↩︎

  2. Technically SSL has been replaced by the more modern protocol TLS, but it’s common to still refer to the certificate/encryption system as SSL, even if it’s using the TLS protocol. ↩︎