Ubiquiti Networks is a popular vendor of networking-related equipment in the SMB / SME space. Their gear is immensely popular among prosumers too, thanks to the combination of ease of use and the ability to customize for specific requirements. I have been running an Ubiquiti UniFi installation at home for the last five years or so, and recently had the opportunity to create a new deployment in another country. There were two main reasons to go with Ubiquiti for the new location - a single management plane for both sites, and the ability to easily create a site-to-site VPN.

The new installation was fairly smooth and the site-to-site VPN was up and running in a stable manner until the ISP at the remote site moved the gateway from a public-facing WAN IP to one behind a carrier-grade NAT (CGNAT). That started a deeper investigation into various options available for site-to-site VPNs with Ubiquiti's gear for different scenarios. In this process, I ended up encountering a host of issues worthy of documentation to help folks who might encounter them in their own installations. This article provides a recount of my trip down the rabbit hole - including a step-by-step guide detailing my attempts to work around the various pitfalls.

Introduction

Ubiquiti Networks offers a range of products targeting the networking market. While wireless ISPs are a key market segment for the company (serviced by the airFiber line), today's piece is focused on their UniFi product line - a range of managed software-defined networking equipment for SMBs, SMEs, and prosumers. There are a number of reasons for UniFi's popularity products among tech-savvy consumers. The company had a first-mover advantage in offering a cost-effective managed SDN solution. Isolating functionality into different devices (security gateways, routers, switches, and wireless access points) allowed users to pick and choose different equipment based on their custom needs. The unified management plane for all the UniFi products enables easy maintenance while retaining deployment flexibility. Network scaling in response to requirement changes is also straightforward. The company started out with a local management controller, which has now been augmented with a cloud-based offering.

My first brush with Ubiquiti was their mFi product line (which has since been unfortunately EOL-ed). Their lineup of network-connected power outlets with energy and power monitoring, as well as remote relay control was (and continues to be) more flexible than anything else in the market - and this was without even taking the low pricing into account. I had purchased a few of their units for my home / AnandTech testing lab use, and written a short review after a couple of months of use (those units are still in deployment).

After I published the mFi review, Ubiquiti's PR department approached me with an offer to review their UniFi product line. Around that time back in 2017, I had the opportunity to lay out a wired Cat 6 backbone for all the rooms in my house here in California. I took up the offer to spec out a UniFi system for testing out. The USG Pro 4 gateway took up the routing duties with a UniFi Cloud Key (first generation) performing controller duties. Access points with varying capabilities were mounted around the house to avoid wireless dead-spots. A number of switches were placed in the media center and different lab locations. I ended up augmenting the system with additional PoE switches and in-wall APs on my own.

The system was configured with the usual guest wireless network, and a bunch of different VLANs (serving the IoT devices in the house, the home lab equipment, and another for devices such as the common family desktop, phones, etc.). On the whole, it was an overkill for a residential installation. That said, the deployment has held its own over five years of stressful usage (and still going strong). The only hiccup I had was when the CloudKey controller became inaccessible on the network a couple of years back. It turned out that a power interruption had ended up corrupting the database - nothing that a few SSH commands (thanks to the helpful community) couldn't resolve. Since then, I ended up investing in a UPS for the rack holding the UniFi equipment to avoid the recurrence of such scenarios.

Such issues are also the reason why I recommend Ubiquiti equipment only to tech-savvy users. In almost all cases, calling up the company's support line and creating a ticket ends up being a waste of time. There are innumerable resources online (both the company's own users forum, as well as countless prosumer bloggers such as Scott Hanselman and Troy Hunt. In light of reviews from such sources, there is not much for readers to gain from posting yet another review of the Ubiquiti UniFi lineup. Instead, I am hoping to take up specific use-cases and figure out how Ubiquiti's product lineup can address those in these series of articles.

Earlier this year, my parents back in India decided to downsize their home. I took the opportunity to revamp their home network from the ground-up. I had been intending to add features to the home network of my parents, but had never had the opportunity because my visits were becoming infrequent. However, with my first visit post-pandemic, I wanted to get a few things set up as part of their move:

• Easier remote management and troubleshooting of network issues without the need for port forwarding.
• Ability to seamlessly use their Indian home network during travel / visits over here to California
• Ability to perform secure remote offsite backups for my data without relying on an external cloud storage provider
• Ability to seamlessly utilize Indian OTT service subscriptions irrespective of user location either in California or in India

When I initially set up the Cloud Key back in 2017, there was no requirement to use a cloud account. Unfortunately, the UniFi Network mobile application user experience became quite onerous without a ui.com ID a couple of years back. I caved in and ended up associating my installation with a cloud ID just for this purpose. Since I was already managing my network through this ID, it became a straightforward decision to go with Ubiquiti for the deployment back in India.

The key to fulfilling the above requirements was a secure VPN tunnel between my home network here in California and my parents' network in India. Prior to traveling, I arranged for a Ubiquiti Dream Machine to be delivered to the new home. The Ubiquiti UniFi Dream Machine is an all-in-one solution / UniFi starter kit. It integrates a 4-port switch, a 4x4 802.11ac access point, a security gateway, and an integrated controller. The Annapurna Labs AL314-based solution comes with a single WAN port, and is an acceptable solution for most home networks in the the 1000 sq. ft - 1200 sq. ft range.

From my use-case perspective, I wanted a solution that would support simple VPN tunnel configuration and easy app-based access for both the US and Indian networks via a single interface.

The Evolution of UniFi - A Short Recap

Ubiquiti's UniFi lineup was launched after their lineup of edge-focused products for WISPs started gaining traction in other markets. These EdgeRouters and EdgeSwitches were based on Vyatta OS, and the UniFi products initially started out with the same EdgeOS firmware base. The UniFi Security Gateway Pro 4 in my primary deployment runs EdgeOS to date.

The USG Pro 4 is based on Cavium's OCTEON II networking SoC, with a MIPS64 application processor. However, Ubiquiti's latest gateways / routers / switches in the UniFi lineup now run a custom Debian-based Linux distribution. The UniFi Dream Machine uses the Annapurna Labs AL314, and runs a distribution meant for the AArch64 platform. The UniFi OS itself runs as a container using podman.

The end result is that there are quite a number of disconnects between the features available on EdgeOS and UbiOS / UniFi OS. Migration from the EdgeOS line to UniFi OS is not straightforward enough for heavily customized installs. With focus shifting to UbiOS / UniFi OS, the updates for the older equipment have become few and far apart. While that might not be a concern for stable networks, it has unfortunately not kept up to date with evolving network security practices. For example, Android's recent releases have completely dropped support for L2TP VPNs, while EdgeOS has L2TP as the recommended VPN server type. This brings us to the topic of VPNs.

VPN Server Options in Ubiquiti's Stack

Ubiquiti offers a range of VPN options depending on the gateway being used. At home here in California with the USG Pro 4, I have been running a L2TP VPN server (allowing me to connect to it from public coffee shops and airports for secure browsing purposes) for several years now. I had minimal trouble setting it up for access from a Windows notebook. However, as mentioned in the previous sub-section, this VPN server is of no use for my mobile phone running Android 12. The USG Pro 4 also supports PPTP VPN, but it is not recommended even by Ubiquiti themselves.

The primary option for a VPN server in the UniFi Dream Machine running UbiOS / UniFi OS is quite different.

Here, Teleport (Ubiquiti's customized Wireguard implementation) takes precedence. This is a one-click VPN more in tune with today's mobile-first ecosystem. Clients are authorized via invites that can be generated either from the configuration page (on the unifi.ui.com cloud, or via the machine's local IP) or the UniFi Network mobile app. The invites can be opened on the client device using the Wifiman mobile application. The unfortunate aspect here is that Windows users are out of luck. While MacOS, Android, and iOS are covered, Windows users are left in the lurch. This is a hugely disappointing situation given that the L2TP option in EdgeOS works with Windows clients, but not Android and the Teleport option in UbiOS / UniFi OS works with Android clients, but not Windows. It must be noted that the UDM still supports L2TP for Windows clients.

Under the Teleport & VPN section, Ubiquiti also provides an option to create site-to-site VPNs, which is where our story starts.

Site-to-Site VPN: Manual IPSec

Despite residing in the heart of Silicon Valley here in California, I have exactly one ISP offering speeds greater than 25 Mbps - Xfinity. For the last few years, I had been running shop with a 100 Mbps down / 5 Mbps up cable plan (which Xfinity has graciously upgraded recently to 400 Mbps / 20 Mbps). As mentioned in the previous section, I do run a relatively heavy network, thanks to the lab infrastructure in place for evaluating systems and storage devices in addition to serving the needs of a typical family of four.

Back in India, there is a lot more competition among ISPs to serve consumers. There are multiple fiber-to-the-home (FTTH) service providers with a wide variety of symmetric speed options. For the new home, my parents opted to go with Airtel's symmetric 100 Mbps plan (costing approximately US $12 inclusive of taxes). Their network demands were not too heavy - a smart TV, couple of mobile phones, a desktop, and a notebook - with only a couple of clients being simultaneously active.

While I had multiple VLANs at home, with a specific subnet for guests isolated from the rest (automatically created when a guest Wi-Fi network is configured), the configuration for the UniFi Dream Machine had only one primary network and another guest network.

During the initial configuration of the UniFi Dream Machine, Airtel had provided a public-facing WAN IP for the UDM to pick up. There was a necessity to call up the ISP to put their gateway (FTTH terminal / Wi-Fi AP) in bridge mode, but that is outside the scope of this article. The UDM was configured with the appropriate credentials to authenticate over PPPoE and pick up the WAN IP via the bridge connection. With the public-facing WAN IPs of both sites at my disposal, configuring the site-to-site VPN was a breeze.

On the US side, activating the site-to-site VPN network creation prompted for the required details - network name, VPN protocol, the pre-shared key, and the server address. The USG Pro 4 supports manual IPSec and OpenVPN, with the former capable of getting hardware-accelerated. A random pre-shared key can be generated and copied over. The server address was set to the WAN IP of the USG Pro 4. Under the 'Remote Device Configurations' section, it was required to specify the remote subnets desired to be made visible locally, along with the WAN IP of the UDM.

Similar information was entered in the UDM, with the pre-shared key generated on the USG Pro 4 placed in the PSK field. Ubiquiti has slightly reworked the UI in the UDM's Network application (Network 7.2.94 vs. 7.1.68 in the USG Pro 4), with the 'server address' tag being replaced by 'UniFi Gateway IP', making things slightly more user friendly. The remote device configuration section is filled with the required subnets from the US side, along with the USG Pro 4's WAN IP.

Upon adding the new VPN network on both ends, there was a handshake between the two devices and I was able to access the devices in the Indian network from the US and vice-versa. The web UI configuration transparently handles all the port openings required on either end.

The site-to-site VPN setup was further augmented with an old NUC connected to the UDM. The PC was set up to run a squid proxy server. In the US, an Android tablet was dedicated to accessing the Indian OTT services and set up to access the Internet using the NUC as a proxy. This configuration worked fine for more than a month. There were a few interruptions due to power failures and DHCP WAN IP changes on the UDM side. The latter had to be reflected in the site-to-site VPN setup and resulted in some downtime, but was not a cause for major concern.

I was fairly happy with the setup and would have left it as-is, if not for waking up one fine morning and finding the VPN link down. Expecting the customary WAN IP change, I fired up the UniFi Network app and tried to figure out the new IP assigned to the UDM.

Using the 100.107.xx.xx IP in the site-to-site setup was not helpful in re-activating the VPN link. Given my lack of formal network administration skills, this ended up being my introduction to the nitty-gritty details of carrier-grade network-address translation (CGNAT) - a term I had only encountered in passing earlier.

The CGNAT Spanner in the Works

Carrier-Grade NAT (CGNAT) is used by ISPs to circumvent the lack of IPv4 addresses available for allocation to their customers. As more and more devices come online and connect to the Internet directly, it is not surprising that there is a dearth of IPv4 addresses to hand out. CGNAT is mostly encountered with WAN connections enabled by mobile service providers (think of smartphones connecting to a 4G or 5G network). However, many wired ISPs (such as Airtel in India) have also started rolling out CGNAT. The easiest way to identify if one is behind a CGNAT is to search for one's IP address on Google. Google helpfully displays your public IP address as the first result. If this address is different from the WAN IP reported by your gateway (and that is usually between 100.64.0.0/16 and 100.127.0.0/16), you are likely behind a CGNAT. This is different from a local IP gathered by a gateway placed behind ISP-supplied equipment that acts as a DHCP server itself.

CGNAT is rarely a problem for regular web browsing and content consumption. Problems start cropping up when dealing with services that require opening up ports on the gateway for bidirectional communication - such as VoIP services, multi-player gaming, and VPN setups. Even simple things taken for granted like systems in the network being able to communicate with a remote NTP server (for time synchronization) are not possible. Since the NAT happens at the ISP end, it is not possible for port forwarding to be configured either. Some services manage to find their way around CGNAT using STUN / TURN, but power users end up finding one roadblock after another. On the whole, CGNAT is a pain in the neck for users accustomed to obtaining public-facing WAN IPs.

NTP Synchronization Attempt behind CGNAT, and the 'Solution' (Using a 'Local' NTP Server)

Ideally speaking, configuring the Site-to-Site Manual IPSec VPN on the USG Pro 4 (having a public WAN IP) with a remote server address of 0.0.0.0, and providing the USG Pro 4's WAN IP as the remote server address on the UDM side (behind the CGNAT) should have worked by allowing the CGNAT side to initiate the site-to-site connection with the server. I did try out this suggestion made by an Ubiquiti employee many years ago - however, the initial key exchange / handshake process appeared to be break down with the response from the USG Pro 4 apparently getting dropped by the UDM's ISP.

An online search for enabling site-to-site VPNs with one site behind a CGNAT brought up two sets of results - one suggesting a connection via an intermediate VPS (cloud server hosted by a 3rd party), and the other trying to make the public WAN IP-possessing side to act as an OpenVPN server and the CGNAT side to act as a client and initiate the connection. I was not interested in bringing a VPS into the picture, and my trials with OpenVPN did not yield any promising results.

At this point, I realized that my Android phone was able to open up a Teleport connection with the UDM despite the gateway going behind CGNAT. So, my first attempt was to get a system to connect to the Teleport VPN and route traffic from specific devices through the system's Teleport VPN connection.

Teleport VPN - Ubiquiti's WireGuard

Ubiquiti recommends Teleport for VPN purposes in the UniFi Dream Machine. Activating this feature requires remote access to be enabled on the controller followed by Wifiman in the Network application. While this is straightforward enough for most setups, there are certain situations that trip up this flow. In my case, one of the power failures ended up corrupting the UDM - Internet access from within the network worked, but the web UI was inaccessible, SSH attempts timed out, and Teleport would simply give up. The unit had to be resuscitated via the recovery IP using a firmware image downloaded offline. After recovery, the UDM was accessible over the local network and Internet access worked. However, it was not visible either in the mobile app or the cloud-based management portal as remote access was disabled (and couldn't be activated despite repeated attempts). It turned out that the inability of the unit to communicate with the default NTP servers from behind the CGNAT had resulted in the date / time of the unit being set based on the firmware build date. Fixing this manually allowed the remote access feature in the controller to be activated.

Enabling WiFiman is done in the System section of the Network application.

After taking care of these two aspects, Teleport invites can be generated under the 'Teleport & VPN' section of the web UI as well as the mobile application.

Teleport Invite Generation using Cloud Management / Local Web UI

Teleport Invite Generation using the UniFi Network Android App

The advantage of using the Android app to generate the invite link is the ability to directly open it with the WiFiman app to get into the target network. Upon accepting the invite in the app, the connection is available for activation as long as the source gateway (UDM in this case) has a connection to the Internet.

The Teleport functionality in the UDM currently forces the invited device into the 192.168.2.0/24 network. This is currently not user-configurable. In my case, this was not a show-stopper, as the UDM automatically took care of enabling seamless communication between the primary 172.16.0.0/24 subnet and the VPN subnet.

In the first few days of grappling with CGNAT, I attempted to set up a dedicated Android device to act as a Teleport client for other devices in the US to connect to. Unfortunately, except for my Pixel 6a, none of the Android devices at my disposal (I even tried setting up an old NUC with Bliss OS) could help in this aspect. Either the Teleport feature in the WiFiman app was not available for the old Android version, or, the connection proved to be extremely unstable. After a fruitless couple of days, I gave up completely on this approach.

The Teleport feature did end up serving me well beyond the attempt, though. Throughout the course of future experiments, I had Teleport as a fall-back to be able to SSH into the UDM network and configure the gateway as well as machines local to that network.

SSH over Teleport via an Android Intermediary
(From L to R) Clumsy SSH access, Activating SSH server on the Pixel 6a, Allowing ADB access

As a good security measure, Ubiquiti does not allow SSH access to the UDM over remote access by default, and the only safe way to achieve this is to create a secure tunnel into the target network prior to attempting a SSH login. Initially, I used the termux app to directly SSH into the UDM (after connecting via Teleport, obviously). Executing shell commands over a mobile keyboard turned out to be frustrating exercise. Fortunately, it is possible to run a SSH server on an Android phone and connect to it via ADB. With the Teleport connection active in the background, it became easy to access the UDM over SSH from a proper desktop.

SSH over Teleport via an Android Intermediary using a PC

Prior to starting the whole exercise, one of my goals was to avoid any sort of third-party relay server or cloud service in the communication between the USG Pro 4 and UDM. However, after exhausting all possibilities within my limited networking knowledge, I regretfully started to look at options involving third-party services. <a href="https://www.vpn.net/'>Hamachi, Tailscale, and ZeroTier appeared to be popular, with the most common use-cases tending to be connection between individual systems. Of these three, Tailscale and ZeroTier had multiple write-ups and guides, with some specifically talking of site-to-site setups involving Ubiquiti gear. Armed with these guides, I took the plunge into ZeroTier first.

Activating ZeroTier - A Virtual SDN

Software-defined networking matured and WireGuard emerged as a credible lightweight VPN protocol in the latter half of the 2010s. Around that time, some companies like TailScale and ZeroTier developed the concept of cloud-managed mesh VPNs on top of WireGuard - primarily with the aim of making it simple to securely connect systems with each other irrespective of their location. It turned out that such solutions had plenty of use-cases - from connecting people on the go with systems back home to LAN parties for gamers with folks spread around the world. ZeroTier can obviously make a better business case for themselves. While the protocol and client software themselves are open-source, it does need a hosting solution to enable relay connections and / or act as coordinators for direct connections between different clients. While most casual users end up using ZeroTier's global network with the free plan (up to 25 nodes), commercial users can sign up for plans allowing more nodes along with other business-centric features. Enterprises can also obtain a license to self-host the entire infrastructure for additional security.

ZeroTier clients are available for a wide variety of platforms. In order to trial out the solution, I started off by signing up and creating a network with two nodes / members. The gallery below gives a quick overview of the relevant features.

Unlike Tailscale which relies on other identity providers like Google, ZeroTier allows sign-ups for users wishing to become a ZeroTier network administrator. There is an optional added layer of protection in the form of multi-factor authentication delivered by Google Authenticator. Upon logging in, the user is presented with the option to create a new network. Each network is allocated a 128-bit ID (16 hex digits). Administrators can configure the network name and also supply a description. Networks can be private or public, with the former requiring administrators to approve the joining of each and every node.

Each ZeroTier network operates on its own subnet - and users have the freedom to choose something that doesn't conflict with their existing subnets. Auto-assignment is also available. In my case, the USG Pro 4 side was replete with clients using the 192.168.0.0/16 subnets, while the UDM side was using 172.16.0.0/16. I opted to go with 10.242.0.0/16 for the ZeroTier network. After configuring these aspects, the focus shifted to the client machines.

After the installation of the ZeroTier One package on a Windows machine on the USG Pro 4 side and a Rocky Linux machine on the UDM side, each node came up with a 10-digit ID that was first authorized by manually adding it via the management web interface (as shown in the gallery). The ZeroTier One CLI was then used on the nodes to make them join the network using its 128-bit identifier.

While the two nodes were able to communicate via the ZeroTier tunnel, I found that pings were prone to packet loss, with around 30 - 40% of the packets getting dropped. The experience with text-based access like SSH was passable, but video streaming was obviously a no-go. A quick perusal of the documentation indicated that a double NAT would cause drastic drop in the user experience - this made sense, given that the machine behind the UDM was behind such a configuration - the CGNAT followed by the NAT in the UDM itself.

Under these circumstances, I had no option but to install the ZeroTier One packages on the gateways themselves. Given the desire to run a site-to-site VPN, this was always going to be on the cards. The install process was further complicated by the MIPS64-based USG Pro 4 on one end and the AArch64-based UDM on the other.

ZeroTier on the USG Pro 4

ZeroTier maintains official packages for the MIPS64-based Ubiquiti products. Though the suggested steps did not work directly, some helpful pointers in the Github bug report resulted in a successful installation.

After joining the network, it became possible to determine the details of the ZeroTier tunnel - in terms of the peers and the connection method to each of them. In the above screenshot, we see direct connections to all the 'planets' - these are servers maintained by ZeroTier to coordinate the connection between the different members of the network. The 10-digit node ID of each member helps in associating the correct machine with the connection status. A direct connection is the ideal scenario, where the data does not flow through a relay server.

ZeroTier on the UDM

After a number of experiments, I determined that it was better to use a containerized version of ZeroTier on the UDM rather than ZeroTier's official ARM64 package. This installation was a two-step process, starting with the 'on-boot-script' from the community-maintained unifios-utilities, followed by starting the zerotier container using the instructions here.

Steps performed with the USG Pro 4 installation were repeated here with the minor modification to call the executable inside the running container. Cross-referencing the node carrying the RELAY tag in the above screenshot with the members list in the gallery, we find that it is one of the machines on the USG Pro 4's side. On the contrary, the machine behind the UDM (on the last line) gets a direct connection.

Usage Experience

Once the two gateways were set up to communicate via a ZeroTier tunnel, setting up the site-to-site VPN was just a matter of placing the appropriate managed routes (refer to the fifth picture in the gallery). The configured routes are automatically pushed to the members by ZeroTier itself. All nodes in the 172.16.0.0/24 subnet behind the UDM are accessible via 10.242.0.2 (the IP of the ZeroTier interface on the UDM). I had initially configured access to the nodes in the 192.168.0.0/16 subnet via 10.242.0.1 (ZeroTier IP on the USG Pro 4). However, I observed that some reboots ended with the routes configured in the ZeroTier web UI getting allocated to a different interface. That was a puzzling debug - wherein the site-to-site VPN would work between the gateways, but fail when clients on either side were involved. Finally, it turned out to be a ZeroTier bug, albeit one that had already been fixed in a later version (not yet available in either the MIPS64 build or the AArch64 Docker image). In the end, I worked around the issue by creating static routes on the UDM side.

The screenshots with the ZeroTier network information were taken recently after I had the whole configuration stabilized. For more than a month after configuring ZeroTier, I had to put up with periodic disconnects (approximately every 30 minutes), wherein the link would drop on the USG side and not come back up until the tunnel was restarted. I ended up writing a script triggered every couple of minutes to check on tunnel status and scheduling it using the config.gateway.json scheme (as this is not configurable via the web UI).

Restarting the tunnel helped, but got progressively worse. After a month or so, the disconnects were such that the tunnel had to be restarted 8 - 10 times before it would come back up again. Around the same time, I also started evaluating Tailscale for the same purpose. Its IP allotment scheme using IPs designated for CGNAT along with overall opaqueness (compared to ZeroTier) meant that it was a bit more challenging to configure for a site-to-site VPN scenario. However, one aspect that did stand out was that it consistently reported the usage of a relay instead of a direct connection. This got me thinking about the ZeroTier setup - once I had OTT streaming work 'reliably' (save for the periodic disconnects), I had not paid much attention to the relay / direct connection. After delving more into the list of peers reported on either side, it turned out that the connection between the USG Pro 4 and the UDM was indeed getting relayed. A quick update to the firewall entries using the web UI (under WAN_LOCAL) to allow UDP communication over port 9993 helped immediately. After the activation of a direct connection between the USG Pro 4 and the UDM, the disconnects stopped.

The nature of commercial off-the-shelf solutions such as the UniFi lineup is such that most of the community-supplied features (such as the installs of ZeroTier that we have done) are not officially supported. Given the possibility of future firmware updates completely blowing these away, I have opted to add a machine each behind each of the gateways to the ZeroTier network. This should give me some degree of protection in the case of either site remaining unstaffed in such a scenario. It can be seen from the peers list that these nodes connect via relays to the gateways at the other end. Performance might not be great, but it should suffice for accessing the gateways over SSH.

With the site-to-site VPN based on ZeroTier providing enough performance for video streaming without interruptions, the time became ripe to add the finishing touches to the setup. Prior to dealing with those, a few benchmarks are in order.

In the direction that matters - OTT streaming from India to the US - I could see speeds of 20 Mbps+. In the reverse direction, I was limited to around 12 Mbps, likely due to my ancient cable modem that is unable to take advantage of the higher upload speeds that Xfinity / Comcast has started to offer.

Miscellaneous Considerations and Concluding Remarks

The site-to-site VPN setup created with the help of ZeroTier is able to provide easy access to systems in both sites without involving a relay in the middle. Remote management of systems connected to the UDM from the US is straightforward - essentially dealing with something in the local LAN. The need for port forwarding and other overheads is done away with. Performing offsite backups is also simplified. A NAS connected to the UDM is equivalent to a NAS connected to the USG Pro 4 as far as backup jobs go, as all the routing is handled transparently within the USG Pro 4 and the UDM. The two remaining aspects I set out to achieve - seamless utilization of OTT services on either side (particularly, Indian ones on the US side) and transparent extension of the Indian Wi-Fi network in the US were achieved only on a per-device basis (by configuring a proxy). My next step was to address that by creating a wireless network on the USG Pro 4 that would defer to the UDM as the gateway.

Policy-Based Routing on the USG Pro 4

The first step involved the configuration of a new network in the web UI, followed by creating a new Wi-Fi SSID and allocating the new network to it. In my case, the new network was allocated VLAN ID 5. The USG was allowed to provision itself after these changes before moving on to the next step. Using the same config.gateway.json scheme adopted for task scheduling, a new routing table was created with the default route pointing to the ZeroTier IP of the UDM deployed in India.

The next step involved creation of new firewall rules with the modify tag. All VLANs except for the newly created one with ID 5 were relegated to the usage of the main routing table. The new VLAN alone (singled out using the source address subnet) was allocated the newly created routing table. Most importantly, source validation was disabled.

Finally, the new firewall rules were allocated to the newly created VLAN.

Upon provisioning the new configuration, the new Wi-Fi SSID became equivalent to the one on the UDM side. For all practical purposes, clients connecting to that SSID appeared to be accessing the web from an Indian IP address. This method of VPN access also circumvents checks put in by some OTT apps, wherein streaming would get disabled upon the recognition of any VPN profile on the device.

Future Work

The satisfaction of getting the site-to-site VPN working aside, I am not particularly happy that I had to involve a third-party service in the mix. Eventually, I would like to get either manual IPSec or WireGuard working. Both are suppoed to not have trouble with the public WAN IP side acting as a server and the machine behind the CGNAT acting as a peer as long as the required ports are open. However, I have not had much luck (as described in detail in the IPSec section). WireGuard debugging is also complicated by the fact that the USG Pro 4 is a relatively closed system (enabling kernel level debugging of WireGuard using the recommended scheme doesn't work).

Given that I am not a professional network administrator, it is entirely possible that there is a much simpler solution for my requirements here - experienced readers are welcome to provide suggestions in the comments.

Final Words

Ubiquiti's UniFi lineup is immensely popular among prosumers, and enjoys a cult following. It is easy to see why - simple aspects are easy to configure over the web UI, while SSH access is available for the more esoteric things. At the same time, the relationship is a bit frustrating too (the propensity of the gateways to corrupt themselves at the drop of a hat in the event of a power failure, and the UDM being silent about unsuccessful NTP synchronization attempts are cases in point). On balance, I would still recommend Ubiquiti equipment, and upgrades to my primary deployment of the USG Pro 4 and host of Wi-Fi 5 APs are likely to be UniFi-based in the future.

Back in 2017 (when I decided to go with UniFi for my residential installation), the concept of economical managed SDN solutions for prosumers was in its infancy, with Ubiquiti's offerings being the only game in town. Since then, we have seen other vendors step in with their own suite of products - Netgear's Insight-managed business networking products, TP-Link's huge Omada line-up with support for both cloud-based and local management controllers, and QNAP's set of products supporting QuWAN are examples. In light of the increased competition, Ubiquiti would do well to address some of the pain points outlined in this article. In particular, the absence of a guaranteed upgrade path for customized USG Pro 4 installs like the one above to one of the newer AArch64-based gateways is a cause of concern. There are also features taken for granted (like the NTP server functionality in the USG Pro 4) that are not present in the newer gateways. Considering that Ubiquiti has opted for a container-based architecture in the UDM and subsequently released UniFi gateways, they might as well create an official curated app store to enable additional functionality.

Moving forward, I hope to address other interesting aspects of Ubiquiti's lineup of products in this series of articles. Suggestions from readers are welcome.