What Can You Do With 1 Mbps?

The Internet connection in Bokondini is a 1 Mbps satellite connection with a 10:1 contention ratio, which means that actual speeds can get as low as 100 Kbps. We haven’t been to Bok since we turned on the network, but when we were there, we had pretty much unlimited access for the three of us, with a request to use it lightly during school-hours. Even though the connection was a wee bit slower than our 100 Mbps line back home, I grew to really, really appreciate it, and am half inclined to think it’s the perfect amount of Internet. When I talk about our work in Bokondini I often get the question of “what can you do with 1 Mbps today anyways?” so I figured I’d write a post explicitly talking about some of our network experiences.

Older Protocols Work Better

First and foremost, older Internet protocols were inherently designed for slower speeds and less-reliable network conditions – so it’s no surprise they tend to work better. Somewhat interestingly, this split maps almost exactly to whether the protocol is designed to be run in the browser, or exists as a standalone application. Load up OSX’s “Mail” app that uses SMTP? No problem at all. Try to load Gmail in my browser? Not a chance!

On this note, a surprising culprit here was Slack. I naively assumed that the Slack app on my computer used XMPP, or any one of those classic standardized chat applications – but I was dead wrong. Turns out, Slack is Web-based, not that resilient, and surprisingly network-intensive for just a chat application. After realizing that I was literally unable to connect to Slack, I switched over all my correspondence to email, which went much, much smoother.

Web Interactivity Is Non-Existent

Our position behind a satellite link means that every round-trip takes a LOT longer than normal. This is a major killer for the new trend in modern (web 2.0? 3.0?) web-applications to (1) quickly load a shell of a page and (2) dynamically request the rest of the page as you interact with it. The biggest offenders of this practice that I noticed are articles that only load the next paragraph when you scroll down, rather than just fetching the whole page (all 20 kb of text, really?) and letting your browser handle the rendering.

Text-Only Webpages Are Awesome!

On the opposite end of the spectrum, we discovered certain news organizations provide stripped-down and text-only versions of their sites! If you’re curious, check out https://text.npr.org and https://lite.cnn.com for some great examples. It made me immensely happy to see content providers showing awareness of constrained-bandwidth scenarios, and is also worth mentioning that (to my understanding) these types of websites are much more friendly and compatible with accessibility tools (such as text-to-speech readers). We had to kinda “guess around” and use google to find some of these sites, but I’d love to see this practice become more widespread and/or standardized. It brings to mind how mobile website encoding started with the “m.” subdomain (still used by Wikipedia and pretty much no one else?) and then evolving into the responsive designs in use today… it would be fascinating to see backhaul-constrained access go the same way, maybe loading bare-bones content first and keying the responsivity off of network performance metrics.

CDNs Matter A Lot

Almost all satellite networks today are “bent-pipes,” meaning that the satellite doesn’t do any routing/switching and simply re-transmits our signal down to a predetermined ground station. In our case, that station was located in Australia, which means that websites with a CDN presence in Australia performed much, much better than US-based websites. Special shout-out to YouTube: I seriously didn’t expect video-streaming to work at all, but our YouTube tests came out remarkably usable, with relatively minimal buffering.

Interruptions are Interrupted

My favorite thing, by far, about working in Bokondini was that it felt like the Internet gave me everything I needed, but nothing I didn’t. The majority of my online work-tools (i.e. email and git) worked okay, if a little bit slower than usual, and my local work tools (text and code editors) were obviously fine. Even Google Docs worked great – I’d set it to offline mode, type my heart out, and then just get online to sync the document every so often.

Most general-use Web pages loaded fast enough to not be obnoxious, but slow enough to require my intent and sustained interest (relevant XKCD). Say I was writing an article and needed to lookup a statistic? Go to Google, search for it, wait a little bit, open the website, wait a little bit, not a problem at all. Unconsciously open a tab and surf over to Facebook to kill time? It loaded slowly enough to allow me some time to question/reflect on what I’m doing (really just wasting time), and I’d often end up closing the tab before it loaded. Turns out, the Internet’s a whole lot less of a distraction-machine when you make the instant gratification even slightly less instant!

1 Mbps For All?

Overall, I was remarkably impressed by how useful 1 Mbps Internet was, even in the moden Web ecosystem. Lately, I’ve been considering conducting another 1 Mbps experiment here in Seattle, to see if my user behavior comes out the same, or if the experience is changed by being geographically closer to common websites (or just being located in a highly connected city). Especially if it ends up helping me ignore my typical time-wasting sites, maybe it’ll turn out that more is less, less is more, and I end up preferring 1 Mbps to my 100!

Taking this idea even further, my colleague and friend Steve Song wrote a great post a while ago about the hypothetical economics of just providing free 2G cellular services (voice and SMS) to everyone. It’s pretty hard to top the social utility provided by voice and text services, but as more and more voice traffic turns into VoIP, I’ve started thinking and wondering about the Internet equivalent. Maybe ”1 Mbps for all” is really what we want to shoot for…

Bokondini Initial Roll-Out

All right! Where I left off last time, we had just built the network, turned it off, and headed back to the States (just in time for another round of pitches and conferences). Once back in Seattle, we expected a relatively quick roll-out, but ended up having to pump the brakes while some regulatory questions got sorted out (and David returned to Papua from the States).

To my understanding, the terms of our license require that we not compete with any existing commercial telecom operator. In Bokondini, there exists a single operator, Telkomsel, who offers 2G coverage only. While the law clearly stated that we couldn’t compete with Telkomsel, “compete” was less clearly defined. Are we another cellular network? Yes. Do we offer 2G coverage? No. Do we offer telecommunications services (i.e. voice or text)? No. Could people use our network for telecommunications services? Via WhatsApp, absolutely. This set of questions ended up being our initial foray into the much larger existential question of “are we a telco or an ISP?” Or, more succinctly, “what the hell type of service do we provide?”

Initial Network Roll-Out:

On October 18, David sent us a WhatsApp message telling us that he had turned the network on in Bokondini and that everything looked good! Turns out, the system was already working as intended: he had traveled to Bokondini, powered everything on, inserted a SIM card, used the network to video-chat with his father in Florida for a hour, and then thought to give us a message. Two days later, we received word that he had distributed 10 SIM cards into the network and sold the first of many data packages to Fadly for resale. Without even realizing it, we were now a live business generating revenue!

Bugs, Bugs, and More Bugs:

I wish I could tell you the story ended there, but I’d be lying. As soon as we added our first ten users, the system started crashing, sometimes as often as every thirty seconds! This was, as you might expect, not the intended system behavior. After a week of frantic work, I was able to pin the issue down to a specific user’s phone, and stabilized the network by sending word to Bokondini kindly asking this user to please keep their phone off until further notice. Another two weeks of analyzing log files eventually revealed the culprit: a single incorrectly parsed header field, deep in our code, that was crashing our system every time the phone tried to join the network. A one-line fix and we were back in business… until we added another ten users and everything started crashing again! We stayed in this holding pattern for approximately two to three months: add ten users, brace for more issues, frantically fix them, take a day off, add ten more users. A grueling process, to be sure, but a necessary step on our march towards a stable LTE platform.

Interestingly, the majority of the bugs we flushed out had to do primarily with diversity in handsets. Phone manufacturers vary wildly in how (or if) they support certain fields and options, and different regions tend to see different manufacturers (For example, our customers predominantly use Oppo and Xiaomi phones, neither of which market their products in the United States). To make even more complex, LTE provides a large number of optional header fields and many different auth/ID workflows, which ends up creating a very wide range of corner-cases for our EPC to support.

End-of-the-Year Recap and 2019 Goals:

Though I didn’t know if we’d make it this far (or if I’d ever stop coding)… I’m proud to report that by the time December hit, we finally had stable, useful, functional LTE network running in Bokondini. I very seldom get bug reports now – and when I do, they’re predominantly fixed by something simple/stupid on my part. Most recently, I accidentally filled up the whole disk by enabling some basic logging tools to see how much traffic was being sent locally.

Moving forward into 2019, we’ve got a wide range of things we want to do, features we want to build, and research questions that we can finally start asking and answering. Now that the network stays running, our top engineering goals are code-cleaning, a couple feature-adds, and streamlining the deployment and configuration process, ideally to the point where a non-expert can download the project and get started without having to coordinate with us.

Research questions and curiosities range from the technical (how congested is the satellite link, how many users can we support?) to the economic (is this network profitable, how is it affecting the development of this community?) to the social (what are our users doing on the network?) to the personal (how much more fieldwork can I put into this project before my girlfriend dumps me?)

Obviously, these questions heavily intersect and interplay with each other. If we better understand what our users are doing on the network (social), we can build specific tools to support those actions (technical) which will undoubtably impact the overall usefulness of the network (economic). We’re hoping to really start digging into some of this research soon, especially with regards to using locally-hosted services as a way to relieve network congestion. As always, thanks for reading, and stay tuned for more!