Category: Bokondini

What Can You Do With 1 Mbps?

The Internet connection in Bokondini is a 1 Mbps satellite connection with a 10:1 contention ratio, which means that actual speeds can get as low as 100 Kbps. We haven’t been to Bok since we turned on the network, but when we were there, we had pretty much unlimited access for the three of us, with a request to use it lightly during school-hours. Even though the connection was a wee bit slower than our 100 Mbps line back home, I grew to really, really appreciate it, and am half inclined to think it’s the perfect amount of Internet. When I talk about our work in Bokondini I often get the question of “what can you do with 1 Mbps today anyways?” so I figured I’d write a post explicitly talking about some of our network experiences.

Older Protocols Work Better

First and foremost, older Internet protocols were inherently designed for slower speeds and less-reliable network conditions – so it’s no surprise they tend to work better. Somewhat interestingly, this split maps almost exactly to whether the protocol is designed to be run in the browser, or exists as a standalone application. Load up OSX’s “Mail” app that uses SMTP? No problem at all. Try to load Gmail in my browser? Not a chance!

On this note, a surprising culprit here was Slack. I naively assumed that the Slack app on my computer used XMPP, or any one of those classic standardized chat applications – but I was dead wrong. Turns out, Slack is Web-based, not that resilient, and surprisingly network-intensive for just a chat application. After realizing that I was literally unable to connect to Slack, I switched over all my correspondence to email, which went much, much smoother.

Web Interactivity Is Non-Existent

Our position behind a satellite link means that every round-trip takes a LOT longer than normal. This is a major killer for the new trend in modern (web 2.0? 3.0?) web-applications to (1) quickly load a shell of a page and (2) dynamically request the rest of the page as you interact with it. The biggest offenders of this practice that I noticed are articles that only load the next paragraph when you scroll down, rather than just fetching the whole page (all 20 kb of text, really?) and letting your browser handle the rendering.

Text-Only Webpages Are Awesome!

On the opposite end of the spectrum, we discovered certain news organizations provide stripped-down and text-only versions of their sites! If you’re curious, check out https://text.npr.org and https://lite.cnn.com for some great examples. It made me immensely happy to see content providers showing awareness of constrained-bandwidth scenarios, and is also worth mentioning that (to my understanding) these types of websites are much more friendly and compatible with accessibility tools (such as text-to-speech readers). We had to kinda “guess around” and use google to find some of these sites, but I’d love to see this practice become more widespread and/or standardized. It brings to mind how mobile website encoding started with the “m.” subdomain (still used by Wikipedia and pretty much no one else?) and then evolving into the responsive designs in use today… it would be fascinating to see backhaul-constrained access go the same way, maybe loading bare-bones content first and keying the responsivity off of network performance metrics.

CDNs Matter A Lot

Almost all satellite networks today are “bent-pipes,” meaning that the satellite doesn’t do any routing/switching and simply re-transmits our signal down to a predetermined ground station. In our case, that station was located in Australia, which means that websites with a CDN presence in Australia performed much, much better than US-based websites. Special shout-out to YouTube: I seriously didn’t expect video-streaming to work at all, but our YouTube tests came out remarkably usable, with relatively minimal buffering.

Interruptions are Interrupted

My favorite thing, by far, about working in Bokondini was that it felt like the Internet gave me everything I needed, but nothing I didn’t. The majority of my online work-tools (i.e. email and git) worked okay, if a little bit slower than usual, and my local work tools (text and code editors) were obviously fine. Even Google Docs worked great – I’d set it to offline mode, type my heart out, and then just get online to sync the document every so often.

Most general-use Web pages loaded fast enough to not be obnoxious, but slow enough to require my intent and sustained interest (relevant XKCD). Say I was writing an article and needed to lookup a statistic? Go to Google, search for it, wait a little bit, open the website, wait a little bit, not a problem at all. Unconsciously open a tab and surf over to Facebook to kill time? It loaded slowly enough to allow me some time to question/reflect on what I’m doing (really just wasting time), and I’d often end up closing the tab before it loaded. Turns out, the Internet’s a whole lot less of a distraction-machine when you make the instant gratification even slightly less instant!

1 Mbps For All?

Overall, I was remarkably impressed by how useful 1 Mbps Internet was, even in the moden Web ecosystem. Lately, I’ve been considering conducting another 1 Mbps experiment here in Seattle, to see if my user behavior comes out the same, or if the experience is changed by being geographically closer to common websites (or just being located in a highly connected city). Especially if it ends up helping me ignore my typical time-wasting sites, maybe it’ll turn out that more is less, less is more, and I end up preferring 1 Mbps to my 100!

Taking this idea even further, my colleague and friend Steve Song wrote a great post a while ago about the hypothetical economics of just providing free 2G cellular services (voice and SMS) to everyone. It’s pretty hard to top the social utility provided by voice and text services, but as more and more voice traffic turns into VoIP, I’ve started thinking and wondering about the Internet equivalent. Maybe ”1 Mbps for all” is really what we want to shoot for…

Bokondini Initial Roll-Out

All right! Where I left off last time, we had just built the network, turned it off, and headed back to the States (just in time for another round of pitches and conferences). Once back in Seattle, we expected a relatively quick roll-out, but ended up having to pump the brakes while some regulatory questions got sorted out (and David returned to Papua from the States).

To my understanding, the terms of our license require that we not compete with any existing commercial telecom operator. In Bokondini, there exists a single operator, Telkomsel, who offers 2G coverage only. While the law clearly stated that we couldn’t compete with Telkomsel, “compete” was less clearly defined. Are we another cellular network? Yes. Do we offer 2G coverage? No. Do we offer telecommunications services (i.e. voice or text)? No. Could people use our network for telecommunications services? Via WhatsApp, absolutely. This set of questions ended up being our initial foray into the much larger existential question of “are we a telco or an ISP?” Or, more succinctly, “what the hell type of service do we provide?”

Initial Network Roll-Out:

On October 18, David sent us a WhatsApp message telling us that he had turned the network on in Bokondini and that everything looked good! Turns out, the system was already working as intended: he had traveled to Bokondini, powered everything on, inserted a SIM card, used the network to video-chat with his father in Florida for a hour, and then thought to give us a message. Two days later, we received word that he had distributed 10 SIM cards into the network and sold the first of many data packages to Fadly for resale. Without even realizing it, we were now a live business generating revenue!

Bugs, Bugs, and More Bugs:

I wish I could tell you the story ended there, but I’d be lying. As soon as we added our first ten users, the system started crashing, sometimes as often as every thirty seconds! This was, as you might expect, not the intended system behavior. After a week of frantic work, I was able to pin the issue down to a specific user’s phone, and stabilized the network by sending word to Bokondini kindly asking this user to please keep their phone off until further notice. Another two weeks of analyzing log files eventually revealed the culprit: a single incorrectly parsed header field, deep in our code, that was crashing our system every time the phone tried to join the network. A one-line fix and we were back in business… until we added another ten users and everything started crashing again! We stayed in this holding pattern for approximately two to three months: add ten users, brace for more issues, frantically fix them, take a day off, add ten more users. A grueling process, to be sure, but a necessary step on our march towards a stable LTE platform.

Interestingly, the majority of the bugs we flushed out had to do primarily with diversity in handsets. Phone manufacturers vary wildly in how (or if) they support certain fields and options, and different regions tend to see different manufacturers (For example, our customers predominantly use Oppo and Xiaomi phones, neither of which market their products in the United States). To make even more complex, LTE provides a large number of optional header fields and many different auth/ID workflows, which ends up creating a very wide range of corner-cases for our EPC to support.

End-of-the-Year Recap and 2019 Goals:

Though I didn’t know if we’d make it this far (or if I’d ever stop coding)… I’m proud to report that by the time December hit, we finally had stable, useful, functional LTE network running in Bokondini. I very seldom get bug reports now – and when I do, they’re predominantly fixed by something simple/stupid on my part. Most recently, I accidentally filled up the whole disk by enabling some basic logging tools to see how much traffic was being sent locally.

Moving forward into 2019, we’ve got a wide range of things we want to do, features we want to build, and research questions that we can finally start asking and answering. Now that the network stays running, our top engineering goals are code-cleaning, a couple feature-adds, and streamlining the deployment and configuration process, ideally to the point where a non-expert can download the project and get started without having to coordinate with us.

Research questions and curiosities range from the technical (how congested is the satellite link, how many users can we support?) to the economic (is this network profitable, how is it affecting the development of this community?) to the social (what are our users doing on the network?) to the personal (how much more fieldwork can I put into this project before my girlfriend dumps me?)

Obviously, these questions heavily intersect and interplay with each other. If we better understand what our users are doing on the network (social), we can build specific tools to support those actions (technical) which will undoubtably impact the overall usefulness of the network (economic). We’re hoping to really start digging into some of this research soon, especially with regards to using locally-hosted services as a way to relieve network congestion. As always, thanks for reading, and stay tuned for more!

Our Time In Bokondini

First and foremost: a sincere apology to everyone following along! After we left for Papua, everything picked up a lot of steam and was a bit uncertain, and the pace didn’t let up until the end of the year. It’s really a shame I didn’t live-blog the process, but in the next posts I’ll try to re-cap the back half of the year. In 2019, I’m hoping to do better, especially since so many people have reached out to me about the blog asking to know how it went!

Getting To Bokondini

If I had to describe Papua in a single word, it would have to be “remote.” It took our team four separate flights over 3-4 days to arrive in Wamena, the major city with an airport in the Baliem Valley region. From there, we met up with Ibel and Helga, who helped us buy our necessary foodstuffs (food for three people for three weeks is no joke), navigate a somewhat convoluted visa process, and figure out the way to Bokondini – a three hour truck ride through muddy off-road terrain, including a river crossing. Making all of these connections while lugging around approximately 150 pounds of telecom equipment was a comical process, to say the least, but we (and the gear) made it safe and sound to Bokondini without any incidents to speak of.

In Bokondini, we stayed in a beautiful house owned by Scotty and Heidi Wisley. The Wisleys are a missionary couple who’ve been working in Bokondini for the past ten years, building out community leadership, infrastructure, and a local school. As it turns out, Scotty and Heidi were in Malaysia at a conference when we were in town, and they were gracious enough to lend us the use of their home during their absence.

Context Dictates Constraints

Even having spent a lot of time in various off-grid locations, I was particularly impressed by the infrastructural situation in Bokondini: a wild, anachronistic, perfect mix of old and new technologies, high-tech solutions paired with rural ingenuity and know-how.

Power:
Power to the village is provided by a combination of a solar array mounted on the schoolhouse roof and a small hydroelectric generator hooked up to a nearby stream, both inputs are wired to a bank of car-batteries. From the batteries, 12VDC is converted to 220VAC via a charge controller, and distributed throughout the village via wires hung from house to house in what can best be described as a nano-grid.

IMG_1315.JPG
Bokondini’s electrical system

For purposes of power cleaning (as well as ensuring significant power for essential functions), the power is turned off daily from approximately 9pm to 6am. Given that the solar panels are already paid for and owned entirely by the community, the system works without any meters or billing – though during times with lower power generation, the news spreads through the community, and members are informally asked to conserve power (i.e. be sparing with their use of stoves and hot water).

 

Water:
There seemed to be no water infrastructure at all, micro or otherwise. Scotty and Heidi’s house is completely self-sufficient, and the taps draw unfiltered water from two main sources: a nearby river, or a rainwater catchment system. We then either boiled the water or ran it through a large, ten-liter Katadyn filter for drinking.

IMG_1387.JPG
“Just hand them up, it’ll be fine”

Internet:
When we arrived, Bokondini already had very limited Internet access, via a 1Mbps satellite link. This link came into the main “technical room” above the schoolhouse, at which point it was turned into a WiFi hotspot, but restricted to the teachers in the community to do educationally things only. Informally, we were told that one of the main drivers of network traffic was teachers using YouTube as an educational resource – more on this below.

IMG_1418.JPG
Raising the tower

Mounting The Tower

Once we were settled in, the work began in earnest. In a nutshell, the main effort was to lower a pole-tower, remove the (unused) dish on the top, replace it with our LTE gear and antennas, and secure the pole back the way it was. Once mounted, we would connect our EPC to the gear on the tower, power the whole thing on, and (hopefully) have a small-scale LTE network that would cover the entire community.

Mounting the gear on the tower was no small feat. Thankfully, with help and guidance from Ibel, who knows the infrastructure in Bokondini like the back of his hand, we got the job done, and done well. Twice a fast-moving thunderstorm chased us off the corrugated roof, and even after the work was done, we had to pull the tower down to replace a malfunctioning eNodeB… but ten days after we arrived, we finally raised the tower with our gear for the last time.

Meanwhile, when we weren’t climbing up on rooftops, we were hard at work configuring the EPC to stay online across a wide range of unorthodox (yet not uncommon) situations. Despite having verified our EPC’s functionality in a controlled lab setting, keeping it alive out in Bokondini required a substantial amount of in-the-field programming. The considerations ranged from the obvious (make sure the box turns itself back on after a power failure) to the subtle (ensuring that if one component crashes, all the correct other components reboot so that the network recovers into a functional and useful state). All in all, the process made for a fascinating crash-course in Keeping Systems Alive, though I’d be lying if I said I enjoyed the course while it was happening.

IMG_1453.JPG
A job well done!

It’s well known that the best way to flush out bugs is to dogfood a product – so during this time, we aggressively dogfooded the network as hard as we could. Connect a phone, walk around town, run a hotspot back at the house all day – all the while, obsessively looking for any network disconnects or stumbles. One of the most noteworthy bugs seemed to be triggered by a specific phone in the community that tried to connect to our network (and subsequently crashed it) every day, once a day, around 3pm. While the fix was anticlimactic (a single incorrectly-parsed header field), the workflow was hilarious: dig through the logs, make a guess, try to write a fix… and then wait for 3pm the next day to see if it worked.

Finally, on our second-to-last day, we had a stable network: No crashes, no bugs, no stumbles: just reliable Internet, from lights-on to lights-off. At this point, we hadn’t yet distributed any SIM cards for a number of reasons, both regulatory and social. We turned off the network to save power, left some instructions for turning it back on (just plug it in), and hit the road, for another three day’s worth of flights back to Seattle – cautiously optimistic about the impending network launch, but without much of a clue what to expect.

Okay, that’s a wrap for now. In Part 2 I’ll cover the initial network launch, the process of bringing additional phones/users online, and some of our initial results of how the network’s doing. Spoiler alert: It’s going really, really well! 🙂

Money Matters (Part 2)

This post is Part 2 in a series about money and billing. Part 1 talked about the technical system we built, and how users purchase data and transfer credit. In this post, I’m going to talk about the basic economics behind our network, including what our expenses look like, what kind of revenue we need to generate, and some different billing models and their tradeoffs.

What Are Our Operating Expenses?

Our network’s recurring costs are relatively straightforward, and consist of two main components: power and Internet backhaul – both of which are surprisingly flat in our case. With respect to power, this is somewhat obvious: the cost to keep the tower running 24/7 is fixed, and depends only on the wattage of the cell tower and the price per kWh in the community. When it comes to backhaul, I didn’t quite know what to expect, but we ended up finding a 1MB/sec (yep, you read that number right) satellite Internet connection for 300 USD/month (yep, you read that number right, too).

Billing Models

As we covered in Part 1, it’s super easy to figure out how many bytes a user has sent or received, and it’s easy for users to buy and sell packages… but that’s where the easy part stops. This brings us to the real question of “how do we want to charge our users?” Note that I said “how” and not “how much” because even before we talk about rates, there’s a bunch of different ways for us to do this. The two main ways we could do billing is either by the amount of data used, as I described previously, or a flat monthly membership model. However, we could also give users discounts on specific services, provide cheaper rates for larger packages, or bill local traffic differently. There’s a lot of different options here, and they’re super important because billing is our main (only) tool to incentivize and influence user behavior.

Incentivizing Network Behavior

If you’ve ever split the cost of the Internet between roommates, or shared a cell phone family plan, then you’ve faced the exact problem we’re up against. Should we just split the cost evenly, or do we make the heavier users pay more? What if someone wants faster speed than the rest of us – do we make them pay for it on their own, or do we split the additional cost because we all benefit? What if someone can’t afford the full price – do we kick them off the network entirely, or take the money that they can afford? Should we offer discount rates for using the Internet during slow times? What’s the actual cost of adding a user to the network, anyway?

The fact that our Internet cost per month is fixed makes these questions much trickier than they might be. If our backhaul was charged per-MB (as a lot of satellite connections do), we’d simply set our rates accordingly, pass the cost onto our users, and be done with it. However, since we’re paying the same monthly rate whether we use the link or not, every second the network isn’t being fully used can be considered a second of service lost. Under this line of thought, if we’re paying for it anyways, let’s use the network as much as possible – and this makes the case for subscription-based pricing that encourages full network utilization.

Unfortunately, the flip size of full utilization is congestion: if too many people want to use the network at any one given moment in time, network performance suffers and slows down for everyone. Inevitably, the best way to reduce usage is to (1) raise rates per MB and (2) bill based on data usage, rather than a flat subscription – even though both of these approaches contradict what I just wrote in the previous paragraph.

The Minimum Viable Network

First, some basic math: Assuming that we split the cost evenly between all users, a network size of ten users would mean each user pays 300 / 10 = 30 USD/month for the satellite connection. If three users decide that they can’t afford it anymore, the cost would rise to 300 / 7 = 42 USD/month, potentially driving other users off the network and raising prices further in what the insurance markets call a “death spiral.” Conversely, the opposite could also occur: if three new users decided that they could afford Internet (at an advertised price of 30 USD/month) the price would drop to 300 / 13 = 23 USD/month and potentially attract even more signups.

This idea can be reduced to what we call the “Minimum Viable Network,” defined as the minimum number of users we need to avoid that death spiral. As long as we’re above the MVN, life is good, and when more people signup we could choose to either give everyone a discount or build up our cash reserves – but once we drop below the MVN, we’ll start to see more drop-offs and a steep price increase that feeds back on itself. This, more than anything else, is the big problem of a fixed cost connection: in a pay-per-MB world, we’d see our costs naturally decrease as users leave the network, but as it stands we’re on the hook for 300 USD/month no matter how many customers we have.

Alternate Funding Models

So far, I’ve mentioned the pay-per-MB model and the monthy subscription model as our two major ways of creating revenue – but there are countless other approaches to keeping the lights on. More and more national level governments are funding/subsidising programs aimed towards connecting rural populations, such as the Universal Service Fund in the United States. Simultaneously, local governments are using taxes to build out telecom infrastructure, simply because it’s the easiest and best way to fund an infrastructural project that comes with a large range of social benefits, including-but-not-limited to education, medicine, commerce, and financial inclusion.

Wrapping Things Up

By now, hopefully I’ve made it clear that this is a challenging space that we’re working in, and provided a useful overview of the tradeoffs. There are no easy answers, and this is probably going to be a topic we explore and experiment with in-depth over our initial deployment. Stay tuned for Part 3, where I’m going to cover some other techniques and approaches that we can build to reduce congestion and cost by keeping things local when possible.

Update: Part 3 is now available.

Ordering SIM Cards

One of the coolest and most interesting parts of this whole project has been the process of designing and purchasing our own SIM cards (1,000 of them, to be precise). Even though the SIM cards are a relatively small part of the whole technical system, they make a great visual, especially with our own design on them. Most importantly, as physical, tangible objects, they really drive home and solidify the idea of “DIY Telecom” like nothing else can.

Where The Heck Do You Get SIM Cards?

On Alibaba.com, apparently! For those of you not in the know, Alibaba is a huge e-commerce site, based mainly in China and geared towards product manufacturers and other parts of the supply chain. It’s where you’d contract with a Chinese producer to buy (normally large) quantities of anything from custom-logo keychains to specialized electronics and machined parts. A quick search for “LTE SIMs” yielded dozens of results, which I eventually pared down to a supplier called GreenCard that appeared to be a small company based out of Shenzhen. GreenCard offered us everything we needed, all for 65 cents a SIM, minimum order size of 1,000 units.

I had never used Alibaba before, and was a little bit nervous about placing such a large order over such a far distance, getting something slightly wrong, and blowing a thousand bucks, but the process was remarkably straightforward and reassuring. The Alibaba website gives you plenty of support, and clearly spells out how to specify things correctly in your order to ensure order satisfaction and CYA in case of a dispute. The vendor I worked with was quite professional, helped me when needed, and delivered a bang-up job all the way through. All in all, the process went incredibly smoothly – the only hitch was communication took longer than I would have liked, but that’s just a time zone issue.

The Visual Design

Given that SIM cards are so tangible and user-facing (they’re literally the first and only object a user gets from their telco), we knew it was worth it to put a lot of thought and effort into the physical design, despite the risk of bikeshedding it. After a couple of brainstorming sessions oriented around themes like “empowerment,” “agency,” “local,” and more, I mocked up about ten different designs on Illustrator and passed them around. We ranked favorites, brainstorrmed some more, fine-tuned the designs, and finally settled on the final design below.

sim card 确认稿 20180228.jpg

The front side is a picture that Kurtis took in Bokondini several years ago, when they set up the initial VBTS network. We wanted to keep the back simple, with the W centered so that even when punched out, our cards would be instantly recognized as the “Purple SIM” or the “W SIM”. Personally, I’m really happy with how these turned out, and am already thinking about a similar design when we expand into future communities: finding a relatively local picture that highlights and emphasizes the community itself, and using that as the main front design, because when it comes down to it, that’s really what this whole project’s about.

Technical Bits

Once we settled on a design, I thought I was almost done… right? Wrong! Turns out there were a ton of other details to take care of before we could place an order: everything from what our Public Land-Mobile Network (PLMN) ID should be to secret keys for security, PINs to lock and unlock the SIMs, even what phone numbers the SIMs should have. We were also able to set the “Network Name” that users see when they connect to our network: though it was tempting to take a page out of WiFi and come up with our best cellular-related puns, we eventually settled on a simple “Bokondini”.

 

%d bloggers like this: