Money Matters (Part 3)

This post is the third in a three-part series about the financial side of things. In Part 1, talked about the systems we built for us to monitor network usage and for our users to buy and sell credit. In Part 2, we covered our operating expenses, and discussed some of the different considerations and models of how to bill users. In this part, I’m going to wrap things up by sharing some of the tricks and approaches that we can use to improve network performance and service for our users, and then providing some alternate funding models.

Local Services

In our network, a huge difference exists (in terms of network performance and congestion as well as cost) between traffic that stays local and traffic that goes out over the satellite link to connect to the Internet. Though our backhaul connection from our tower to the Internet is only 1 Mbps, the fronthaul connection from phones to the tower is much higher – we’ve seen speeds of 75 Mpbs for an individual user, and LTE offers theoretical speeds of up to 150 Mbps. Local communication, like sending a file from one person to another, doesn’t actually use the satellite link at all, and works even if we get cut off from the Internet. This brings up the question of “should we offer a local-only option?”

If we do decide to provide the option of local-only service to users, we also have to figure out what it should cost. There’s one argument that says we should charge them only for power, because that’s the really the only cost service they’re using. On the other hand, there are very good reasons for us to charge them more, and put that money towards the cost of the backhaul connection. Not only does everyone in the community benefit from Internet connectivity (non-paying users often request paying users to perform services for them), but we could potentially offer “lower priority” Internet connectivity when available. Remember – because the backhaul cost is fixed per month, every second we’re not using the Internet is a second that we paid for and simply didn’t use. Moreover, putting more money towards the backhaul could result in cheaper Internet access per user, which in turn incentivizes more users to join the network and creates one of those positive cycles I talked about in Part 2.

At the same time, we’re also actively working to improve what services are “local” by hosting them within the community on our server. If Gmail struggles behind a slow satellite connection, that’s a great reason for us to host a local email server. The exact same thing could be said for higher-bandwidth services, like streaming video with our own media server, or caching web content. I’ll save this discussion for another post, but we packaged CoLTE with many such services, including OpenStreetMaps and a chat server. Going further, improving Internet/Web performance at the network edge is one of my primary research interests, and I hope to pursue this topic at a painful level of detail before too long.

Limited Services

In addition to a local services option, we’ve also considered a “limited services” package that provides Internet access only to a select set of websites/services at a reduced (or free) cost. This package would likely be focused around calling and texting services (e.g. WhatsApp and Skype), but could also include other select websites such as Wikipedia.

The “limited services” idea stems primarily from the observation that many of the most socially useful services are also very low-bandwidth. For example, a standard-quality VoIP call requires only 20 kbps of bandwidth, potentially less under certain conditions, and WhatsApp text messages barely any bandwidth at all (Though these conclusions are intuitive and based largely on observed experience, we’re hoping to formally quantify this traffic soon!) These characteristics stand in stark contrast to other Internet services that provide a relatively low social benefit at a much larger cost of bandwidth, such as video streaming or online gaming. This dichotomy naturally led us to discussions around the idea of “we’re not making any money off of WhatsApp anyways, so why not just give it away for free?” Going even further, I’m almost certain that we could strongly prioritize VoIP traffic such as Skype and WhatsApp, and dramatically improve the social utility of our network, without our general Internet users even noticing the difference.

Thanks For Reading!

A fascinating theme that’s emerged in this series of posts is the intersection of technical engineering, socio-economic engineering, and what’s best for the individual community. One of the major takeaways of this entire series should be that the options are literally endless – especially since when the community is it’s own ISP, their interests are perfectly aligned. As we deploy additional networks in different areas, we’re hoping to build and research literally all of these topics, with the end goal being something as close to a set of “best practices” as possible. As always, thanks for reading, and please comment or reach out if you’ve got any thoughts or questions.

Our Time In Bokondini

First and foremost: a sincere apology to everyone following along! After we left for Papua, everything picked up a lot of steam and was a bit uncertain, and the pace didn’t let up until the end of the year. It’s really a shame I didn’t live-blog the process, but in the next posts I’ll try to re-cap the back half of the year. In 2019, I’m hoping to do better, especially since so many people have reached out to me about the blog asking to know how it went!

Getting To Bokondini

If I had to describe Papua in a single word, it would have to be “remote.” It took our team four separate flights over 3-4 days to arrive in Wamena, the major city with an airport in the Baliem Valley region. From there, we met up with Ibel and Helga, who helped us buy our necessary foodstuffs (food for three people for three weeks is no joke), navigate a somewhat convoluted visa process, and figure out the way to Bokondini – a three hour truck ride through muddy off-road terrain, including a river crossing. Making all of these connections while lugging around approximately 150 pounds of telecom equipment was a comical process, to say the least, but we (and the gear) made it safe and sound to Bokondini without any incidents to speak of.

In Bokondini, we stayed in a beautiful house owned by Scotty and Heidi Wisley. The Wisleys are a missionary couple who’ve been working in Bokondini for the past ten years, building out community leadership, infrastructure, and a local school. As it turns out, Scotty and Heidi were in Malaysia at a conference when we were in town, and they were gracious enough to lend us the use of their home during their absence.

Context Dictates Constraints

Even having spent a lot of time in various off-grid locations, I was particularly impressed by the infrastructural situation in Bokondini: a wild, anachronistic, perfect mix of old and new technologies, high-tech solutions paired with rural ingenuity and know-how.

Power:
Power to the village is provided by a combination of a solar array mounted on the schoolhouse roof and a small hydroelectric generator hooked up to a nearby stream, both inputs are wired to a bank of car-batteries. From the batteries, 12VDC is converted to 220VAC via a charge controller, and distributed throughout the village via wires hung from house to house in what can best be described as a nano-grid.

IMG_1315.JPG
Bokondini’s electrical system

For purposes of power cleaning (as well as ensuring significant power for essential functions), the power is turned off daily from approximately 9pm to 6am. Given that the solar panels are already paid for and owned entirely by the community, the system works without any meters or billing – though during times with lower power generation, the news spreads through the community, and members are informally asked to conserve power (i.e. be sparing with their use of stoves and hot water).

 

Water:
There seemed to be no water infrastructure at all, micro or otherwise. Scotty and Heidi’s house is completely self-sufficient, and the taps draw unfiltered water from two main sources: a nearby river, or a rainwater catchment system. We then either boiled the water or ran it through a large, ten-liter Katadyn filter for drinking.

IMG_1387.JPG
“Just hand them up, it’ll be fine”

Internet:
When we arrived, Bokondini already had very limited Internet access, via a 1Mbps satellite link. This link came into the main “technical room” above the schoolhouse, at which point it was turned into a WiFi hotspot, but restricted to the teachers in the community to do educationally things only. Informally, we were told that one of the main drivers of network traffic was teachers using YouTube as an educational resource – more on this below.

IMG_1418.JPG
Raising the tower

Mounting The Tower

Once we were settled in, the work began in earnest. In a nutshell, the main effort was to lower a pole-tower, remove the (unused) dish on the top, replace it with our LTE gear and antennas, and secure the pole back the way it was. Once mounted, we would connect our EPC to the gear on the tower, power the whole thing on, and (hopefully) have a small-scale LTE network that would cover the entire community.

Mounting the gear on the tower was no small feat. Thankfully, with help and guidance from Ibel, who knows the infrastructure in Bokondini like the back of his hand, we got the job done, and done well. Twice a fast-moving thunderstorm chased us off the corrugated roof, and even after the work was done, we had to pull the tower down to replace a malfunctioning eNodeB… but ten days after we arrived, we finally raised the tower with our gear for the last time.

Meanwhile, when we weren’t climbing up on rooftops, we were hard at work configuring the EPC to stay online across a wide range of unorthodox (yet not uncommon) situations. Despite having verified our EPC’s functionality in a controlled lab setting, keeping it alive out in Bokondini required a substantial amount of in-the-field programming. The considerations ranged from the obvious (make sure the box turns itself back on after a power failure) to the subtle (ensuring that if one component crashes, all the correct other components reboot so that the network recovers into a functional and useful state). All in all, the process made for a fascinating crash-course in Keeping Systems Alive, though I’d be lying if I said I enjoyed the course while it was happening.

IMG_1453.JPG
A job well done!

It’s well known that the best way to flush out bugs is to dogfood a product – so during this time, we aggressively dogfooded the network as hard as we could. Connect a phone, walk around town, run a hotspot back at the house all day – all the while, obsessively looking for any network disconnects or stumbles. One of the most noteworthy bugs seemed to be triggered by a specific phone in the community that tried to connect to our network (and subsequently crashed it) every day, once a day, around 3pm. While the fix was anticlimactic (a single incorrectly-parsed header field), the workflow was hilarious: dig through the logs, make a guess, try to write a fix… and then wait for 3pm the next day to see if it worked.

Finally, on our second-to-last day, we had a stable network: No crashes, no bugs, no stumbles: just reliable Internet, from lights-on to lights-off. At this point, we hadn’t yet distributed any SIM cards for a number of reasons, both regulatory and social. We turned off the network to save power, left some instructions for turning it back on (just plug it in), and hit the road, for another three day’s worth of flights back to Seattle – cautiously optimistic about the impending network launch, but without much of a clue what to expect.

Okay, that’s a wrap for now. In Part 2 I’ll cover the initial network launch, the process of bringing additional phones/users online, and some of our initial results of how the network’s doing. Spoiler alert: It’s going really, really well! 🙂