Category: CoLTE

Setting Up A Community LTE Network in Oaxaca, Mexico

Despite the fact that four billion users are on the Internet, three billion people in rural and developing areas still have no access to the Internet [1]. Due to the lower population density and the lack of infrastructure, the return of investment in these areas does not compare to cities. In our lab, the Information and Communication Technology for Development (ICTD) lab at the University of Washington, we expand Internet access by developing a small-scale LTE core network that the communities can deploy and maintain by themselves. The project is called Community LTE Networks, or CoLTE for short.

I have been working on this project for over a year now. During that time, we deployed CoLTE in two places: Bokondini, Indonesia and Oaxaca, Mexico. In January this year, I was a part of an awesome team of three that deployed the network in Oaxaca. However, unexpected scenarios always happen during a field deployment. In this blog post, I would like to share the experience of how we handled those situations. First, I will discuss how CoLTE works, followed by why Oaxaca was our site, how we set up the network, and what we learned from this field trip.

How CoLTE Works

With an Internet connection, a computer to run the EPC, and a base station, we can set up a small LTE network anywhere! The EPC is the network core that manages everything from network subscribers, packet forwarding, to mobility management [3]. At the ICTD lab, we wrote a lot of the code to operate the EPC. In the figure below, you can see that the EPC is connected to the rest of the Internet via a connection on the left side. The  connection links this small town to the rest of the world, and could be a satellite, wireless link, or fiber-optic cable. To the right of the EPC is the base station (eNodeB). The base station sends out the LTE radio signal, and mobile phones in the network connect to it.

A CoLTE network comprised of an Internet connection, EPC, and eNodeB

Why Oaxaca?

Before jumping into the details of the deployment, I will discuss why we chose to go to Oaxaca. Oaxaca is a state in the southeastern part of Mexico, and is known for its well-preserved culture and Indigenous people [2]. Because the surrounding area is mountainous, each community is separated by natural barriers which make it difficult to build an extensive mobile network infrastructure. This makes the area a great fit for community network research. Telecomunicaciones Indígenas Comunitarias (TIC) is an organization based in Oaxaca City that works to connect Indigenous communities in and around the Oaxaca area. TIC has already deployed several 2G networks, but just recently became more interested in LTE. Thus, our lab came to Oaxaca to collaborate with TIC on this project to deploy an LTE network in a community outside Oaxaca.

How did we set up CoLTE in Oaxaca?

There were a total of three people on this trip: Spencer, Esther, and I. Spencer is a postdoctoral researcher who has been leading this project from the beginning, and Esther is a Ph.D. student in our lab. While I spent only a week in Oaxaca due to academic constraints, Spencer and Esther spent two weeks.

Before we left Washington for Oaxaca, we prepared and configured the equipment, including mini-PCs with the EPC installed. To make travel easier, the eNodeBs (base stations), had been shipped directly to TIC’s office in Oaxaca city before we arrived. One was manufactured by BaiCells, and the other one by Nokia. We expected that we could easily configure them to connect to our EPC at the TIC office in the first few days, then we would go to the site to install them. Unfortunately, things did not go as planned.

Spencer had setup a CoLTE network with a BaiCells eNodeB before, so we thought this should cause us no problems. However, it turned out that the equipment was a different version from the one that Spencer had worked with. Accidentally, we misconfigured the settings of the eNodeB, so we were locked out and no longer had access to the eNodeB settings. The only way to connect to it again was via the optical fiber port. Unfortunately, we did not have an optical fiber cable, so we ordered one from the source that would deliver it to us the fastest.

We did not want to waste our time, so while we waited for the fiber optic to ethernet converter, we attempted to work with the Nokia eNodeB. We followed the instructions in the manual, but it did not start up. We reached out to Nokia support immediately as they were the best people to help, but because we were in different timezones, email was not the best mode of communication. Soon, Spencer set up a phone call, so we could communicate effectively.

As the converter arrived, we agreed that we had a better chance of getting BaiCells eNodeB to work, so we came back to work on it. After two days, we were ready to install the network! Sadly, I had to fly back to Seattle on the next day, so I did not get the opportunity to install the components on site with Spencer and Esther.

Conclusion

Field deployments do not always go as planned. Although we had prepared our EPC in advance and were certain of how to connect to the eNodeB, something unexpected could always come up. Time constraint is a big part of field deployment, so adjust your plan accordingly to what happens and do what you think is the best in that situation. In the next deployment, we would definitely check the devices that we will work with, simulate the set up of the network, and test it in our lab if possible. Although I did not have the opportunity to go to the site, I am really grateful to be a part of this team and this effort to increase the Internet accessibility all around the world.

References

[1] International Telecommunication Union. (2017). ICT Facts and Figures 2017. Geneva, Switzerland

[2] Martin, G. J., Camacho Benavides, C. I., Del Campo García, C. A., Anta Fonseca, S., Chapela Mendoza, F., & González Ortíz, M. A. (2011). Indigenous and community conserved areas in Oaxaca, Mexico. Management of Environmental Quality: An International Journal, 22(2), 250-266.

[3] Sevilla, S., Kosakanchit, P., Johnson, M., & Heimerl, K. (2018, November). Building Community LTE Networks with CoLTE. The Community Network Manual: How to Build the Internet Yourself, 75-102.

Oaxaca

On January 23rd this year, the CoLTE team, this time consisting of myself (Spencer Sevilla), Pat Kosakanchit, and Esther Jang, headed down to Oaxaca, Mexico, to collaborate with the amazing folks at Rhizomatica and Telecomunicaciones Indígenas Comunitarias (TIC). While we were there, we wrestled with equipment, erected a tower, and eventually launched a second pilot network. After some minor stabilization efforts and a sprint to submit a formal academic paper, I’m finally getting around to this write-up of our time down there.

Team and Context:

Led by Peter Bloom, Rhizomatica and TIC have been working in and around Oaxaca for the last eight years, where they provide 2G (voice and text) cellular service to the indigenous communities in the surrounding area. Both organizations are widely considered to be world leaders in the rural connectivity space, and it shows. Everything from their load-out checklists and choice of tools and vehicles to their innate knowledge of when they can wing it and when they can’t, demonstrates the honed expertise of people who know their craft inside and out.

There are many things I like about my job – but my favorite thing, by far, is that it provides me a unique opportunity to see some different corners of the world. Though two weeks does not an expert make, I found Oaxaca to be a particularly beautiful corner. The city of Oaxaca de Juarez is nestled in the highlands of Southern Mexico, at an elevation of approximately 5,000 feet (1,600 meters). Though we were there in late January, the weather was pleasantly hot and dry, with temperatures reaching the mid-eighties (Farenheight) and beautiful, fiery sunsets every evening. Oaxaca sprawls out to fill a valley (the Valle Central), and is bordered by two mountain ranges, the Sierra Madre and Sierra Azul. Oaxaca is known for an active youth scene, and the city certainly feels alive with a vibrant art and food scene, set against an anachronistic backdrop of colonial architecture reclaimed and repurposed by a modern Indigenous rights movement into an empowered, proud, self-actualized culture.

eNode Blues:

One of the biggest takeaways from our work in Papua was that safely transporting eNodeBs is a tremendous pain in the ass! Given that we typically just order commercial eNodeBs from international manufacturers, we concluded that it makes more sense to just have the eNodeBs shipped directly to the deployment location, when possible. This time around, we configured our EPCs in the lab, hand-carried them on the plane flight down, and met up with our friend Peter, who had already received the eNodeBs and was holding them in the office. The initial plan was to install three separate sites and have a bake-off: one site with the exact same BaiCells gear we deployed in Papua, and the other two sites with some new gear from a small Finnish group called Kuha.

While this was a good idea in theory… in practice, it turned out to create some serious complications, and almost derailed the entire trip. The BaiCells eNodeB, which we expected to be the exact same model that we deployed in Papua, turned out to be a slightly-newer generation of this model. This particular generation shipped with a couple of unfamiliar bugs, as well as a redesigned configuration interface that led an unnamed teammate* to accidentally lock the team out of the unit for several days. Much credit and many thanks go to BaiCells’ North American support team: not only did they help us get back into the unit, they even gave me a personal cell phone number to call once they understood that we were deploying their gear on a limited time schedule. From Papua to Oaxaca, I continue to have a very high impression of the BaiCells team, and have nothing but good things to say about both their products and support.

*it was Spencer.

Unfortunately… our team had a less positive experience with the Kuha gear. We lost a lot of time in Oaxaca fighting through one configuration challenge after another, usually with somewhat unintuitive or unhelpful symptoms. Though the Kuha team was friendly and helpful over email, their lack of a North American support center meant that all support work was done from 9 to 5 Helsinki time, forcing me to stay up late and wake up early to try and get one more round of emails before their team would go home for the evening. We eventually got all the issues worked out and finally got their eNodeB to connect to our EPC… two days after we got back home. Deployment of these two radios has yet to be arranged 😦

Network Complications:

A major difference from the situation in Bokondini, one that I hadn’t really thought about and probably could have considered, was that of EPC access. In Bokondini, our backhaul to Wamena was provided by Airwaves Missions, who were thoughtful enough to setup a forwarded port for us, so that we could SSH into the EPC and access it from back home in Seattle. Given how often the EPC crashed, especially in the early days, this was a pretty vital requirement.

In Oaxaca, this was not the case at all: Internet backhaul to the sites is provided by a third-party ISP that is, apparently, hard to get ahold of and doesn’t respond to any requests of this nature. This issue dawned on us two days before our deployment date, at which point it became an immediate and big problem – since it meant that after we deployed the EPC in the field, we’d literally have no way to access it, check on it, or fix any issues that may arise!

Conceptually, the solution to this problem is simple: setup a Virtual Private Network (VPN), host it on a server that’s reachable over the public Internet, and configure the EPC to connect to this VPN every time it boots-up. As long as the EPC is connected to the VPN, any other computer (like my laptop) should be able to join the VPN and then access the EPC. Added bonus, this solution is far more secure than the port-forwarding setup in Bokondini, and also more robust in terms of accessible services. Sounds great, right?

Wrong! Though theoretically simple, in practice this situation was all sorts of screwed-up, and took a surprising amount of pain and ugly code to un-stick. As it turns out, most (all?) VPN clients are somewhat failure-prone, and seem to be built with the assumption that they’re run on a machine being operated by a user. In this context, VPN sessions are naturally short-lived (I log on, I work, I log off) and the consequences of failure are minimal (the VPN disconnects, I manually reconnect it). In our context, these assumptions are wildly off-base, and incredibly problematic: we expect the EPC to be online pretty much indefinitely, and a VPN disconnection is essentially a fatal problem, given that there’s no way for us to tell the EPC to do anything if the VPN fails. To make matters worse, the VPN often failed silently, without any obvious hooks for us to programmatically restart it.

After quickly buying a license for OpenVPN and hosting it on EC2, the team spent a large chunk of time configuring and reconfiguring the OpenVPN client, begging it to stay online for more than seven hours at a time. Lacking any success in convincing commercial software to just do its job, we eventually wrote some cron scripts to (1) restart the VPN every four hours and (2) hard-reboot the EPC every 24 hours, just in case. This solution was remarkably effective, and got us to a workable state… just in time to deploy our code in the field.

An Eventual Deployment:

truck.jpeg
Driving with a tower strapped to the roof

On Day 13 of 14, racing the clock down to the wire, we finally headed out and got a site up! Early in the morning, Esther, myself, and a TIC crew led by Javi loaded up a truck, strapped a tower to the top, and headed out to a small mountain village approximately three hours northwest of Oaxaca. After arriving, we met the town leadership, chatted a bit about what we were building and what it would mean for the community, and then got down to it.

WhatsApp Image 2019-02-07 at 16.33.04.jpeg
Enjoying a laugh during assembly

We spent the rest of the day working under a hot sun – hauling the gear (and the tower) up three flights of stairs, drilling anchor holes in the floor and walls, mounting the radio equipment on the tower, hoisting and securing everything with steel cable guylines, and running power and network cables through the building. After eight hours of this, we finally got everything setup and running just before sundown.

WhatsApp Image 2019-02-07 at 16.34.17.jpeg
Mounting the eNodeBs and hooking them up

Our test-phone connected to the network and got on the Internet, we high-fived all around, packed up the truck, grabbed some well-earned dinner cooked over an open flame, and headed back home, eventually falling into bed slightly after midnight. The next day, that was that: we woke up late, said our goodbyes, packed our bags, and hit the airport.

My Biggest Takeaways:

I’m incredibly proud of the effort our team put in to make things work, and our final success owes more to our persistence and drive than it does any particular innate talent. Despite the repeated curveballs of software and network configuration, and working on an increasingly limited time-schedule, our team relentlessly ground through issues until we found ourselves over the finish line. Both Pat and Esther shone under pressure, diving in and setting up VPNs and IPSec tunnels with minimal background knowledge or prior experience. Watching Pat learn how VPNs work over breakfast, and then having one setup and working by lunch, was a particular stand-out memory 🙂

As the person leading the deployment, well… they say good judgement comes from experience, and experience comes from bad judgement. While some of the curveballs were out of our control, I could have anticipated many of them had I thought through the deployment a bit more. A wide range of naive assumptions on my end could have been cleared up by asking more pointed and explicit questions of our partners, and this knowledge would have enabled me to front-load a large part of our fieldwork in Seattle and use our time in Oaxaca more effectively. On the VPN front, I’m starting some work to stress-test and compare VPNs back here in the lab, with the intent of having a truly bombproof solution built and ready to go for our next deployment. On the eNodeB front, I’ve realized that many hardware companies now use an agile/rolling release cycle, and we cannot/should not expect uniformity between versions. The best solution we’ve come up with for future deployments is every time we order a shipment of eNodeBs to a site, we should have one of them sent to our team in Seattle, so that we can simply catch any surprises ahead of time. This will mean we’ll have to hand-carry a single eNodeB down, which shouldn’t be too much of a problem if the other ones are already on-site.

On the bright side, all’s well that ends well. The network is up and operational, and the TIC team has already started distributing SIM cards for an initial trial. Moving forward, I’m incredibly curious to see how CoLTE impacts the community (initial reports have been good!) and I’m very pleased to get our second site up and operational. Longer-term, it was fantastic to collaborate with the Oaxaca team, and I’m sincerely looking forward to building this relationship and teaming up on more installations in the future. As always, thanks for reading.

 

 

Pat Kosakanchit Won a CRA Honorable Mention!

One of our outstanding undergraduates, Pat Kosakanchit, earned a Honorable Mention from the Computing Research Association’s Outstanding Undergraduate Researcher Awards for her work on CoLTE. Well done, Pat!

https://news.cs.washington.edu/2019/03/08/allen-school-undergraduates-recognized-for-excellence-in-research/

https://cra.org/about/awards/outstanding-undergraduate-researcher-award/#2019

Bokondini Initial Roll-Out

All right! Where I left off last time, we had just built the network, turned it off, and headed back to the States (just in time for another round of pitches and conferences). Once back in Seattle, we expected a relatively quick roll-out, but ended up having to pump the brakes while some regulatory questions got sorted out (and David returned to Papua from the States).

To my understanding, the terms of our license require that we not compete with any existing commercial telecom operator. In Bokondini, there exists a single operator, Telkomsel, who offers 2G coverage only. While the law clearly stated that we couldn’t compete with Telkomsel, “compete” was less clearly defined. Are we another cellular network? Yes. Do we offer 2G coverage? No. Do we offer telecommunications services (i.e. voice or text)? No. Could people use our network for telecommunications services? Via WhatsApp, absolutely. This set of questions ended up being our initial foray into the much larger existential question of “are we a telco or an ISP?” Or, more succinctly, “what the hell type of service do we provide?”

Initial Network Roll-Out:

On October 18, David sent us a WhatsApp message telling us that he had turned the network on in Bokondini and that everything looked good! Turns out, the system was already working as intended: he had traveled to Bokondini, powered everything on, inserted a SIM card, used the network to video-chat with his father in Florida for a hour, and then thought to give us a message. Two days later, we received word that he had distributed 10 SIM cards into the network and sold the first of many data packages to Fadly for resale. Without even realizing it, we were now a live business generating revenue!

Bugs, Bugs, and More Bugs:

I wish I could tell you the story ended there, but I’d be lying. As soon as we added our first ten users, the system started crashing, sometimes as often as every thirty seconds! This was, as you might expect, not the intended system behavior. After a week of frantic work, I was able to pin the issue down to a specific user’s phone, and stabilized the network by sending word to Bokondini kindly asking this user to please keep their phone off until further notice. Another two weeks of analyzing log files eventually revealed the culprit: a single incorrectly parsed header field, deep in our code, that was crashing our system every time the phone tried to join the network. A one-line fix and we were back in business… until we added another ten users and everything started crashing again! We stayed in this holding pattern for approximately two to three months: add ten users, brace for more issues, frantically fix them, take a day off, add ten more users. A grueling process, to be sure, but a necessary step on our march towards a stable LTE platform.

Interestingly, the majority of the bugs we flushed out had to do primarily with diversity in handsets. Phone manufacturers vary wildly in how (or if) they support certain fields and options, and different regions tend to see different manufacturers (For example, our customers predominantly use Oppo and Xiaomi phones, neither of which market their products in the United States). To make even more complex, LTE provides a large number of optional header fields and many different auth/ID workflows, which ends up creating a very wide range of corner-cases for our EPC to support.

End-of-the-Year Recap and 2019 Goals:

Though I didn’t know if we’d make it this far (or if I’d ever stop coding)… I’m proud to report that by the time December hit, we finally had stable, useful, functional LTE network running in Bokondini. I very seldom get bug reports now – and when I do, they’re predominantly fixed by something simple/stupid on my part. Most recently, I accidentally filled up the whole disk by enabling some basic logging tools to see how much traffic was being sent locally.

Moving forward into 2019, we’ve got a wide range of things we want to do, features we want to build, and research questions that we can finally start asking and answering. Now that the network stays running, our top engineering goals are code-cleaning, a couple feature-adds, and streamlining the deployment and configuration process, ideally to the point where a non-expert can download the project and get started without having to coordinate with us.

Research questions and curiosities range from the technical (how congested is the satellite link, how many users can we support?) to the economic (is this network profitable, how is it affecting the development of this community?) to the social (what are our users doing on the network?) to the personal (how much more fieldwork can I put into this project before my girlfriend dumps me?)

Obviously, these questions heavily intersect and interplay with each other. If we better understand what our users are doing on the network (social), we can build specific tools to support those actions (technical) which will undoubtably impact the overall usefulness of the network (economic). We’re hoping to really start digging into some of this research soon, especially with regards to using locally-hosted services as a way to relieve network congestion. As always, thanks for reading, and stay tuned for more!

Our Time In Bokondini

First and foremost: a sincere apology to everyone following along! After we left for Papua, everything picked up a lot of steam and was a bit uncertain, and the pace didn’t let up until the end of the year. It’s really a shame I didn’t live-blog the process, but in the next posts I’ll try to re-cap the back half of the year. In 2019, I’m hoping to do better, especially since so many people have reached out to me about the blog asking to know how it went!

Getting To Bokondini

If I had to describe Papua in a single word, it would have to be “remote.” It took our team four separate flights over 3-4 days to arrive in Wamena, the major city with an airport in the Baliem Valley region. From there, we met up with Ibel and Helga, who helped us buy our necessary foodstuffs (food for three people for three weeks is no joke), navigate a somewhat convoluted visa process, and figure out the way to Bokondini – a three hour truck ride through muddy off-road terrain, including a river crossing. Making all of these connections while lugging around approximately 150 pounds of telecom equipment was a comical process, to say the least, but we (and the gear) made it safe and sound to Bokondini without any incidents to speak of.

In Bokondini, we stayed in a beautiful house owned by Scotty and Heidi Wisley. The Wisleys are a missionary couple who’ve been working in Bokondini for the past ten years, building out community leadership, infrastructure, and a local school. As it turns out, Scotty and Heidi were in Malaysia at a conference when we were in town, and they were gracious enough to lend us the use of their home during their absence.

Context Dictates Constraints

Even having spent a lot of time in various off-grid locations, I was particularly impressed by the infrastructural situation in Bokondini: a wild, anachronistic, perfect mix of old and new technologies, high-tech solutions paired with rural ingenuity and know-how.

Power:
Power to the village is provided by a combination of a solar array mounted on the schoolhouse roof and a small hydroelectric generator hooked up to a nearby stream, both inputs are wired to a bank of car-batteries. From the batteries, 12VDC is converted to 220VAC via a charge controller, and distributed throughout the village via wires hung from house to house in what can best be described as a nano-grid.

IMG_1315.JPG
Bokondini’s electrical system

For purposes of power cleaning (as well as ensuring significant power for essential functions), the power is turned off daily from approximately 9pm to 6am. Given that the solar panels are already paid for and owned entirely by the community, the system works without any meters or billing – though during times with lower power generation, the news spreads through the community, and members are informally asked to conserve power (i.e. be sparing with their use of stoves and hot water).

 

Water:
There seemed to be no water infrastructure at all, micro or otherwise. Scotty and Heidi’s house is completely self-sufficient, and the taps draw unfiltered water from two main sources: a nearby river, or a rainwater catchment system. We then either boiled the water or ran it through a large, ten-liter Katadyn filter for drinking.

IMG_1387.JPG
“Just hand them up, it’ll be fine”

Internet:
When we arrived, Bokondini already had very limited Internet access, via a 1Mbps satellite link. This link came into the main “technical room” above the schoolhouse, at which point it was turned into a WiFi hotspot, but restricted to the teachers in the community to do educationally things only. Informally, we were told that one of the main drivers of network traffic was teachers using YouTube as an educational resource – more on this below.

IMG_1418.JPG
Raising the tower

Mounting The Tower

Once we were settled in, the work began in earnest. In a nutshell, the main effort was to lower a pole-tower, remove the (unused) dish on the top, replace it with our LTE gear and antennas, and secure the pole back the way it was. Once mounted, we would connect our EPC to the gear on the tower, power the whole thing on, and (hopefully) have a small-scale LTE network that would cover the entire community.

Mounting the gear on the tower was no small feat. Thankfully, with help and guidance from Ibel, who knows the infrastructure in Bokondini like the back of his hand, we got the job done, and done well. Twice a fast-moving thunderstorm chased us off the corrugated roof, and even after the work was done, we had to pull the tower down to replace a malfunctioning eNodeB… but ten days after we arrived, we finally raised the tower with our gear for the last time.

Meanwhile, when we weren’t climbing up on rooftops, we were hard at work configuring the EPC to stay online across a wide range of unorthodox (yet not uncommon) situations. Despite having verified our EPC’s functionality in a controlled lab setting, keeping it alive out in Bokondini required a substantial amount of in-the-field programming. The considerations ranged from the obvious (make sure the box turns itself back on after a power failure) to the subtle (ensuring that if one component crashes, all the correct other components reboot so that the network recovers into a functional and useful state). All in all, the process made for a fascinating crash-course in Keeping Systems Alive, though I’d be lying if I said I enjoyed the course while it was happening.

IMG_1453.JPG
A job well done!

It’s well known that the best way to flush out bugs is to dogfood a product – so during this time, we aggressively dogfooded the network as hard as we could. Connect a phone, walk around town, run a hotspot back at the house all day – all the while, obsessively looking for any network disconnects or stumbles. One of the most noteworthy bugs seemed to be triggered by a specific phone in the community that tried to connect to our network (and subsequently crashed it) every day, once a day, around 3pm. While the fix was anticlimactic (a single incorrectly-parsed header field), the workflow was hilarious: dig through the logs, make a guess, try to write a fix… and then wait for 3pm the next day to see if it worked.

Finally, on our second-to-last day, we had a stable network: No crashes, no bugs, no stumbles: just reliable Internet, from lights-on to lights-off. At this point, we hadn’t yet distributed any SIM cards for a number of reasons, both regulatory and social. We turned off the network to save power, left some instructions for turning it back on (just plug it in), and hit the road, for another three day’s worth of flights back to Seattle – cautiously optimistic about the impending network launch, but without much of a clue what to expect.

Okay, that’s a wrap for now. In Part 2 I’ll cover the initial network launch, the process of bringing additional phones/users online, and some of our initial results of how the network’s doing. Spoiler alert: It’s going really, really well! 🙂

Money Matters (Part 2)

This post is Part 2 in a series about money and billing. Part 1 talked about the technical system we built, and how users purchase data and transfer credit. In this post, I’m going to talk about the basic economics behind our network, including what our expenses look like, what kind of revenue we need to generate, and some different billing models and their tradeoffs.

What Are Our Operating Expenses?

Our network’s recurring costs are relatively straightforward, and consist of two main components: power and Internet backhaul – both of which are surprisingly flat in our case. With respect to power, this is somewhat obvious: the cost to keep the tower running 24/7 is fixed, and depends only on the wattage of the cell tower and the price per kWh in the community. When it comes to backhaul, I didn’t quite know what to expect, but we ended up finding a 1MB/sec (yep, you read that number right) satellite Internet connection for 300 USD/month (yep, you read that number right, too).

Billing Models

As we covered in Part 1, it’s super easy to figure out how many bytes a user has sent or received, and it’s easy for users to buy and sell packages… but that’s where the easy part stops. This brings us to the real question of “how do we want to charge our users?” Note that I said “how” and not “how much” because even before we talk about rates, there’s a bunch of different ways for us to do this. The two main ways we could do billing is either by the amount of data used, as I described previously, or a flat monthly membership model. However, we could also give users discounts on specific services, provide cheaper rates for larger packages, or bill local traffic differently. There’s a lot of different options here, and they’re super important because billing is our main (only) tool to incentivize and influence user behavior.

Incentivizing Network Behavior

If you’ve ever split the cost of the Internet between roommates, or shared a cell phone family plan, then you’ve faced the exact problem we’re up against. Should we just split the cost evenly, or do we make the heavier users pay more? What if someone wants faster speed than the rest of us – do we make them pay for it on their own, or do we split the additional cost because we all benefit? What if someone can’t afford the full price – do we kick them off the network entirely, or take the money that they can afford? Should we offer discount rates for using the Internet during slow times? What’s the actual cost of adding a user to the network, anyway?

The fact that our Internet cost per month is fixed makes these questions much trickier than they might be. If our backhaul was charged per-MB (as a lot of satellite connections do), we’d simply set our rates accordingly, pass the cost onto our users, and be done with it. However, since we’re paying the same monthly rate whether we use the link or not, every second the network isn’t being fully used can be considered a second of service lost. Under this line of thought, if we’re paying for it anyways, let’s use the network as much as possible – and this makes the case for subscription-based pricing that encourages full network utilization.

Unfortunately, the flip size of full utilization is congestion: if too many people want to use the network at any one given moment in time, network performance suffers and slows down for everyone. Inevitably, the best way to reduce usage is to (1) raise rates per MB and (2) bill based on data usage, rather than a flat subscription – even though both of these approaches contradict what I just wrote in the previous paragraph.

The Minimum Viable Network

First, some basic math: Assuming that we split the cost evenly between all users, a network size of ten users would mean each user pays 300 / 10 = 30 USD/month for the satellite connection. If three users decide that they can’t afford it anymore, the cost would rise to 300 / 7 = 42 USD/month, potentially driving other users off the network and raising prices further in what the insurance markets call a “death spiral.” Conversely, the opposite could also occur: if three new users decided that they could afford Internet (at an advertised price of 30 USD/month) the price would drop to 300 / 13 = 23 USD/month and potentially attract even more signups.

This idea can be reduced to what we call the “Minimum Viable Network,” defined as the minimum number of users we need to avoid that death spiral. As long as we’re above the MVN, life is good, and when more people signup we could choose to either give everyone a discount or build up our cash reserves – but once we drop below the MVN, we’ll start to see more drop-offs and a steep price increase that feeds back on itself. This, more than anything else, is the big problem of a fixed cost connection: in a pay-per-MB world, we’d see our costs naturally decrease as users leave the network, but as it stands we’re on the hook for 300 USD/month no matter how many customers we have.

Alternate Funding Models

So far, I’ve mentioned the pay-per-MB model and the monthy subscription model as our two major ways of creating revenue – but there are countless other approaches to keeping the lights on. More and more national level governments are funding/subsidising programs aimed towards connecting rural populations, such as the Universal Service Fund in the United States. Simultaneously, local governments are using taxes to build out telecom infrastructure, simply because it’s the easiest and best way to fund an infrastructural project that comes with a large range of social benefits, including-but-not-limited to education, medicine, commerce, and financial inclusion.

Wrapping Things Up

By now, hopefully I’ve made it clear that this is a challenging space that we’re working in, and provided a useful overview of the tradeoffs. There are no easy answers, and this is probably going to be a topic we explore and experiment with in-depth over our initial deployment. Stay tuned for Part 3, where I’m going to cover some other techniques and approaches that we can build to reduce congestion and cost by keeping things local when possible.

Update: Part 3 is now available.

Ordering SIM Cards

One of the coolest and most interesting parts of this whole project has been the process of designing and purchasing our own SIM cards (1,000 of them, to be precise). Even though the SIM cards are a relatively small part of the whole technical system, they make a great visual, especially with our own design on them. Most importantly, as physical, tangible objects, they really drive home and solidify the idea of “DIY Telecom” like nothing else can.

Where The Heck Do You Get SIM Cards?

On Alibaba.com, apparently! For those of you not in the know, Alibaba is a huge e-commerce site, based mainly in China and geared towards product manufacturers and other parts of the supply chain. It’s where you’d contract with a Chinese producer to buy (normally large) quantities of anything from custom-logo keychains to specialized electronics and machined parts. A quick search for “LTE SIMs” yielded dozens of results, which I eventually pared down to a supplier called GreenCard that appeared to be a small company based out of Shenzhen. GreenCard offered us everything we needed, all for 65 cents a SIM, minimum order size of 1,000 units.

I had never used Alibaba before, and was a little bit nervous about placing such a large order over such a far distance, getting something slightly wrong, and blowing a thousand bucks, but the process was remarkably straightforward and reassuring. The Alibaba website gives you plenty of support, and clearly spells out how to specify things correctly in your order to ensure order satisfaction and CYA in case of a dispute. The vendor I worked with was quite professional, helped me when needed, and delivered a bang-up job all the way through. All in all, the process went incredibly smoothly – the only hitch was communication took longer than I would have liked, but that’s just a time zone issue.

The Visual Design

Given that SIM cards are so tangible and user-facing (they’re literally the first and only object a user gets from their telco), we knew it was worth it to put a lot of thought and effort into the physical design, despite the risk of bikeshedding it. After a couple of brainstorming sessions oriented around themes like “empowerment,” “agency,” “local,” and more, I mocked up about ten different designs on Illustrator and passed them around. We ranked favorites, brainstorrmed some more, fine-tuned the designs, and finally settled on the final design below.

sim card 确认稿 20180228.jpg

The front side is a picture that Kurtis took in Bokondini several years ago, when they set up the initial VBTS network. We wanted to keep the back simple, with the W centered so that even when punched out, our cards would be instantly recognized as the “Purple SIM” or the “W SIM”. Personally, I’m really happy with how these turned out, and am already thinking about a similar design when we expand into future communities: finding a relatively local picture that highlights and emphasizes the community itself, and using that as the main front design, because when it comes down to it, that’s really what this whole project’s about.

Technical Bits

Once we settled on a design, I thought I was almost done… right? Wrong! Turns out there were a ton of other details to take care of before we could place an order: everything from what our Public Land-Mobile Network (PLMN) ID should be to secret keys for security, PINs to lock and unlock the SIMs, even what phone numbers the SIMs should have. We were also able to set the “Network Name” that users see when they connect to our network: though it was tempting to take a page out of WiFi and come up with our best cellular-related puns, we eventually settled on a simple “Bokondini”.

 

Introducing CoLTE!

Hi everyone! My name’s Spencer, I’m a postdoc at the University of Washington, and I’m super excited to introduce the Community LTE Project (CoLTE, pronounced colt-ee). We’re going to use this blog for two main goals. First, we want to keep everyone informed of the project development, major milestones, and community engagement efforts around the Community LTE Project. Second, we’re going to be sharing the story of building and deploying a rural community-run LTE network in Bokondini, Indonesia.

This blog is going to be written pretty casually – our target audience is everyone, regardless of technical savvyness. We want everyone who’s interested in this project to understand and follow what’s going on: whether you’re a friend, web developer, network technician, VC, or community member, if you’re interested in CoLTE, we want you to know about it! In this first post, I’m going to tell you all about this project, what led to it, where it’s at now, and where we’re headed.

What is CoLTE?

The Community LTE Project is a new approach to high-speed, LTE cellular networking that is decentralized, democratic, and community-based. The goal of CoLTE is to empower anyone (and we mean anyone!) to start their own small-scale cellular network, for fun, profit, or both. Install CoLTE, hook it up to a radio, distribute SIM cards, turn the network on, and watch phones connect. Our goal is for the whole process to be as straightforward as hooking up a WiFi router: a little bit technical, but totally doable for a regular person, and hopefully easier than setting the clock on your VCR.

From a more-technical perspective, CoLTE is a set of many different software packages (all open source!) that we’ve integrated into a single platform that works nicely together. Everything from the “guts” of the network (LTE attach, security, mobility, etc.) to network management and billing are all taken care of, and each component can be turned on, turned off, or replaced without disrupting the other parts. We’ll have a more detailed technical breakdown of this very soon, so stay tuned.

What’s the point?

The core goal of the CoLTE project is to empower individuals and communities to connect themselves to each other and the global Internet. Right now, approximately four billion people (slightly over half the world’s population!) don’t have any Internet access at all. This population is primarily located in rural and remote areas, where infrastructure is more difficult and expensive to install and maintain. Combine these increased costs with decreased revenue due to lower population density, and it’s no surprise that large telecom companies and ISPs don’t want to invest.

To make headway on this problem, we have to ensure that these incentives are taken into account. Our stance is that if a community wants Internet access, the best way for them to get it is for them to literally form a local telecom company and connect themselves. This ensures that incentives are aligned, keeps value within the local community, and empowers local actors to develop a relationship with their network infrastructure. To this effect, the best thing that we can do to support this effort is to build tools that help, enable, and empower these communities as they connect themselves.

How are things going right now?

Right now, we’re hard at work on three main thrusts: the network core, the peripheral web services, and the production environment.We’ve still got plenty more work to do, but the main parts are all operational, and we finally got everything working last week. Additionally, we’ve been coordinating with some folks on the ground in Bokondini, and are making plans to stand up our network there in Summer 2018. Over the next couple of weeks, I’ll be writing in greater detail about each of these efforts, and chronicling our day-by-day process as we start bringing extra services online and getting everything ready for production.

How can I get involved?

Rural Communities: We’re currently looking to partner with rural communities that are underserved by traditional Internet providers. Does your community not have Internet access? Does it want Internet access? We want to hear from you! Use our contact page to get in touch.

Network Operators and Developers: Want to play around with our stack? Curious about our network architecture, or think it might be a good fit for you? Want to contribute to the project or get involved? Contact us here, explore our site to learn more, and check out our Github here.

Friends: Drop us a line, spread the word, and stay tuned for more adventures. Thanks for reading, and until next time, here’s a couple of pictures of our day-to-day work.

IMG_20180515_163606074
Our basic testbed on my desk, much cleaner than usual
IMG_20180515_164310540
The commercial-grade cell tower we’re bringing to Bokondini
IMG_20180410_135202429_HDR
Our custom-ordered SIM cards
%d bloggers like this: