Setting Up A Community LTE Network in Oaxaca, Mexico

Despite the fact that four billion users are on the Internet, three billion people in rural and developing areas still have no access to the Internet [1]. Due to the lower population density and the lack of infrastructure, the return of investment in these areas does not compare to cities. In our lab, the Information and Communication Technology for Development (ICTD) lab at the University of Washington, we expand Internet access by developing a small-scale LTE core network that the communities can deploy and maintain by themselves. The project is called Community LTE Networks, or CoLTE for short.

I have been working on this project for over a year now. During that time, we deployed CoLTE in two places: Bokondini, Indonesia and Oaxaca, Mexico. In January this year, I was a part of an awesome team of three that deployed the network in Oaxaca. However, unexpected scenarios always happen during a field deployment. In this blog post, I would like to share the experience of how we handled those situations. First, I will discuss how CoLTE works, followed by why Oaxaca was our site, how we set up the network, and what we learned from this field trip.

How CoLTE Works

With an Internet connection, a computer to run the EPC, and a base station, we can set up a small LTE network anywhere! The EPC is the network core that manages everything from network subscribers, packet forwarding, to mobility management [3]. At the ICTD lab, we wrote a lot of the code to operate the EPC. In the figure below, you can see that the EPC is connected to the rest of the Internet via a connection on the left side. The  connection links this small town to the rest of the world, and could be a satellite, wireless link, or fiber-optic cable. To the right of the EPC is the base station (eNodeB). The base station sends out the LTE radio signal, and mobile phones in the network connect to it.

A CoLTE network comprised of an Internet connection, EPC, and eNodeB

Why Oaxaca?

Before jumping into the details of the deployment, I will discuss why we chose to go to Oaxaca. Oaxaca is a state in the southeastern part of Mexico, and is known for its well-preserved culture and Indigenous people [2]. Because the surrounding area is mountainous, each community is separated by natural barriers which make it difficult to build an extensive mobile network infrastructure. This makes the area a great fit for community network research. Telecomunicaciones Indígenas Comunitarias (TIC) is an organization based in Oaxaca City that works to connect Indigenous communities in and around the Oaxaca area. TIC has already deployed several 2G networks, but just recently became more interested in LTE. Thus, our lab came to Oaxaca to collaborate with TIC on this project to deploy an LTE network in a community outside Oaxaca.

How did we set up CoLTE in Oaxaca?

There were a total of three people on this trip: Spencer, Esther, and I. Spencer is a postdoctoral researcher who has been leading this project from the beginning, and Esther is a Ph.D. student in our lab. While I spent only a week in Oaxaca due to academic constraints, Spencer and Esther spent two weeks.

Before we left Washington for Oaxaca, we prepared and configured the equipment, including mini-PCs with the EPC installed. To make travel easier, the eNodeBs (base stations), had been shipped directly to TIC’s office in Oaxaca city before we arrived. One was manufactured by BaiCells, and the other one by Nokia. We expected that we could easily configure them to connect to our EPC at the TIC office in the first few days, then we would go to the site to install them. Unfortunately, things did not go as planned.

Spencer had setup a CoLTE network with a BaiCells eNodeB before, so we thought this should cause us no problems. However, it turned out that the equipment was a different version from the one that Spencer had worked with. Accidentally, we misconfigured the settings of the eNodeB, so we were locked out and no longer had access to the eNodeB settings. The only way to connect to it again was via the optical fiber port. Unfortunately, we did not have an optical fiber cable, so we ordered one from the source that would deliver it to us the fastest.

We did not want to waste our time, so while we waited for the fiber optic to ethernet converter, we attempted to work with the Nokia eNodeB. We followed the instructions in the manual, but it did not start up. We reached out to Nokia support immediately as they were the best people to help, but because we were in different timezones, email was not the best mode of communication. Soon, Spencer set up a phone call, so we could communicate effectively.

As the converter arrived, we agreed that we had a better chance of getting BaiCells eNodeB to work, so we came back to work on it. After two days, we were ready to install the network! Sadly, I had to fly back to Seattle on the next day, so I did not get the opportunity to install the components on site with Spencer and Esther.

Conclusion

Field deployments do not always go as planned. Although we had prepared our EPC in advance and were certain of how to connect to the eNodeB, something unexpected could always come up. Time constraint is a big part of field deployment, so adjust your plan accordingly to what happens and do what you think is the best in that situation. In the next deployment, we would definitely check the devices that we will work with, simulate the set up of the network, and test it in our lab if possible. Although I did not have the opportunity to go to the site, I am really grateful to be a part of this team and this effort to increase the Internet accessibility all around the world.

References

[1] International Telecommunication Union. (2017). ICT Facts and Figures 2017. Geneva, Switzerland

[2] Martin, G. J., Camacho Benavides, C. I., Del Campo García, C. A., Anta Fonseca, S., Chapela Mendoza, F., & González Ortíz, M. A. (2011). Indigenous and community conserved areas in Oaxaca, Mexico. Management of Environmental Quality: An International Journal, 22(2), 250-266.

[3] Sevilla, S., Kosakanchit, P., Johnson, M., & Heimerl, K. (2018, November). Building Community LTE Networks with CoLTE. The Community Network Manual: How to Build the Internet Yourself, 75-102.

Oaxaca

On January 23rd this year, the CoLTE team, this time consisting of myself (Spencer Sevilla), Pat Kosakanchit, and Esther Jang, headed down to Oaxaca, Mexico, to collaborate with the amazing folks at Rhizomatica and Telecomunicaciones Indígenas Comunitarias (TIC). While we were there, we wrestled with equipment, erected a tower, and eventually launched a second pilot network. After some minor stabilization efforts and a sprint to submit a formal academic paper, I’m finally getting around to this write-up of our time down there.

Team and Context:

Led by Peter Bloom, Rhizomatica and TIC have been working in and around Oaxaca for the last eight years, where they provide 2G (voice and text) cellular service to the indigenous communities in the surrounding area. Both organizations are widely considered to be world leaders in the rural connectivity space, and it shows. Everything from their load-out checklists and choice of tools and vehicles to their innate knowledge of when they can wing it and when they can’t, demonstrates the honed expertise of people who know their craft inside and out.

There are many things I like about my job – but my favorite thing, by far, is that it provides me a unique opportunity to see some different corners of the world. Though two weeks does not an expert make, I found Oaxaca to be a particularly beautiful corner. The city of Oaxaca de Juarez is nestled in the highlands of Southern Mexico, at an elevation of approximately 5,000 feet (1,600 meters). Though we were there in late January, the weather was pleasantly hot and dry, with temperatures reaching the mid-eighties (Farenheight) and beautiful, fiery sunsets every evening. Oaxaca sprawls out to fill a valley (the Valle Central), and is bordered by two mountain ranges, the Sierra Madre and Sierra Azul. Oaxaca is known for an active youth scene, and the city certainly feels alive with a vibrant art and food scene, set against an anachronistic backdrop of colonial architecture reclaimed and repurposed by a modern Indigenous rights movement into an empowered, proud, self-actualized culture.

eNode Blues:

One of the biggest takeaways from our work in Papua was that safely transporting eNodeBs is a tremendous pain in the ass! Given that we typically just order commercial eNodeBs from international manufacturers, we concluded that it makes more sense to just have the eNodeBs shipped directly to the deployment location, when possible. This time around, we configured our EPCs in the lab, hand-carried them on the plane flight down, and met up with our friend Peter, who had already received the eNodeBs and was holding them in the office. The initial plan was to install three separate sites and have a bake-off: one site with the exact same BaiCells gear we deployed in Papua, and the other two sites with some new gear from a small Finnish group called Kuha.

While this was a good idea in theory… in practice, it turned out to create some serious complications, and almost derailed the entire trip. The BaiCells eNodeB, which we expected to be the exact same model that we deployed in Papua, turned out to be a slightly-newer generation of this model. This particular generation shipped with a couple of unfamiliar bugs, as well as a redesigned configuration interface that led an unnamed teammate* to accidentally lock the team out of the unit for several days. Much credit and many thanks go to BaiCells’ North American support team: not only did they help us get back into the unit, they even gave me a personal cell phone number to call once they understood that we were deploying their gear on a limited time schedule. From Papua to Oaxaca, I continue to have a very high impression of the BaiCells team, and have nothing but good things to say about both their products and support.

*it was Spencer.

Unfortunately… our team had a less positive experience with the Kuha gear. We lost a lot of time in Oaxaca fighting through one configuration challenge after another, usually with somewhat unintuitive or unhelpful symptoms. Though the Kuha team was friendly and helpful over email, their lack of a North American support center meant that all support work was done from 9 to 5 Helsinki time, forcing me to stay up late and wake up early to try and get one more round of emails before their team would go home for the evening. We eventually got all the issues worked out and finally got their eNodeB to connect to our EPC… two days after we got back home. Deployment of these two radios has yet to be arranged 😦

Network Complications:

A major difference from the situation in Bokondini, one that I hadn’t really thought about and probably could have considered, was that of EPC access. In Bokondini, our backhaul to Wamena was provided by Airwaves Missions, who were thoughtful enough to setup a forwarded port for us, so that we could SSH into the EPC and access it from back home in Seattle. Given how often the EPC crashed, especially in the early days, this was a pretty vital requirement.

In Oaxaca, this was not the case at all: Internet backhaul to the sites is provided by a third-party ISP that is, apparently, hard to get ahold of and doesn’t respond to any requests of this nature. This issue dawned on us two days before our deployment date, at which point it became an immediate and big problem – since it meant that after we deployed the EPC in the field, we’d literally have no way to access it, check on it, or fix any issues that may arise!

Conceptually, the solution to this problem is simple: setup a Virtual Private Network (VPN), host it on a server that’s reachable over the public Internet, and configure the EPC to connect to this VPN every time it boots-up. As long as the EPC is connected to the VPN, any other computer (like my laptop) should be able to join the VPN and then access the EPC. Added bonus, this solution is far more secure than the port-forwarding setup in Bokondini, and also more robust in terms of accessible services. Sounds great, right?

Wrong! Though theoretically simple, in practice this situation was all sorts of screwed-up, and took a surprising amount of pain and ugly code to un-stick. As it turns out, most (all?) VPN clients are somewhat failure-prone, and seem to be built with the assumption that they’re run on a machine being operated by a user. In this context, VPN sessions are naturally short-lived (I log on, I work, I log off) and the consequences of failure are minimal (the VPN disconnects, I manually reconnect it). In our context, these assumptions are wildly off-base, and incredibly problematic: we expect the EPC to be online pretty much indefinitely, and a VPN disconnection is essentially a fatal problem, given that there’s no way for us to tell the EPC to do anything if the VPN fails. To make matters worse, the VPN often failed silently, without any obvious hooks for us to programmatically restart it.

After quickly buying a license for OpenVPN and hosting it on EC2, the team spent a large chunk of time configuring and reconfiguring the OpenVPN client, begging it to stay online for more than seven hours at a time. Lacking any success in convincing commercial software to just do its job, we eventually wrote some cron scripts to (1) restart the VPN every four hours and (2) hard-reboot the EPC every 24 hours, just in case. This solution was remarkably effective, and got us to a workable state… just in time to deploy our code in the field.

An Eventual Deployment:

truck.jpeg
Driving with a tower strapped to the roof

On Day 13 of 14, racing the clock down to the wire, we finally headed out and got a site up! Early in the morning, Esther, myself, and a TIC crew led by Javi loaded up a truck, strapped a tower to the top, and headed out to a small mountain village approximately three hours northwest of Oaxaca. After arriving, we met the town leadership, chatted a bit about what we were building and what it would mean for the community, and then got down to it.

WhatsApp Image 2019-02-07 at 16.33.04.jpeg
Enjoying a laugh during assembly

We spent the rest of the day working under a hot sun – hauling the gear (and the tower) up three flights of stairs, drilling anchor holes in the floor and walls, mounting the radio equipment on the tower, hoisting and securing everything with steel cable guylines, and running power and network cables through the building. After eight hours of this, we finally got everything setup and running just before sundown.

WhatsApp Image 2019-02-07 at 16.34.17.jpeg
Mounting the eNodeBs and hooking them up

Our test-phone connected to the network and got on the Internet, we high-fived all around, packed up the truck, grabbed some well-earned dinner cooked over an open flame, and headed back home, eventually falling into bed slightly after midnight. The next day, that was that: we woke up late, said our goodbyes, packed our bags, and hit the airport.

My Biggest Takeaways:

I’m incredibly proud of the effort our team put in to make things work, and our final success owes more to our persistence and drive than it does any particular innate talent. Despite the repeated curveballs of software and network configuration, and working on an increasingly limited time-schedule, our team relentlessly ground through issues until we found ourselves over the finish line. Both Pat and Esther shone under pressure, diving in and setting up VPNs and IPSec tunnels with minimal background knowledge or prior experience. Watching Pat learn how VPNs work over breakfast, and then having one setup and working by lunch, was a particular stand-out memory 🙂

As the person leading the deployment, well… they say good judgement comes from experience, and experience comes from bad judgement. While some of the curveballs were out of our control, I could have anticipated many of them had I thought through the deployment a bit more. A wide range of naive assumptions on my end could have been cleared up by asking more pointed and explicit questions of our partners, and this knowledge would have enabled me to front-load a large part of our fieldwork in Seattle and use our time in Oaxaca more effectively. On the VPN front, I’m starting some work to stress-test and compare VPNs back here in the lab, with the intent of having a truly bombproof solution built and ready to go for our next deployment. On the eNodeB front, I’ve realized that many hardware companies now use an agile/rolling release cycle, and we cannot/should not expect uniformity between versions. The best solution we’ve come up with for future deployments is every time we order a shipment of eNodeBs to a site, we should have one of them sent to our team in Seattle, so that we can simply catch any surprises ahead of time. This will mean we’ll have to hand-carry a single eNodeB down, which shouldn’t be too much of a problem if the other ones are already on-site.

On the bright side, all’s well that ends well. The network is up and operational, and the TIC team has already started distributing SIM cards for an initial trial. Moving forward, I’m incredibly curious to see how CoLTE impacts the community (initial reports have been good!) and I’m very pleased to get our second site up and operational. Longer-term, it was fantastic to collaborate with the Oaxaca team, and I’m sincerely looking forward to building this relationship and teaming up on more installations in the future. As always, thanks for reading.