On January 23rd this year, the CoLTE team, this time consisting of myself (Spencer Sevilla), Pat Kosakanchit, and Esther Jang, headed down to Oaxaca, Mexico, to collaborate with the amazing folks at Rhizomatica and Telecomunicaciones Indígenas Comunitarias (TIC). While we were there, we wrestled with equipment, erected a tower, and eventually launched a second pilot network. After some minor stabilization efforts and a sprint to submit a formal academic paper, I’m finally getting around to this write-up of our time down there.
Team and Context:
Led by Peter Bloom, Rhizomatica and TIC have been working in and around Oaxaca for the last eight years, where they provide 2G (voice and text) cellular service to the indigenous communities in the surrounding area. Both organizations are widely considered to be world leaders in the rural connectivity space, and it shows. Everything from their load-out checklists and choice of tools and vehicles to their innate knowledge of when they can wing it and when they can’t, demonstrates the honed expertise of people who know their craft inside and out.
There are many things I like about my job – but my favorite thing, by far, is that it provides me a unique opportunity to see some different corners of the world. Though two weeks does not an expert make, I found Oaxaca to be a particularly beautiful corner. The city of Oaxaca de Juarez is nestled in the highlands of Southern Mexico, at an elevation of approximately 5,000 feet (1,600 meters). Though we were there in late January, the weather was pleasantly hot and dry, with temperatures reaching the mid-eighties (Farenheight) and beautiful, fiery sunsets every evening. Oaxaca sprawls out to fill a valley (the Valle Central), and is bordered by two mountain ranges, the Sierra Madre and Sierra Azul. Oaxaca is known for an active youth scene, and the city certainly feels alive with a vibrant art and food scene, set against an anachronistic backdrop of colonial architecture reclaimed and repurposed by a modern Indigenous rights movement into an empowered, proud, self-actualized culture.
eNode Blues:
One of the biggest takeaways from our work in Papua was that safely transporting eNodeBs is a tremendous pain in the ass! Given that we typically just order commercial eNodeBs from international manufacturers, we concluded that it makes more sense to just have the eNodeBs shipped directly to the deployment location, when possible. This time around, we configured our EPCs in the lab, hand-carried them on the plane flight down, and met up with our friend Peter, who had already received the eNodeBs and was holding them in the office. The initial plan was to install three separate sites and have a bake-off: one site with the exact same BaiCells gear we deployed in Papua, and the other two sites with some new gear from a small Finnish group called Kuha.
While this was a good idea in theory… in practice, it turned out to create some serious complications, and almost derailed the entire trip. The BaiCells eNodeB, which we expected to be the exact same model that we deployed in Papua, turned out to be a slightly-newer generation of this model. This particular generation shipped with a couple of unfamiliar bugs, as well as a redesigned configuration interface that led an unnamed teammate* to accidentally lock the team out of the unit for several days. Much credit and many thanks go to BaiCells’ North American support team: not only did they help us get back into the unit, they even gave me a personal cell phone number to call once they understood that we were deploying their gear on a limited time schedule. From Papua to Oaxaca, I continue to have a very high impression of the BaiCells team, and have nothing but good things to say about both their products and support.
*it was Spencer.
Unfortunately… our team had a less positive experience with the Kuha gear. We lost a lot of time in Oaxaca fighting through one configuration challenge after another, usually with somewhat unintuitive or unhelpful symptoms. Though the Kuha team was friendly and helpful over email, their lack of a North American support center meant that all support work was done from 9 to 5 Helsinki time, forcing me to stay up late and wake up early to try and get one more round of emails before their team would go home for the evening. We eventually got all the issues worked out and finally got their eNodeB to connect to our EPC… two days after we got back home. Deployment of these two radios has yet to be arranged 😦
Network Complications:
A major difference from the situation in Bokondini, one that I hadn’t really thought about and probably could have considered, was that of EPC access. In Bokondini, our backhaul to Wamena was provided by Airwaves Missions, who were thoughtful enough to setup a forwarded port for us, so that we could SSH into the EPC and access it from back home in Seattle. Given how often the EPC crashed, especially in the early days, this was a pretty vital requirement.
In Oaxaca, this was not the case at all: Internet backhaul to the sites is provided by a third-party ISP that is, apparently, hard to get ahold of and doesn’t respond to any requests of this nature. This issue dawned on us two days before our deployment date, at which point it became an immediate and big problem – since it meant that after we deployed the EPC in the field, we’d literally have no way to access it, check on it, or fix any issues that may arise!
Conceptually, the solution to this problem is simple: setup a Virtual Private Network (VPN), host it on a server that’s reachable over the public Internet, and configure the EPC to connect to this VPN every time it boots-up. As long as the EPC is connected to the VPN, any other computer (like my laptop) should be able to join the VPN and then access the EPC. Added bonus, this solution is far more secure than the port-forwarding setup in Bokondini, and also more robust in terms of accessible services. Sounds great, right?
Wrong! Though theoretically simple, in practice this situation was all sorts of screwed-up, and took a surprising amount of pain and ugly code to un-stick. As it turns out, most (all?) VPN clients are somewhat failure-prone, and seem to be built with the assumption that they’re run on a machine being operated by a user. In this context, VPN sessions are naturally short-lived (I log on, I work, I log off) and the consequences of failure are minimal (the VPN disconnects, I manually reconnect it). In our context, these assumptions are wildly off-base, and incredibly problematic: we expect the EPC to be online pretty much indefinitely, and a VPN disconnection is essentially a fatal problem, given that there’s no way for us to tell the EPC to do anything if the VPN fails. To make matters worse, the VPN often failed silently, without any obvious hooks for us to programmatically restart it.
After quickly buying a license for OpenVPN and hosting it on EC2, the team spent a large chunk of time configuring and reconfiguring the OpenVPN client, begging it to stay online for more than seven hours at a time. Lacking any success in convincing commercial software to just do its job, we eventually wrote some cron scripts to (1) restart the VPN every four hours and (2) hard-reboot the EPC every 24 hours, just in case. This solution was remarkably effective, and got us to a workable state… just in time to deploy our code in the field.
An Eventual Deployment:
On Day 13 of 14, racing the clock down to the wire, we finally headed out and got a site up! Early in the morning, Esther, myself, and a TIC crew led by Javi loaded up a truck, strapped a tower to the top, and headed out to a small mountain village approximately three hours northwest of Oaxaca. After arriving, we met the town leadership, chatted a bit about what we were building and what it would mean for the community, and then got down to it.
We spent the rest of the day working under a hot sun – hauling the gear (and the tower) up three flights of stairs, drilling anchor holes in the floor and walls, mounting the radio equipment on the tower, hoisting and securing everything with steel cable guylines, and running power and network cables through the building. After eight hours of this, we finally got everything setup and running just before sundown.
Our test-phone connected to the network and got on the Internet, we high-fived all around, packed up the truck, grabbed some well-earned dinner cooked over an open flame, and headed back home, eventually falling into bed slightly after midnight. The next day, that was that: we woke up late, said our goodbyes, packed our bags, and hit the airport.
My Biggest Takeaways:
I’m incredibly proud of the effort our team put in to make things work, and our final success owes more to our persistence and drive than it does any particular innate talent. Despite the repeated curveballs of software and network configuration, and working on an increasingly limited time-schedule, our team relentlessly ground through issues until we found ourselves over the finish line. Both Pat and Esther shone under pressure, diving in and setting up VPNs and IPSec tunnels with minimal background knowledge or prior experience. Watching Pat learn how VPNs work over breakfast, and then having one setup and working by lunch, was a particular stand-out memory 🙂
As the person leading the deployment, well… they say good judgement comes from experience, and experience comes from bad judgement. While some of the curveballs were out of our control, I could have anticipated many of them had I thought through the deployment a bit more. A wide range of naive assumptions on my end could have been cleared up by asking more pointed and explicit questions of our partners, and this knowledge would have enabled me to front-load a large part of our fieldwork in Seattle and use our time in Oaxaca more effectively. On the VPN front, I’m starting some work to stress-test and compare VPNs back here in the lab, with the intent of having a truly bombproof solution built and ready to go for our next deployment. On the eNodeB front, I’ve realized that many hardware companies now use an agile/rolling release cycle, and we cannot/should not expect uniformity between versions. The best solution we’ve come up with for future deployments is every time we order a shipment of eNodeBs to a site, we should have one of them sent to our team in Seattle, so that we can simply catch any surprises ahead of time. This will mean we’ll have to hand-carry a single eNodeB down, which shouldn’t be too much of a problem if the other ones are already on-site.
On the bright side, all’s well that ends well. The network is up and operational, and the TIC team has already started distributing SIM cards for an initial trial. Moving forward, I’m incredibly curious to see how CoLTE impacts the community (initial reports have been good!) and I’m very pleased to get our second site up and operational. Longer-term, it was fantastic to collaborate with the Oaxaca team, and I’m sincerely looking forward to building this relationship and teaming up on more installations in the future. As always, thanks for reading.
The Community LTE Project 