The Nyaruka Blog

We’ve been doing quite a bit of work on RapidSMS as of late, and one of the neat projects we did was to build an interactive form builder for both SMS and simple XForms. You can check out a little video of it in action if you’d like, it’s pretty neat.

Anyways, RapidSMS has finally been moving to coming in a packaged form, and a few people have showed interest in using our XForms app so I thought I’d see what it took to bring it into the Cheese Shop for easy pip install goodness.

So here’s my little recipe for the next time.

Register with PyPi

First things first, you’ll want to go go create an account on PyPi.

Directory Structure

Next up, package structure.  There’s a lot of debates on this, but here’s what I think seems reasonably sane and is how our rapidsms-xforms project is organized:

rapidsms-xforms/
     setup.py          ; our main setup file
     MANIFEST.in       ; definition for extra files to include
     README.rst        ; the README.rst we'll show in PyPi

     docs/             ; our Sphinx docs
     rapidsms_xforms/  ; our Django app code and templates

The three files that are interesting to us here are setup.py, MANIFEST.in, and README.rst. We’ll go into those in detail. You’ll notice that the name of our project on github is actually rapidsms-xforms, while the package name itself is rapidsms_xforms. Converting dashes to underscores seems to be the convention.

Setting up setup.py

Ok, let’s look at our setup.py. Here’s ours:

from setuptools import setup, find_packages

 setup(
    name='rapidsms-xforms',
    version=__import__('rapidsms_xforms').__version__,
    license="BSD",

    install_requires = [
        "rapidsms",
        "django-uni-form"
    ],

    description='Interactive form builder for both XForms and SMS submissions into RapidSMS',
    long_description=open('README.rst').read(),

    author='Nicolas Pottier, Eric Newcomer',
    author_email='code@nyaruka.com',

    url='http://github.com/nyaruka/rapidsms-xforms',
    download_url='http://github.com/nyaruka/rapidsms-xforms/downloads',

    include_package_data=True,

    packages=['rapidsms_xforms'],

    zip_safe=False,
    classifiers=[
        'Development Status :: 4 - Beta',
        'Environment :: Web Environment',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: BSD License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Framework :: Django',
    ]
 )

The interesting bits to pay attention to:

• we’ve added a __version__ field to our packages __init__.py, and we pull the PyPi version from that file.
• we add our dependencies to install_requires, pip will do the rest.
• the code we want to include is in the rapidsms_xforms package, we just include that in the packages list.
• we are going to use the contents of the README.rst file, which is what github shows by default on our repo page, to also be our long description on the PyPi pages.

The rest is pretty much boilerplate, obviously substituting your own packages information.

Adding non .py files

Now as is this will work reasonably well, but we’ll be missing our static and templates subdirectories in our rapidsms_xforms dir. By default setuptools is only concerned with .py files, so we have to specify the rest ourselves. That’s where the MANIFEST.in file comes in:

include README.rst
recursive-include rapidsms_xforms/static *
recursive-include rapidsms_xforms/templates *

Here we specify what other files apart from Python files we want included in our package. We are going to throw in our README.rst, because who doesn’t love those, and then also our static and templates subdirectories.

Testing things out

Ok, let’s test this out by building a source distribution package:

% python setup.py sdist
    .. gobs of output ..
% ls dist
rapidsms-xforms-0.1.0.tar.gz

Hooray, that looks like a success. Before uploading to the Cheese Shop, I’d recommend untarring that file and making sure the contents look like you expect. Better yet, set up a virtual environment, then throw that package in there and test everything out. Repeat as needed until you feel confident it all looks good.

Uploading to PyPi

Alright, time to upload it to the Cheese Shop, here we go:

% python setup.py register sdist upload
running register
running egg_info
  .. gobs of output ..
Submitting dist/rapidsms-xforms-0.1.1.tar.gz to http://pypi.python.org/pypi
Server response (200): OK

If you get a 200 back, then things probably worked, wahoo.

Installing

That’s it. You should now be able to find your package on PyPi. You’ll also be able to install it using pip:

% pip install rapidsms-xforms

Updating

When you need to update, just update your version in your __init__.py, then rerun the upload command, the rest is taken care of for you.

The MoinMoin wiki has to be where documentation goes to die.  I've never worked with a piece of software that has such crummy docs, everything is half done etc..  It makes a strong argument towards never using a Wiki for documentation.

Anyways, here's my recipe to get HTTP Digest authentication working with MoinMoin on apache2.  It wasted some of my time to get it all right, so hopefully this helps someone since the docs won't.

I recently had to put together a wiki for the RapidSMS project and one of our goals was to make the content as similar to Sphinx documentation as possible. The idea was that as certain pages on the Wiki matured, they could easily be rolled into the official Sphinx docs, all without changing the actual format.

MoinMoin 1.9.3 actually supports Restructured Text as a markup, the same format as Sphinx, and it also support Pygments to do syntax highlighting, also the same as Sphinx, but the two didn't play well together: although you could create pages using RST, Pygments would not highlight code segments in them.

So I hacked on MoinMoin a bit to make it work. It isn't the prettiest of patches, but they rarely are. Here's a ReST page with some highlighting:

The next problem was that I wanted a slightly better editor for RST. I've used CodeMirror in the past and always thought it struck a nice balance of performance and functionality. Sadly, there wasn't an RST parser written for CodeMirror, so I threw a very rough one together.

More ugly MoinMoin hacking let me insert it as the default editor. MoinMoin really, really needs to adopt a templating language, it is especially ridiculous for me to edit a .py file to add some javascript. In any case, here's the editor:

You can choose to add just the Pygments highlighting to ReST or to also include the CodeMirror editor, up to you. Grab the gist and check out the README for installation instructions. It is slightly laborious, but mostly just copying files around. Hope you find it useful. You can check out the version we have running on the RapidSMS wiki.

As I've detailed before, our internet here in Rwanda can be a bit flakey.  As such, it is pretty useful having documentation for the libraries we use available when the network is down. (or when it is just really slow, which is often)

I used to use the HTML version of the Sphinx docs for Python and Django, but discovered CHM (Compiled HTML) the other day and it has even better search capabilities.  On OS X the best CHM client I've found is ArchMoch, though there seem to be some files it doesn't get along with.

Anyways, to save you the time of tracking all those CHM files down, here they are hosted off my DropBox:

Python 2.6.2 CHM Docs

Django 1.2RC1 CHM Docs

JQuery CHM Docs

Note that the JQuery CHM doesn't seem to have an index built for it, so searching isn't quite as great for it.

I haven't found a good CSS reference in CHM format, my ideal would be something like the sitepoint or w3 schools references.  Maybe if I get the time I'll compile one later.  But I did find a nice little cheatsheet for CSS:

CSS CheatSheet PDF

Of course, even having these you won't get the full power of Google's amazing answering powers, but it is something.  If you have any to add, let me know and I'll link them up.

A few days ago, some articles appeared in the New Times and the Rwandaise, boasting about how Rwanda has the fastest internet speeds in the region, and the third fastest in Africa, beating even South Africa.  The articles used the recent results posted by Ookla as a reference, which does indeed show Rwanda as 89th in the world, with download speeds of 2.36Mb/s.

Now for anyone in Rwanda, this probably seemed a bit surprising.  I have an unhealthy obsession with bandwidth, so I know first hand that we get nowhere near 2 Mb/s, that much is clear every single day as I go about my work at Nyaruka.

So what explains the discrepancy?  Is Ookla wrong?

The answer lies in how sites like speedtest.net came about and what they are trying to measure.

Ten years ago, DSL and Cable were just starting to make their presence in the United States.  There was fierce competition from various providers on pricing and claimed bandwidth, but few ways for consumers to judge how fast their service really was.  To address that need, sites like dslreports.com and speedtest.net came about.  Their goal was to measure the speed of the DSL and Cable connections.

The way these sites work is quite simple.  Speedtest.net gets various internet providers to host a large file on one of their servers.  Then you and I go and try to download the file, timing how long it takes, this is what is happening when you are measuring the bandwidth using the speedtest client.  They divide the size of the file by how long it takes and you have a rough approximation of your bandwidth speed.

Now in order to accurately measure the speed of only the DSL or Cable portion of the connection, you need to have the shortest path to the test file you download.  Speedtest.net has done a good job of getting these files hosted on various servers across the world.  And one of those places happens to be the Rwandatel offices in Kigali.  This is where the validity of the tests gets into trouble.

A diagram might help explain.

This diagram gives a simple view of what a typical set up might be in America, in this case Atlanta, Georgia, where one of my friends Joe George lives. (more on him later)  The green arrow represents his connection to his internet service provider (ISP).  In his case, he's using cable, which gets about 10Mb/s.  Since his ISP is also a speedtest.net test site, when he tests his connection, he is testing his cable connection only.  Speedtest.net is NOT testing how fast he can access the rest of the internet, only the connection between his computer and his ISP.

That is valid, because his internet provider has a far far faster connection to the internet.  The purple arrow represents that, the connection between the ISP and the rest of the internet.  In Atlanta, that is fiber, probably multiple fiber optic lines from different providers, incredibly fast and reliable.  So the bottleneck for Joe is almost always going to be his connection to the ISP, the green arrow, not his ISP's connection to the internet (the purple arrow).

Ok, so now let's take a look at the same situation in Rwanda.

Rwanda has a pretty great network internally.  We have a fantastic 3G network from three different providers, at reasonable rates, which makes getting a connection quite accessible.  But what we don't have, is a great connection to the internet.

Here, the green arrow represents a typical 3G modem, which tops out about 2.5Mb/s in practice.  That's fast, and you'll notice that's also about the speed that Ookla says our internet is.  The reason for that is that Rwandatel has a server based in Kigali that is a speedtest.net testing site.  So when you go to speedtest.net and pick Kigali as your testing location, you are actually testing the green arrow only, how fast your internet is to the Rwandatel server in Kigali.

That works fine in the case of Joe, where the ISP's connection to the internet is far far faster, but it doesn't work in Rwanda.  Our ISP's connection to the internet is still very slow, either going over satellite or microwave, the purple arrows here.  So although we can reach our ISP very quickly, as soon as we try to reach the rest of the world, the rest of the internet, then our speeds slow down to a crawl.

This is pretty easy to test in practice, you just need to change which site you are testing against when you go to speedtest.net.  Yesterday evening I did just that using my Rwandatel connection and here were the results.

Kigali to Kigali: 2.12 Mb/s

Kigali to Kampala: 0.11 Mb/s

Kigali to Atlanta: 0.09 Mb/s

So we see there that yes, our connection to Rwandatel is indeed very fast, but as soon as we try to leave Kigali, it slows down to a crawl.  That's because the connections from Rwanda to the world are still very, very slow despite our great internal network.

We can validate this by going in the opposite direction.  Remember Joe?  Well I asked him to use his crazy fast internet to test against the same sites, here's what he got:

Atlanta to Atlanta: 10.01 Mb/s

Atlanta to Kampala: 1.15 Mb/s

Atlanta to Kigali: 0.10 Mb/s

Here we see that in Kampala, while their ISP's do not provide quite as quick of an internet connection on average (the green arrow), their connection to the rest of the world, the internet at large, is much faster than Rwanda's.  So although Ookla, which measures the green arrow, says they are slower in some cases, that is actually incorrect in practice.

The silver lining with all this is that backbones are on the way, and once hooked up and given they have sufficient capacity, those claims made by Ookla may actually become true.  Our bottleneck is our provider's connection to the internet, so once that is fixed we will indeed have much, much, faster internet.

You can actually get a taste of that if you use the internet at unusual hours, like 2AM.  At that time of the day, there aren't enough users to saturate the connections Rwanda does have, so the speed is great.  But we need the fiber to be hooked up before we can honestly claim to have fast internet.  That's a day I'm looking forward to, hope it happens soon.

PS. In the interest of verification, here are all the internet speed test results we used for this article

Git submodules are a convenient way to create a dependency on another git project.  This can be useful if you depend on another project and want to make it easy for others to get your system up and running.

I won't cover creating submodules here as that is pretty well convered in the git manual, but updating them is slightly less obvious.  When creating submodule dependencies, git actually saves a reference to the version of the tree at that point in time.  That's a good thing, it prevents things from breaking due to your dependency changing out from under you.  But there are times when you'll want to update that reference.  The trick is to realize that git is keeping a reference to whatever version of the git tree is currently checked out.  So all you need to do is go and pull the submodule, then commit the project as a whole.

1. Make sure that your submodule is already checked out, ie, do 'git submodule init', 'git submodule update' if necessary
2. CD into the root directory for your submodule.
3. Change the branch for the submodule to master: 'git checkout master'
4. Pull the latest version: 'git pull'
5. That's it, now you can go back to your project's root directory, do a 'git status' to see the state of things, add the changed submodule with 'git add' then commit your changes with 'git commit'

That should do it.  I'm still coming to terms whether I like this particular style of dependencies, for those of us with limited bandwidth it does cause rather large repositories to be sent over the wire just to get things running.  Having dependencies be managed via PiPi in Python is certainly the better option if possible.

Update: It's been a long time since this post first came out, so long that Rwandatel doesn't even exist anymore and Airtel is the new carrier in town. A few of these codes still work, but if you are in East Africa in general and have an Android device, download SimBuddy from the Play Store for an easy way to get access to all these shortcuts.

Since I'm always playing with all three carriers either for data access or to test something I figured I'd put up a quick cheat sheet for the various carriers.

MTN

• check balance: *110#
• add credit: *111*[code]#
• apn: internet.mtn
• buying internet bundle (1000 MB): *345*1000#
• checking bundle: *345#

Tigo

• check balance: *131#
• add credit: *130*[code]#
• apn: web.tigo.rw

Rwandatel

• check balance: *221#
• add credit: *220*[code]#
• apn: rtel3g
• username: rwandatel
• password: rwandatel
• auth type: chap

What usually comes next is a blank stare.  I can see the gears turning, see them use what little they know about Rwanda and try to resolve that with a technology startup.  So to answer all those blank stares, I thought I'd write up how this came to be, why it isn't crazy at all but rather so obviously right.

The story starts in the summer of '09.  My friend Nell Grey, a film maker, had gone to Arusha in the fall of '08 to help film interviews with prosecutors and other staff involved in the UN trials surrounding the Rwandan genocide.  That project, Voices from the Rwanda Tribunal, was returning to Rwanda that summer, and she suggested that maybe I could be of use helping them find ways of making the content available in creative ways.

Now I should preface this with the brief aside that I'd been looking for a way to 'give back' for a while, but wasn't sure how exactly my set of skills fit in.  My interest was in the developing world but I was neither a doctor nor engineer.  I believed in my own ability to figure things out, to be able to help somehow, but had to be honest with myself that I didn't seem particularly qualified to help.

One small part of the project's plans for the trip was to make segments of the interviews available to the public.  We brainstormed on various ways of doing so, from distributing CDs or DVDs, to radio broadcasts, USB keys etc..  After some research it seemed to me that one promising approach might be to try to deliver the content via phones. 

The penetration of mobile phones in Rwanda is roughly 30% and growing quickly.  That might seem low until you realize that the penetration of electricity is only 10%.  Most of the country has cell coverage, even the rural areas, and there is fierce competition from three different carriers.  Handsets are available for just a few dollars used and once you have a phone, you can walk into any number of shops and walk out five minutes later with a new number, without contracts to sign and without any commitments.  You only pay for outgoing calls or text messages, and money you add to your account is valid for 18 months, so the economic cost for a cell phone is tiny.

My proposal was to deliver short, key, segments of the interviews using an SMS callback system.  A user would send a text message to a number, specifying which segment they wanted to hear, and our system would call them back, playing the segment and allowing the user to leave a voice comment.  The user would only pay for the cheap outgoing text, and we could not only reach a broad audience but also collect their thoughts.  I spent a weekend hacking together a system that worked on a laptop, crossed my fingers and hopped on the plane for my three weeks in Kigali.

That's when I got my first lesson that the West, and I include myself in that bucket, is terrible at evaluating the needs and demands of the developing world.  My exposure to Rwanda's genocide was limited at this point, though I was the technical counsel for the project, I only knew what I had read.  Once we arrived, we found that the general population wasn't terribly interested in the UN's ICTR process.  They viewed it as ineffective, taking a decade to try a few dozen people, and largely irrelevant to their own problems of trying to put a country back together, of dealing with hundreds of thousands of genocide suspects.  They had crafted their own solution to that in the Gacaca court system and weren't particularly interested in the pontifications of various UN staff on their own process.  There was much good that would come out of the Tribunal Voices project, but my part wasn't going to be it.

But from that failure came a silver lining.  It freed me up to spend more time learning about the state of software in Rwanda, to learn of their goals and ambitions.  I was soon exposed to Rwanda's Vision 2020 Plan, a concerted and well organized effort by the government to reinvent the country in the next decade.  Part of that vision is to set the groundwork for an information based economy.  Rwanda, a land locked country, with very few natural resources and no large tourist draw, is in a tough place when it comes to growing more prosperous.  You can't set up manufacturing facilities in a country where transportation costs are so high, and the largest cash crops of coffee and tea bring in less than $100M a year combined, a pittance for a country of ten million.  So Rwanda has made the bold decision to try to build their economy on information instead.  In a digital world of bits and bytes your transportation costs are zero and there is no reason why a country in the middle of Africa can't participate in the global software economy.  All it needs is the infrastruture and people.

Rwanda is making great strides on both fronts.  The country will soon have multiple fiber optic connections to the internet, replacing their slow and expensive satellite uplinks.  Internet kioks are planned for the rural areas and they have even decided to participate in the One Laptop Per Child (OLPC) project, setting the stage for the entirety of their next generation to be computer litterate in grade school.

The people are there too.  Computer Science curriculums are full at the universities, students eager to take part in the new economy.  Individual entrepreneurs are everywhere and more and more small software companies are popping up.

But one thing they don't have is experience.  Classes are filled with bright students, but many are being taught by professors who have never written software themselves, how could they have?  So the learning is largely book based despite computers being available, because some of the teachers themselves are nervous about coding.  What companies do exist are still working through the growing pains of how to build software repeatedly.  They are doing their best and learning quickly, but still at a disadvantage, the field brand new to them.

Here I saw an opportunity.  There was clearly a need, a need for experience in building software, a need I was actually perfectly suited to satisfy.  Along the way I had met a professor and software engineer in Kigali, Emile Bniz NIYIBIZI, and through him I did some teaching at a local university, doing a few lectures on web technologies.  That reminded me of how much I loved teaching, especially software, and helped spawn the idea of starting a software company there.  A company that would hire and train local Rwandans to build software the way I knew how.  I spent my last few days talking to various Rwandans about this, and what I heard back was an enthusiastic yes, yes that it was needed, yes that it would work, yes to do it.

I returned to Seattle excited, but still unsure as to how to move forward, conflicted.  On one hand I had found something that seemed like a perfect fit for my desire for adventure, a way to give back that fit my skills, but on the other I had a successful company with my best friend Eric Newcomer.  We had built Trileet up from nothing over the past four years and it was still doing well.  But as it happened, when I talked to Eric about it, his reaction was a shared enthusiasm.  He too saw the uniqueness of the situation, and was excited by it.  So with my best friend and business partner on board, we flew back to Kigali in January, this time to really evaluate how realistic we were being and whether to move forward.

As we prepared for the trip, we started contacting others doing similar work.  One particularly fortuitous connection was with Matt Berg, who hooked us into the small but growing world of RapidSMS and UNICEF. 

SMS is being used in more and more projects in Africa, as a way of both collecting and spreading information.  Its incredible penetration and simplicity is well suited to the challenging environments there, and the applications using it are only limited by your imagination.  Many countries have implemented systems which allow farmers and fishermen to access the local market prices for their goods, reducing the information inequality present when selling to middlemen.  Other systems have been built to help keep track of drug or blood supplies in clinics, allowing clinics with no power, phone or internet connections to be part of regional supply systems.

As it worked out, Matt was going to be in nearby Uganda meeting with Sean Blaschke at UNICEF and various others.  We tweaked our plans a bit and ended up meeting them both for a weekend in Kampala.  What we found was exciting.  The number of projects involving SMS was only growing, and there was a high demand for companies fluent in those systems, doubly so if they were in Africa.  We also found out that there was a UNICEF project currently stalled in Kigali.  They had a design and prototype for an SMS system to track maternity and child health, but needed an engineer to help train local developers and deploy that system, all as soon as possible.

This seemed like too good of an opportunity to pass up, so I decided to prolong my stay in Rwanda for another month.  That month only reinforced our belief that this made sense.  The system was exciting to build, both in helping train the developers, and in the final product.  It did real good, had a real potential for impact, moreso than any other product I had ever worked on, and my skills and experience couldn't have been more perfectly suited.

Upon my return we started the process of setting up Nyaruka in earnest.  I started packing up my place, Eric and I discussed how we would structure things and we began planning.  We aren't so naive as to think we know how things will really turn out, but the current plan is this: Our company will be based in Kigali, where we will find bright, motivated students to train on Python, Django and RapidSMS.  Initially, we will specialize in building SMS based systems.  From there we will see.  The software market in Rwanda, and East Africa in general is wide open.  Opportunities abound for building software for the government and private sectors, there is even opportunity to help build localized software for the OLPCs, bringing our experience making games to bear on interactive learning software, another of our passions.

I can say without hesitation that this is the most exciting opportunity I've ever had.  On some level it feels too easy, too perfect, so I have to temper that excitement with a little forced pessimism.  But in the end, it comes down to having the chance to take the one thing I am best at, building software, and do good with it.  And that is exciting on a whole different level.  That I get to do it with my best friend makes it doubly unbelievable.

We'll do our best to keep this blog updated with our progress, our challenges, our successes and even our failings.  We don't expect to succeed everywhere or everytime, but we'll try our best and learn from it, and hope to share those lessons here.  We'll see you then.