uWSGI Mule Magic and Web2py Scheduler

Here is me trying to put life back into this blog.

This post is about running python workers with uWSGI, as unbit puts it - "uWSGI is a fast, self-healing and developer/sysadmin-friendly application container server coded in pure C."

Back Story:

Sahana Eden is developed with Web2py, at Eden we have tasks that sometime require a worker to perform Asynchronously after a job is submitted. One such task is sending out notification events via the Messaging module. We use Web2py's shiny new scheduler library for this, which needs long running Web2py process' to churn out the jobs. We already use uWSGI for our new deployments, so I thought why not see if uWSGI can do the worker  management as well.


The Story:

uWSGI has a nice config option called the mule. Mule - allows you to spawn a Python worker controlled by the uWSGI master. It respawns the worker on death and signalling the master pulls down both the webserver and the worker at the same time. Needed a small script to make Web2py play nice as a uWSGI mule.

Code snippets below:

Script to spawn a Web2py long running scheduler process (You can replace this with your own long running Python process) :



uWSGI config to run the mule:

Sahana Community Meetup @ Lisbon

Sahana Software Foundation had its Annual Meeting, Community Bar Camp and a Disaster Workshop at ISCRAM, in Lisbon from the 6th-8th this month. The whole event was fun and a great learning experience. Interacting with members of the community only acquainted through IRC and the mailing list was well worth the 24 hour journey from Kanpur to Lisbon. A special thanks to Google and Sahana software foundation for providing support for travel and stay for the meet.

At the Community Bar Camp, We had a couple of productive discussions on interoperability and the functioning of Sahana Software Foundation. Thanks to the face to face time with the other members of the Eden community - I was able to understand the framework a bit better and work out the requirements for a possible Asynchronous processing API in Eden. The activities for the first 2 days were located at a space, generously donated by IBM Portugal, who also provided us with a huge supply of delicious pastries (Yum!), orange juice and coffee. Evenings were spent in long pleasant dinner sessions followed by late night social meetups at the hotel lounge, supplemented with great wine and amazing ideas/thoughts/people. In short it was a great event with some of the most dynamic and vibrant people I have ever met, who were fun to interact and work with. I very eagerly look forward to the next Sahana Software Foundation's annual meeting, hoping to meet the few who couldn't make it this time around.

Sahana Software Foundation's Blog has a finer and a well written account of the entire event (Click here).


I would like to thank the following people/organisations for making my visit possible:

1) Google and SSF for providing support for Travel.
2) Mark Prutsalis for patiently sending me all the documentation required for the visa.
3) To the unknown person who set the end semester exam timetable at IIT Kanpur, leaving me with the exact time to make it to the event.

This blog post has been long over due - Mostly because, I just finished my Masters in Physics (Yay!) and am in the process of kick starting our tiny private limited (More about this later).

Reading raw disks with python - Windows/Linux/Mac

We have been working for a while on an embedded solution that geo-tags jerks using an accelerometer (I know it sounds funny :)). Not wanting to increase the overhead on our tiny < 16MHz 8 bit AVR and maintain our precise sampling requirements, we came up with a simple filesystem that does the job and this fancy format meant that we needed some code on the computer end to read the sdcard. Took us a while to search out the raw disk file for Windows. The following snippet shows how to search the disk at a particular address - [Our disk structure had the word MOMS at that address.]

tl;dr Snippet to read the raw disk using python on Windows/Linux/MacOSX

Extract POIs from OSM PBF ("Protocolbuffer Binary Format") dumps with python

• I am a GIS n00b - this is my first attempt at handling OSM data.
• This code would have never been written if it wasn't for Chris Hill's excellent parsepbf.py - blog post here.

I looked at OSM to obtain railway station locations in the country for an in-house project we are running.Parsing through their data dumps sounded like an easy job. I grabbed india.osm.bz2 and india.osm.pbf from Geofabrik. Uncompressing the bz2 file resulted in a 614MB xml whereas the pbf was just 26MB. Intrigued by the small file size of the pbf files ( I never read up on google protocol buffers before) I went to the OSM wiki to read up the format and see if any python libraries are available for this. I found Chris' parsepbf script and ran it with the pbf I had. Turns out running the the script without asking it to spit out osm xml was a bad idea - ended up eating all the memory on my machine [ no swap enabled ] and crashing the system.

I modified the parsepbf file to make as somewhat generic class for picking out nodes with specified tags.


Stats:

• It took about 5 minutes to pickout all the railway stations on my linode ( 512MB ) VPS.
• I think a speed up can be achieved by using multiprocessing (?)

Example usage:



Current code can be found here.

Daemonize Selenium-grid with init scripts on Debian

In the process of setting up selenium grid for Sahana Eden CI - I needed to run selenium-grid as a daemon, as I could not find any init scripts for it on the internets, I hacked Jenkins ( Hudson ) init script for this.

The script could be generalized to run generic ant builds as a daemon.

ATM + Windows = :|

I happened to notice this at the ATM - Windows 2000 :|

Note: Frozen screen

--

Owl at the CC canteen

 

Excuse me for the frequent blog posts :D

New Phone + Airtel mobile office + wvdial

So in the last one week I got a new phone ( Nokia 7210c Supernova) to develop / test the J2ME app that I am making as a part of Google Summer Of Code. The phone is a relatively basic model with Edge, J2ME ( :) ) and bluetooth. Me being what I am .. was fiddling around with the phone and some of the Airtel services. The basic Airtel GPRS service called Airtel live provides internet access through a proxy => Apps which don't support proxy fail ( >90% ). I went searching for ways to have Airtel Mobile Office activated after considerble amount of searching I was able to activate the mobile office by dailing *567*1# and following the instructions that followed, If the settings don't arrive in a day repeat the instructions in the SMS that follows the number dial, note that it costs 30 paise per 50kB on Airtel Prepaid.

This was followed by the installation of jmIrc and MidpSSH for that occasional itch to do something weird while on the move. People who know the rough amount of data that is sent during IRC will know that you can chat for ~1 hour with the 50 kB data transfer :) . After this the usual Opera mini , Google maps were tried and of course setting configuring the phones mailbox to send mail through my gmail account.

Today morning I had the weird urge to connect to the internet through my phone over bluetooth :D. The KDE4 bluetooth apps are as not the best ones out there :) , so I ended up installing blueman . Paired my phone and then selected the Dial Up Networking ( DUN ) in the device manager window, it asks for the root pass and walah you have a /dev/rfcomm0 ( There are other ways of binding the rfcomm through command line but GUI sometimes is simpler :) ). This was followed by searching for settings to connect to the internet , here are the things I did to get the internet working through the phone.

1) Install wvdial
2) edit /etc/wvdial.conf

[Dialer GPRS]
Modem = /dev/rfcomm0
Baud = 115200
Dial Command = ATDT
Init1 = ATZ
Init2 = AT+CGDCONT=1,"IP","airtelgprs.com";
FlowControl = None
Username= internet
Password= internet
Phone = *99***1#

The username and pass can be anything ( they are not validated )

3) As root run wvdial GPRS

4) Tadaa you are connected to the internets

Code highlighting courtesy : http://lukabloga.blogspot.com/2008/10/to-test-new-highlighting.html