07 4 / 2014

Over period of time few people have asked me in meetups, online I want to learn python. Suggest me few ways to learn. Everyone who asked me had different background and different intentions. Before answering the question I try to collect more information about their interest and their previous approaches. Some learnt basics from codecademy, some attended beginners session in Bangpypers meetup. In this post I will cover general questions asked and my suggested approach.

Q: Suggest some online resources and books to learn python ?

A: I suggest three resources, How to think like computer scientist, Learn Python The Hardway, CS101 from Udacity. This is highly subjective because it depends on previous programming experience etc … I have a blog post with lot of python snippet without explanation (I know it is like sea without waves).

Q: It takes too much time to complete the book. I want to learn it soon.

A: I have been programming in python over 3 years now, still I don’t know in depth python. You may learn it in six months or in a week. It is the journey which is interesting more than destination.

Q: How long will it take to learn python ?

A: It depends what you want to learn in python. I learnt python in 3 hours while commuting to college. You should be able to grasp basic concepts in few hours. Practice will make you feel confident.

Q: I learnt basics of Python, can you give me some problems to solve and I will get back with solutions ?

A: No. I would be glad to help you if you are stuck. Learning to solve your problem is great way to learn. My first usable python program was to download Tamil songs. I still use the code :-). So find the small problem or project to work on. I would be happy to review the code and give suggestions to it.

Q: I want to contribute to open source python projects can you suggest ?

A: Don’t contribute to project because you want to, rather find a library or project which is interesting to you and see if things can be made better. Your contribution can be small like fixing spelling mistake (I have contributed with single character change). Linux kernel accepts patch which fixes spelling mistake. Every contribution has its own effect. So contribute if it adds value.

In case you are reading this blog post and want to learn python or need help I would be glad to help.

22 3 / 2014

We are writing a small utility function called is_valid_mime_type. The function takes a mime_type as an argument and checks if the mime type is one of the allowed types. Code looks like

ALLOWED_MIME_TYPE = ('application/json', 'text/plain', 'text/html')

def is_valid_mimetype(mime_type):
    """Returns True or False.

    :param mime_type string or unicode: HTTP header mime type
    """
    for item in ALLOWED_MIME_TYPE:
        if mime_type.startswith(item):
            return True
    return False

Above code can refactored into single line using any.

def is_valid_mimetype(mime_type):
    """Returns True or False.

    :param mime_type string or unicode: HTTP header mime type
    """
    return any([True for item in ALLOWED_MIME_TYPE
                              if mime_type.startswith(item)])

One liner. It is awesome, but not performant. How about using next ?

def is_valid_mimetype(mime_type):
    """Returns True or False.

    :param mime_type string or unicode: HTTP header mime type
    """
    return next((True for item in ALLOWED_MIME_TYPE 
                              if mime_type.startswith(item)), False)

(True for item in ALLOWED_MIME_TYPE if mime_type.startswith(item) is generator expression. When ALLOWED_MIME_TYPE is None or EMPTY exception will be raised. In order to avoid that False is passed as an argument to next.

Edit:

def is_valid_mimetype(mime_type):
    """Returns True or False.

    :param mime_type string or unicode: HTTP header mime type
    """
    return any(mime_type.startswith(item) for item in ALLOWED_MIME_TYPE)

Cleaner than Next.

07 3 / 2014

Python has sorted function which sorts iterable in ascending or descending order.

# Sort descending
In [95]: sorted([1, 2, 3, 4], reverse=True)
Out[95]: [4, 3, 2, 1]

# Sort ascending
In [96]: sorted([1, 2, 3, 4], reverse=False)
Out[96]: [1, 2, 3, 4]

sorted(iterable, reverse=True)[:n] will yield first n largest numbers. There is an alternate way.

Python has heapq which implements heap datastructure. heapq has function nlargest and nsmallest which take arguments n number of elements, iterable like list, dict, tuple, generator and optional argument key.

In [85]: heapq.nlargest(10, [1, 2, 3, 4,])
Out[85]: [4, 3, 2, 1]

In [88]: heapq.nlargest(10, xrange(1000))
Out[88]: [999, 998, 997, 996, 995, 994, 993, 992, 991, 990]

In [89]: heapq.nlargest(10, [1000]*10)
Out[89]: [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000]

In [99]: heapq.nsmallest(3, [-10, -10.0, 20.34, 0.34, 1])
Out[99]: [-10, -10.0, 0.34]

Let’s say marks is a list of dictionary containing students marks. Now with heapq it is possible to find highest and lowest mark in a subject.

In [113]: marks = [{'name': "Ram", 'chemistry': 23},{'name': 'Kumar', 'chemistry': 50}, {'name': 'Franklin', 'chemistry': 89}]

In [114]: heapq.nlargest(1, marks, key=lambda mark: mark['chemistry'])
Out[114]: [{'chemistry': 89, 'name': 'Franklin'}]

In [115]: heapq.nsmallest(1, marks, key=lambda mark: mark['chemistry'])
Out[115]: [{'chemistry': 23, 'name': 'Ram'}]

heapq can be used for building priority queue.

Note: IPython is used in examples where In [114]: means Input line number 114 and Out[114] means Output line number 114.

27 2 / 2014

Let’s say you want to find how many times each element is present in the list or tuple.

Normal approach

words = ['a', 'the', 'an', 'a', 'an', 'the']
d = {}
for word in words:
    if word in d:
        d[word] += 1
    else:
        d[word] = 1
print d
{'a': 2, 'the': 2, 'an': 2} 

Better approach

words = ['a', 'the', 'an', 'a', 'an', 'the']
d = {}
for word in words:
    d[word] = d.get(word, 0) + 1

print d
{'a': 2, 'the': 2, 'an': 2

Both the approach returned same values. The first one has 6 lines of logic and second has 3 lines of logic (less code less management).

Second approach uses d.get method. d.get(word, 0) return count of the word if key is present else 0. If 0 isn’t passed get will return None.

Pythonic approach:

import collections

words = ['a', 'b', 'a']

res = collections.Counter(words)

print res
Counter({'a': 2, 'b': 1})

Last approach is just one line and pythonic.

Snippet is extracted from Transforming Code into Beautiful, Idiomatic Python. Do watch and enjoy.

08 2 / 2014

It is very common to update single attribute of a model instance (say update first name in user profile) and save it to db.

In [18]: u = User.objects.get(id=1)

In [19]: u.first_name = u"kracekumar"

In [20]: u.save()

Very straight forward approach. How does django send the sql query to database ?

In [22]: from django.db import connection

In [22]: connection.queries
Out[22]: 
[... 
{u'sql': u'UPDATE "auth_user" SET "password" = \'pbkdf2_sha256$12000$vsHWOlo1ZhZg$DrC46wq+a2jEtEzxmUEw4vQw8oV/rxEK7zVi30QLGF4=\', "last_login" = \'2014-02-01 06:55:44.741284+00:00\', "is_superuser" = true, "username" = \'kracekumar\', "first_name" = \'kracekumar\', "last_name" = \'\', "email" = \'me@kracekumar.com\', "is_staff" = true, "is_active" = true, "date_joined" = \'2014-01-30 18:41:18.174353+00:00\' WHERE "auth_user"."id" = 1 ', u'time': u'0.001'}]

Not happy. Honestly it should be UPDATE auth_user SET first_name = 'kracekumar' WHERE id = 1. Django should ideally update modified fields.

Right way to do is

In [23]: User.objects.filter(id=u.id).update(first_name="kracekumar")
Out[23]: 1

In [24]: connection.queries
Out[24]:
[...
{u'sql': u'UPDATE "auth_user" SET "first_name" = \'kracekumar\' WHERE "auth_user"."id" = 1 ', u'time': u'0.001'}]

Yay! Though both queries took same amount of time, latter is better.

Edit: There is one more cleaner way to do it.

In [60]: u.save(update_fields=['first_name'])

In [61]: connection.queries
Out[61]: 
[...
{u'sql': u'UPDATE "auth_user" SET "first_name" = \'kracekumar\'  WHERE "auth_user"."id" = 1 ',
u'time': u'0.001'}]

26 12 / 2013

This is the material which I use for teaching python to beginners.

tld;dr: Very minimal explanation more code.

Python?

  • Interpreted language
  • Multiparadigm

Introduction

hasgeek@hasgeek-MacBook:~/codes/python/hacknight$ python
Python 2.7.3 (default, Aug  1 2012, 05:14:39) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>


>>> print "Let's learn Python"
Let's learn Python

Numbers

>>> 23 + 43
66
>>> 23 - 45
-22
>>> 23 * 45
1035
>>> 23 ** 4
279841
>>> 23 / 4
5
>>> 23 / 4.0
5.75
>>> 7 % 2
1

Expressions

>>> 3 < 2
False
>>> 3 > 2
True
>>> 3 > 2 < 1
False
>>> (3 > 2) and (2 < 1)
False
>>> 3 > 2 > 1 > 0
True
>>> (3 > 2) and (2 > 1) and (1 > 0)
True
>>> 1 or 2
1
>>> 2 or 1
2
>>> 1 + 2 + 3 * 4 + 5
20
1 + 2 + 3 * 4 + 5
↓
3   + 3 * 4 + 5
        ↓
3   +   12  + 5
      ↓
    15      + 5
          ↓
         20

>>> "python" > "perl"
True
>>> "python" > "java"
True

Variables

>>> a = 23
>>> print a
23
>>> a = "Python"
>>> print a
Python

Guess the output

True = False
False = True
print True, False
print 2 > 3

Parallel Assignment

>>> language, version = "Python", 2.7
>>> print language, version
Python 2.7
>>> x = 23
>>> x = 23
>>> y = 20
>>> x, y = x, x + y
>>> print x, y
23 43

Guess the output

z, y = 23, z + 23
a, b = 23, 12, 20
a = 1, 2

Swap Variable

>>> x = 12
>>> y = 21
>>> x, y = y, x
>>> print x, y
21 12
>>>

String

>>> language = "Python"
>>> print language
Python
>>> language = 'Python'
>>> print language
Python
>>> language = """Python"""
>>> print language
Python
>>> description = """Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability.
... It is an expressive language which provides language constructs intended to enable clear programs on both a small and large scale.
... Python supports multiple programming paradigms, including object-oriented, imperative and functional programming styles.
... """
>>> print description
Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability.
It is an expressive language which provides language constructs intended to enable clear programs on both a small and large scale.
Python supports multiple programming paradigms, including object-oriented, imperative and functional programming styles.
>>> 

Guess output

name = "krace" + "kumar"
print name
print name[0]
name[0] = "K"

Guess output

print 1 + 2.5
print "kracekumar" + 23

Condition

Write a program to find greatest of two numbers.

>>> a = 12
>>> b = 23
>>> if a > b:
...     print "a is greater than b"
... else:
...     print "b is greater than a"
... 
b is greater than a
>>> if a > 0:
...     print "a is positive"
... elif a == 0:
...     print "a is zero"
... elif a < 0:
...     print "a is negative"
... 
a is positive

Data Structure

List

List is a collection of heterogenous data types like integer, float, string.

>>> a = [1, 2, 3]
>>> b = ["Python", 2.73, 3]
>>> len(a)
3
>>> len(b)
3
>>> a[0]
1
>>> a[-1]
3
>>> b[2]
3
>>> [1, 2] + [3, 4]
[1, 2, 3, 4]
>>> all = [a, b]
>>> all[0]
[1, 2, 3]
>>> all[-1]
['Python', 2.73, 3]
>>> all[3]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> all.append("Bangalore")
>>> all
[[1, 2, 3], ['Python', 2.73, 3], 'Bangalore']
>>> del all[-1]
>>> all
[[1, 2, 3], ['Python', 2.73, 3]]
>>> all[1] = "insert"
>>> all
[[1, 2, 3], 'insert']
>>> all
[[1, 2, 3], 'insert']
>>> 'insert' in all
True
>>> range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> range(10, 2)
[]
>>> range(10, 0, -1)
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> range(0, 12, 1)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]



range() -> `range([start,] stop[, step]) -> list of integers`

Slicing

>>> l = [1, 2, 3, 4, 5, 6, 7]
[1, 2, 3, 4, 5, 6, 7]
>>> l[:2] #first two elements
[1, 2]
>>> l[2:] #exclude first two elements
[3, 4, 5, 6, 7]
>>> l[::2] #every second element
[1, 3, 5, 7]
>>> l[::1] #every element
[1, 2, 3, 4, 5, 6, 7]
>>> l[::3] #every third element
[1, 4, 7]
>>> l[::10] #every tenth element
[1]
>>> l[::-1]
[7, 6, 5, 4, 3, 2, 1]

Guess the output

l[1:7:2] [][:2] [1][:2]

Accessing list elements

>>> for item in all:
...     print item
... 
[1, 2, 3]
insert
>>> for number in range(10):
...     print number
... 
0
1
2
3
4
5
6
7
8
9

Find all odd numbers from 0 to 9

>>> for number in range(0, 10):
...     if number % 2:
...         print number
... 
1
3
5
7
9

inbuilt functions

>>> help([])
>>> min([1, 2, 3])
1
>>> max([1, 2, 3])
3
>>> sum([1, 2, 3])
6
>>> pow(2, 3)
8

Write program which takes a number as input and if number is divisible by 3, 5 print Fizz, Buzz, FizzBuzz respectively

import sys


if __name__ == "__main__":
    if len(sys.argv) == 2:
        number = int(sys.argv[1])
    if number % 15 == 0:
        print "FizzBuzz"
    elif number % 3 == 0:
        print "Fizz"
    elif number % 5 == 0:
        print "Buzz"
    else:
        print number

else:
    print "python filename.py 23 is the format"

Tuples

Tuple is a sequence type just like list, but it is immutable. A tuple consists of a number of values separated by commas.

>>> t = (1, 2)
>>> t
(1, 2)
>>> t[0]
1
>>> t[0] = 1.1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t = 1, 2
>>> t
(1, 2)
>>> del t[0]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object doesn't support item deletion
>>> for item in t:
...     print item
... 
1
2

Sets

Sets are unordered collection of unique elements.

>>> x = set([1, 2, 1])
>>> x
set([1, 2])
>>> x.add(3)
>>> x
set([1, 2, 3])
>>> x = {1, 3, 4, 1}
>>> x
set([1, 3, 4])
>>> 1 in x
True
>>> -1 in x
False
>>> 

Again Lists

>>> even_numbers = []
>>> for number in range(0, 9):
...     if number % 2 == 0:
...         even_numbers.append(number)
... 
>>> even_numbers
[0, 2, 4, 6, 8]

As a programmer your job is write lesser code

List Comprehensions

>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [x + 1 for x in range(10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> numbers = []
>>> for x in range(10):
...     numbers.append(x + 1)
... 
>>> print numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> even_numbers = [x for x in range(10) if x %2 == 0]
>>> even_numbers
[0, 2, 4, 6, 8]
>>> [(x, y) for x in range(5) for y in range(5) if (x+y)%2 == 0]
[(0, 0), (0, 2), (0, 4), (1, 1), (1, 3), (2, 0), (2, 2), (2, 4), (3, 1), (3, 3    ), (4, 0), (4, 2), (4, 4)]
>>> 

Dictionaries

>>> d = {'a': 1, 'b': 2, 'c': 3}
>>> d['a']
1
>>> d.get('a')
1
>>> d['z']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'z'
>>> d.get('z')
>>>
>>> d['a'] = 2
>>> d
{'a': 2, 'c': 3, 'b': 2}
>>> d['z'] = 26
>>> d
{'a': 2, 'c': 3, 'b': 2, 'z': 26}
>>> d.keys()
['a', 'c', 'b', 'z']
>>> d.values()
[2, 3, 2, 26]
>>> d.items()
[('a', 2), ('c', 3), ('b', 2), ('z', 26)]
>>> type(d.items())
<type 'list'>
>>> d = {'a': 2, 'b': 2, 'c': 3, 'z': 26}
>>> for key in d:
...     print key
... 
a
c
b
z
>>> for key, value in d.items():
...     print key, value
... 
a 2
c 3
b 2
z 26
>>> 'a' in d
True
>>> d.has_key('a')
True

Function

Just like a value can be associated with a name, a piece of logic can also be associated with a name by defining a function.

>>> def square(x):
...     return x * x
... 
>>> square(2)
4
>>> square(2+1)
9
>>> square(x=5)
25
>>> def dont_return(name):
...     print "Master %s ordered not to return value" % name
... 
>>> dont_return("Python")
Master Python ordered not to return value
>>> def power(base, to_raise=2):
...     return base ** to_raise
... 
>>> power(3)
9
>>> power(3, 3)
27
>>> def power(to_raise=2, base):
...     return base ** to_raise
... 
File "<stdin>", line 1
SyntaxError: non-default argument follows default argument 
>>> square(3) + square(4)
25
>>> power(base=square(2))
16
>>> def sum_of_square(x, y):
...     return square(x) + square(y)
... 
>>> sum_of_square(2, 3)
13
>>> s = square
>>> s(4)
16
>>> def fxy(f, x, y):
...     return f(x) + f(y)
... 
>>> fxy(square, 3, 4)
25

Methods - Methods are special kind of functions that work on an object.

>>> lang = "Python"
>>> type(lang)
<type 'str'>
>>> dir(lang)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
'__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__',
'__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__',
'__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser',
'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum',
'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition',
'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip',     'split', 'splitlines',
'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>> lang.upper()
'PYTHON'
>>> help(lang.upper)
>>> lang.startswith('P')
True
>>> help(lang.startswith)

>>> lang.startswith('y', 1)
True

Files

>>> f = open('foo.txt', 'w')
>>> help(f)

>>> f.write("First line")
>>> f.close()
>>> f = open('foo.txt', 'r')
>>> f.readline()
'First line'
>>> f.readline()
''
>>> f = open('foo.txt', 'a')
>>> f.write('Second line')
>>> f.close()
>>> f = open('foo.txt', 'r')
>>> f.readline()
'First lineSecond line'
>>> f = open('foo.txt', 'a')
>>> f.write("New line\n")
>>> f.write("One more new line")
>>> f.close()
>>> f = open('foo.txt', 'r')
>>> f.readline()
'First lineSecond lineNew line\n'
>>> f.readline()
'One more new line'
>>> f.readline()
''
>>> f.close()
>>> f = open('foo.txt')
>>> f.readlines()
['First lineSecond lineNew line\n', 'One more new line']
>>> f = open('foo.txt', 'w')
>>> f.writelines(["1\n", "2\n"])
>>> f.close()
>>> f.readlines()
>>> f = open('foo.txt')
>>> f.readlines()
['1\n', '2\n']
>>> f.close()

Exception Handling

>>> f = open('a.txt')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'a.txt'
>>> try:
...     f = open('a.txt')
... except:
...     print "Exception occured"
... 
Exception occured
>>> try:
...     f = open('a.txt')
... except IOError, e:
...     print e.message
... 

>>> e
IOError(2, 'No such file or directory')
>>> dir(e)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', 'args', 'errno',     'filename', 'message', 'strerror']
>>> e.strerror
'No such file or directory'
>>> try:
...     print l[4]
... except IndexError, e:
...     print e
... 
list index out of range
>>> raise Exception("error message")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception: error message
>>> try:
...     print "a"
...     raise Exception("doom")
... except:
...     print "b"
... else:
...     print "c"
... finally:
...     print "d"
... 
a
b
d

Object Oriented Programming

>>> class BankAccount:
        def __init__(self):
            self.balance = 0

        def withdraw(self, amount):
            self.balance -= amount
            return self.balance

        def deposit(self, amount):
            self.balance += amount
            return self.balance

>>> a = BankAccount()
>>> b = BankAccount()
>>> a.deposit(200)
200
>>> b.deposit(500)
500
>>> a.withdraw(20)
180
>>> b.withdraw(1000)
-500
>>> class MinimumBalanceAccount(BankAccount):
...    def __init__(self, minimum_balance):
...        BankAccount.__init__(self)
...        self.minimum_balance = minimum_balance
...
...    def withdraw(self, amount):
...        if self.balance - amount < self.minimum_balance:
...            print "Sorry, you need to maintain minimum balance"
...        else:
...            return BankAccount.withdraw(self, amount)
>>> a = MinimumBalanceAccount(500)
>>> a
<__main__.MinimumBalanceAccount instance at 0x7fa0bf329878>
>>> a.deposit(2000)
2000
>>> a.withdraw(1000)
1000
>>> a.withdraw(1000)
Sorry, you need to maintain minimum balance 
>>> class A:
...     def f(self):
...         return self.g()
...     def g(self):
...         return "A"
... 
>>> a = A()
>>> a.f()
'A'
>>> a.g()
'A'
>>> class A:
...     def __init__(self):
...         self._protected = 1
...         self.__private = 2
... 
>>> a = A()
>>> a._protected
1
>>> a.__private
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: A instance has no attribute '__private'

Sample Python Program

#! /usr/bin/env python
# -*- coding: utf-8 -*-


class BankAccount:
    def __init__(self):
        self.balance = 0

    def withdraw(self, amount):
        self.balance -= amount
        return self.balance

    def deposit(self, amount):
        self.balance += amount
        return self.balance


class MinimumBalanceAccount(BankAccount):
    def __init__(self, minimum_balance):
        BankAccount.__init__(self)
        self.minimum_balance = minimum_balance

    def withdraw(self, amount):
        if self.balance - amount < self.minimum_balance:
            print "Sorry, you need to maintain minimum balance"
        else:
            return BankAccount.withdraw(self, amount)

    def __repr__(self):
        return "MinimuBalanceAccount, Balance: %d" %(self.balance)


if __name__ == "__main__":
    a = MinimumBalanceAccount(500)
    print a.deposit(5000)
    print a.withdraw(4500)
    print a.withdraw(500)

Few examples are taken from python practice book.

Github repo: https://github.com/kracekumar/python-training

25 12 / 2013

This article will focus on deploying flask app starting from scratch like creating separate linux user, installating database, web server. Web server will be nginx, database will be postgres, python 2.7 middleware will be uwsgi, server ubuntu 13.10 x64. Flask app name is fido. Demo is carried out in Digital ocean.

Step 1 - Installation


Python header

root@fido:~# apt-get install -y build-essential python-dev

Install uwsgi dependencies

root@fido:~# apt-get install -y libxml2-dev libxslt1-dev

Nginx, uwsgi

root@fido:~#  apt-get install -y nginx uwsgi uwsgi-plugin-python

Start nginx

root@fido:~# service nginx start
* Starting nginx nginx                                                          [ OK ]

Postgres

root@fido:~# apt-get install -y postgresql postgresql-contrib libpq-dev

Step 2 - User

Create a new linux user fido

root@fido:~# adduser fido

Enter all the required details.

root@fido:~# ls /home
fido

Successfully new user is created.

Grant fido root privilege.

root@fido:~# /usr/sbin/visudo
# User privilege specification

root    ALL=(ALL:ALL) ALL 
fido    ALL=(ALL:ALL) ALL

Since fido is not normal user, delete fido’s home directory.

root@fido:~# rm -rf /home/fido
root@fido:~# ls /home
root@fido:~#

Create a new db user fido

root@fido:~# su - postgres
postgres@fido:~$ createuser --pwprompt
Enter name of role to add: fido
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) y

--pwprompt will prompt for password. release is the password I typed (we need this to connect db from app).

Create a new database fido

postgres@fido:~$ createdb fido;
postgres@fido:~$ psql -U fido -h localhost
Password for user fido:
psql (9.1.10)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.
fido=# \d
No relations found.

Done. New database role fido and database is created. We are successfully able to login.

Step 3 - Python dependencies

Install pip

root@fido:# cd /tmp
root@fido:/tmp# wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
root@fido:/tmp# python ez_setup.py install
root@fido:/tmp# easy_install pip 
# What is easy_install ? Python package manager.
# what is pip ? Python package manager.
# How to install pip ? easy_install pip. 
# Shame on python :-(
...
Installed /usr/local/lib/python2.7/dist-packages/pip-1.4.1-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

Install virtualenv

root@fido:/tmp# pip install virtualenv

Step 4 - Install app dependencies

Here is the sample app code. The app is just for demo. The app will be placed in /var/www/fido. Normally in production, this will be similar to git clone <url> or hg clone <url> inside directory. Make sure you aren’t sudo while cloning.

root@fido:/tmp# cd /var
root@fido:/var# mkdir www
root@fido:/var# mkdir www/fido

Change the owner of the repo to fido.

root@fido:/var# chown fido:fido www/fido
root@fido:/var# ls -la www/
total 12
drwxr-xr-x  3 root root 4096 Dec 25 03:18 .
drwxr-xr-x 14 root root 4096 Dec 25 03:18 ..
drwxr-xr-x  2 fido fido 4096 Dec 25 03:18 fido

app.py - fido application.

# /usr/bin/env python

from flask import Flask, request
from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = "postgres://fido:release@localhost:5432/fido"
db = SQLAlchemy(app)


class Todo(db.Model):
    id = db.Column(db.Integer(), nullable=False, primary_key=True)
    name = db.Column(db.UnicodeText(), nullable=False)
    status = db.Column(db.Boolean(), default=False, nullable=True)


@app.route("/")
def index():
    return "Index page. Use /new create a new todo"


@app.route('/new', methods=['POST'])
def new():
    form = request.form
    name, status = form.get('name'), form.get('status') or False
    todo = Todo(name=name, status=status)
    db.session.add(todo)
    db.session.commit()
    return "Created todo: {}".format(name)


if __name__ == "__main__":
    db.create_all()
    app.run('0.0.0.0', port=3333, debug=True)

Add a wsgi file website.py

root@fido:/var/www/fido# cat website.py
import sys
import os.path
sys.path.insert(0, os.path.dirname(__file__))
from app import app as application

Files in fido directory.

root@fido:/var/www/fido# tree .
.
├── app.py
├── __init__.py
└── website.wsgi

0 directories, 3 files

Step 5 - Virtual env and dependencies

root@fido:/var/www/fido# virtualenv --no-site-packages env
root@fido:/var/www/fido# . env/bin/activate
(env)root@fido:/var/www/fido# pip install flask sqlalchemy flask-sqlalchemy psycopg2

Step 6 - final setup

Create uwsgi config file

# Add following lines to fido.ini file
root@fido:/etc# cat uwsgi/apps-enabled/fido.ini
[uwsgi]
socket = 127.0.0.1:5000
threads = 2
master = true
uid = fido
gid = fido
chdir = /var/www/fido
home = /var/www/fido/env/
pp = ..
module = website

Check whether uwsgi is booting up properly.

root@fido:/var/www/fido# uwsgi --ini /etc/uwsgi/apps-enabled/fido.ini
...
...
Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09)  [GCC 4.8.1]
Set PythonHome to /var/www/fido/env/
Python main interpreter initialized at 0xb96a70
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 165920 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
added ../ to pythonpath.
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xb96a70 pid:  17559 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 17559)
spawned uWSGI worker 1 (pid: 17562, cores: 2)

uwsgi is able to load the app without any issues. Kill the uwsgi using keyboard interrupt.

Now lets create table.

root@fido:/var/www/fido# . env/bin/activate
(env)root@fido:/var/www/fido# python app.py
* Running on http://0.0.0.0:3333/
* Restarting with reloader

Exit the program, db.create_all() must have created the table. Normally in production environment, it is advised to use python manage.py db create or any similar approach.

Configure nginx

root@fido:/var/www/fido# cat /etc/nginx/sites-enabled/fido.in
upstream flask {
    server 127.0.0.1:5000;
}

# configuration of the server
server {
    # the domain name it will serve for
    listen 127.0.0.1; # This is very important to test the server locally
    server_name fido.in; # substitute your machine's IP address or FQDN
    charset     utf-8;

    location / {
        uwsgi_pass  flask;
        include uwsgi_params;
    }
}

Now nginx and uwsgi will be running in background. Restart them.

root@fido:/var/www/fido# service nginx restart
* Restarting nginx nginx                                                             [ OK ]
root@fido:/var/www/fido# service uwsgi restart
* Restarting app server(s) uwsgi                                                    [ OK ]

Machine name is fido, so lets try curl http://fido

root@fido:/var/www/fido# curl http://fido
Index page. Use /new create a new todo

Create a new task.

root@fido:/var/www/fido#curl --data "name=write blog post about flask deployment"  http://fido/new
Created todo: write blog post about flask deployment

We have successfully deployed flask + uwsgi + nginx.

Since we installed uwsgi from ubuntu repo, it is started as upstart process, that is why we issue commands like service uwsgi restart.

To see all upstart service, try service --status-all.

If you are running multiple web application in a single server, create one user per application.

22 12 / 2013

ipython nbconvert has lot of handy options to convert ipynb to markdown, html etc… But I wanted to upload ipynb to gist.github.com and create a link in nbviewer.ipython.org. Started with curl and soon realized, it is getting messy. So wrote a small python program ipynb2viewer, Source code.

Install

  • pip install ipynb2viewer

Usage

Upload all ipynb files in the given path to gist.github.com and return nbviewer urls.

  • ipynb2viewer all <path>

Upload mentioned file to gist.github.com and return nbviewer url.

  • ipynb2viewer file <filename>

Upload mentioned file to gist.github.com and open nbviewer url in webbrowser.

  • ipynb2viewer file <filename> --open

Upload all ipynb files in the given path to gist.github.com and open nbviewer urls in webbrowser.

  • ipynb2viewer all <path> --open

Example

➜  ipynb2viewer git:(master) ipynb2viewer all tests --open

Uploaded test.ipynb file to gist https://gist.github.com/8081202
nbviewer url: http://nbviewer.ipython.org/gist/anonymous/8081202  
Uploaded try.ipynb file to gist https://gist.github.com/8081203
nbviewer url: http://nbviewer.ipython.org/gist/anonymous/8081203

22 11 / 2013

I hardly use easy_install. Nowadays all the python requirements are installed via pip.

IPython is my primary python console. After installing mavericks, I installed IPython and fired IPython console. Following warning message appeared

➜  ~  ipython
/Library/Python/2.7/site-packages/IPython/utils/rlineimpl.py:94: RuntimeWarning: 
******************************************************************************
libedit detected - readline will not be well behaved, including but not limited to:
   * crashes on tab completion
   * incorrect history navigation
   * corrupting long-lines
   * failure to wrap or indent lines properly
It is highly recommended that you install readline, which is easy_installable:
     easy_install readline
Note that `pip install readline` generally DOES NOT WORK, because
it installs to site-packages, which come *after* lib-dynload in sys.path,
where readline is located.  It must be `easy_install readline`, or to a custom
location on your PYTHONPATH (even --user comes after lib-dyload).
******************************************************************************
  RuntimeWarning)
Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
Type "copyright", "credits" or "license" for more information.

IPython complains readline is missing and insisting to use easy_install. Then I tried

➜   ~  sudo easy_install-2.7 --upgrade setuptools
Traceback (most recent call last):
  File "/usr/bin/easy_install-2.7", line 7, in <module>
    from pkg_resources import load_entry_point
  File "/Library/Python/2.7/site-packages/pkg_resources.py", line 2797, in <module>
    parse_requirements(__requires__), Environment()
  File "/Library/Python/2.7/site-packages/pkg_resources.py", line 576, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: setuptools==0.6c12dev-r88846

I know I can’t use pip, but distribute is already installed.

Now only way to install readline is to use easy_install in Python site-packages. Normally easy_install is installed in /usr/bin.

➜   ~  which easy_install
/usr/bin/easy_install
➜  ~  sudo python /Library/Python/2.7/site-packages/easy_install.py readline
Searching for readline
Reading https://pypi.python.org/simple/readline/
Best match: readline 6.2.4.1
Downloading https://pypi.python.org/packages/2.7/r/readline/readline-6.2.4.1-py2.7-macosx-10.7-intel.egg#md5=6ede61046a61219a6d97c44a75853c23
Processing readline-6.2.4.1-py2.7-macosx-10.7-intel.egg
creating /Library/Python/2.7/site-packages/readline-6.2.4.1-py2.7-macosx-10.7-intel.egg
Extracting readline-6.2.4.1-py2.7-macosx-10.7-intel.egg to /Library/Python/2.7/site-packages
Adding readline 6.2.4.1 to easy-install.pth file

Installed /Library/Python/2.7/site-packages/readline-6.2.4.1-py2.7-macosx-10.7-intel.egg
Processing dependencies for readline
Finished processing dependencies for readline

➜  ~  ipython 
Python 2.7.5 (default, Aug 25 2013, 00:04:04) 
Type "copyright", "credits" or "license" for more information.

IPython 1.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Whenever issue pops up with easy_install and setuptools, remember using easy_install from site packages like sudo python /Library/Python/2.7/site-packages/easy_install.py

12 10 / 2013

How to check given text is palindrome or not

def sanitize(text):
    for char in [" ", ".", ",", ";", "\n"]:
        text = text.replace(char, "")
    return text

def palindrome(word):
    # This approach is o(n), problem can be solved with o(n/2)
    # I am using this approach for brevity
    return word == word[::-1]

palindrome(sanitize("madam")) # True
palindrome(sanitize(u"விகடகவி")) # False

Here is the hand made version for Tamil

# Assign the variable meal the value 44.50 on line 3!
# hex values from க..வ
def sanitize(text):
    for char in [" ", ".", ",", ";", "\n"]:
        text = text.replace(char, "")
    return text

dependent_vowel_range = range(0xbbe, 0xbce)

def palindrome_tamil(text):
    front, rear = 0, len(text) - 1
    while True:
        #We will start checking from both ends
        #If code reached centre exit
        if front == rear or abs(front - rear) == 1:
            return True
        else:
            if ord(text[front+1]) in dependent_vowel_range and ord(text[rear]) in dependent_vowel_range:
                if text[front] == text[rear-1] and text[front+1] == text[rear]:
                    front += 2
                    rear -= 2
                else:
                    return False
            else:
                if text[front] == text[rear]:
                    front += 1
                    rear -= 1
                else:
                    return False




print palindrome_tamil(sanitize(u"விகடகவி")) == True
text = u"""
யாமாமாநீ யாமாமா யாழீகாமா காணாகா
காணாகாமா காழீயா மாமாயாநீ மாமாயா
"""
print palindrome_tamil(sanitize(text)) == True

#output
True
True