How long do Python Postgres tools take to load data?

Data is crucial for all applications. While fetching a significant amount of data from database multiple times, faster data load times improve performance. The post considers tools like SQLAlchemy statement, SQLAlchemy ORM, Pscopg2, psql for measuring latency. And to measure the python tool timing, jupyter notebook’s timeit is used. Psql is for the lowest time taken reference. Table Structure annotation=> \d data; Table "" Column | Type | Modifiers --------+-----------+--------------------------------------------------- id | integer | not null default nextval('data_id_seq'::regclass) value | integer | label | integer | x | integer[] | y | integer[] | Indexes: "data_pkey" PRIMARY KEY, btree (id) "ix_data_label" btree (label) annotation=> select count(*) from data; count --------- 1050475 (1 row) SQLAlchemy ORM Declaration class Data(Base): __tablename__ = 'data' id = Column(Integer, primary_key=True) value = Column(Integer) # 0 => Training, 1 => test label = Column(Integer, default=0, index=True) x = Column(postgresql. [Read More]

Debugging Python multiprocessing program with strace

Debugging is a time consuming and brain draining process. It’s essential part of learning and writing maintainable code. Every person has their way of debugging, approaches and tools. Sometimes you can view the traceback, pull the code from memory, and find a quick fix. Some other times, you opt different tricks like the print statement, debugger, and rubber duck method. Debugging multi-processing bug in Python is hard because of various reasons. [Read More]

Notes from Root Conf Day 2 - 2017

On day 2, I spent a considerable amount of time networking and attend only four sessions. Spotswap: running production APIs on Spot instance Amazon EC2 spot instances are cheaper than on-demand server costs. Spot instances run when the bid price is greater than market/spot instance price. Mapbox API server uses spot instances which are part of auto-scaling server Auto scaling group is configured with min, desired, max parameters. Latency should be low and cost effective EC2 has three types of instances: On demand, reserved and spot. [Read More]

Notes from Root Conf Day 1 - 2017

Root Conf is a conference on DevOps and Cloud Infrastructure. 2017 edition’s theme is service reliability. Following is my notes from Day 1. State of the open source monitoring landscape The speaker of the session is the co-founder of Icinga monitoring system. I missed first ten minutes of the talk.-The talk is a comparison of all available OSS options for monitoring, visualization. Auto-discovery is hard. As per 2015 monitoring tool usage survey, Nagios is widely used. [Read More]

Book Review: The Culture Map

The Culture Map: Breaking Through the Invisible Boundaries of Global Business is a book on cultural differences in communication by Erin Meyer. Last year, I spent three months in NYC. Whenever I entered food outlet or made an eye contact, all the conversation started with “How are you doing?”. I replied, “I’m good and how are you doing?”. Most of the times, I didn’t receive a response. It was a sign. [Read More]

Book Review: Software Architecture with Python

The book Software Architecture with Python is by Anand B Pillai. The book explains various aspects of software architecture like testability, performance, scaling, concurrency and design patterns. The book has ten chapters. The first chapter speaks about different architect roles like solution architect, enterprise architect, technical architect what is the role of an architect and difference between design and architecture. The book covers two lesser spoken topics debugging and code security which I liked. [Read More]

RC checklist for Indian Applicants

One Sunny Sunday morning one can get up and question their self-existence, or one can ask every few days or few months what am I doing at the current job? The answer will push you to a place you have never been. One can meticulously plan for extravagant programming tour or programmer’s pilgrimage for three months. Yes, that what my outlook of RC is! RC is a different place from a usual workplace, meetups, college or any educational institute. [Read More]

Return Postgres data as JSON in Python

Postgres supports JSON and JSONB for a couple of years now. The support for JSON-functions landed in version 9.2. These functions let Postgres server to return JSON serialized data. This is a handy feature. Consider a case; Python client fetches 20 records from Postgres. The client converts the data returned by the server to tuple/dict/proxy. The application or web server converts tuple again back to JSON and sends to the client. [Read More]

RFCS We Love

A simple question can open a door for new exploration. While I was at RC, I tweeted “Is there any meetup group similar to papers we love for discussing RFCS?”. In an interval of nine minutes, Jaseem replied: “Let’s start one :)” Is there any meetup group similar to @papers_we_love for discussing RFCs? — kracekumar (@kracetheking) November 30, 2016 Above discussion on Twitter, lead to a new focus group RFCS We Love in Bangalore by Nemo, Jaseem, Avinash and Kracekumar. [Read More]

Expose jupyter notebook over the network

What is the Jupyter notebook? The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualisations and explanatory text. The definition is from the official site. I use IPython/Jupyter shell all time. If you haven’t tried, spend 30 minutes and witness the power! At times, I want to share some code snippet with folks in the same building during work, workshop or training. [Read More]