segfault.in

RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data.

For this tutorial I’m going to assume you understand how to get RRDTool installed and working from the command line. If not you should take a look at this tutorial (if you are in a hurry look at the “A Real World Example” section) or any of the tutorials on this page.

Setup and introduction

Py-rrdtool wrapper implementation has been written from the scratch (without SWIG). So we will require python dev file for compiling the py-rrdtool module.

$ sudo apt-get install python-dev

Download the Python RRDTool from here.

$ tar xfz py-rrdtool-1.0b1.tar.gz
$ cd py-rrdtool-1.0b1
$ sudo python setup.py install

Creating a RRD

We first need to create a database for our data. For these examples we will be creating a database that could be used for generating network graphs. This database is created for updates every 5 minutes (300 seconds), has input and output counters that store data for the average and max counts and stores enough samples for hourly, daily, monthly and yearly graphs of both average and max.

import sys
import rrdtool
 
ret = rrdtool.create("net.rrd", "--step", "300", "--start", '0',
 "DS:input:COUNTER:600:U:U",
 "DS:output:COUNTER:600:U:U",
 "RRA:AVERAGE:0.5:1:600",
 "RRA:AVERAGE:0.5:6:700",
 "RRA:AVERAGE:0.5:24:775",
 "RRA:AVERAGE:0.5:288:797",
 "RRA:MAX:0.5:1:600",
 "RRA:MAX:0.5:6:700",
 "RRA:MAX:0.5:24:775",
 "RRA:MAX:0.5:444:797")
 
if ret:
 print rrdtool.error()

After running you will have a file called net.rrd in the current directory. This is your RRD and will contain all the samples for your graphs.

For more information on the options to rrd_create see: rrdcreate

Updating a RRD

The next step is to update your RRD on the frequency you set when you created it. In the case above the frequency was set to 5 minutes (300 seconds). The following script generates random input and output values as input to the update function, sleeps for 300 seconds and then loops.

import sys
import time
import random
import rrdtool
 
total_input_traffic = 0
total_output_traffic = 0
 
while 1:
 total_input_traffic += random.randrange(1000, 1500)
 total_output_traffic += random.randrange(1000, 3000)
 ret = rrdtool.update('speed.rrd','N:' + `total_input_traffic` + ':' + `total_output_traffic`);
 if ret:
 print rrdtool.error()
 time.sleep(300)

Displaying RRD data as a graph

Now that you have data in your RRD you will want to graph it. The following code will graph the average input and output for 1 day as well as the max for that day.

import sys
import rrdtool
 
ret = rrdtool.graph( "net.png", "--start", "-1d", "--vertical-label=Bytes/s",
 "DEF:inoctets=test1.rrd:input:AVERAGE",
 "DEF:outoctets=test1.rrd:output:AVERAGE",
 "AREA:inoctets#00FF00:In traffic",
 "LINE1:outoctets#0000FF:Out traffic\\r",
 "CDEF:inbits=inoctets,8,*",
 "CDEF:outbits=outoctets,8,*",
 "COMMENT:\\n",
 "GPRINT:inbits:AVERAGE:Avg In traffic\: %6.2lf %Sbps",
 "COMMENT:  ",
 "GPRINT:inbits:MAX:Max In traffic\: %6.2lf %Sbps\\r",
 "GPRINT:outbits:AVERAGE:Avg Out traffic\: %6.2lf %Sbps",
 "COMMENT: ",
 "GPRINT:outbits:MAX:Max Out traffic\: %6.2lf %Sbps\\r")

You should end up with a file named net.png in your working directory that looks something like:

You can also do weekly and monthly graphs by changing the start parameter to -1w or -1m:

For more information on the options to rrd_graph see: rrdgraph

Paramiko is a module for python 2.2 (or higher) that implements the SSH2 protocol for secure (encrypted and authenticated) connections to remote machines.

Emphasis is on using SSH2 as an alternative to SSL for making secure connections between python scripts. All major ciphers and hash methods are supported. SFTP client and server mode are both supported too.

Installing paramiko

First, we need to install paramiko, if you don’t have it already.

On Ubuntu/Debian:

$ sudo apt-get install python-paramiko

On Gentoo Linux:

$ emerge paramiko

Or install from source:

$ wget http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz
$ tar xzf paramiko-1.7.6.tar.gz
$ cd paramiko-1.7.6
$ python setup.py build
$ su -c "python setup.py install"

Working with paramiko

SSHClient is the main class provided by the paramkio module. It provides the basic interface you are going to want to use to instantiate server connections. The above code creates a new SSHClient object, and then calls ”connect()” to connect us to the local SSH server.

Here’s a simple example:

import paramiko
ssh = paramiko.SSHClient()
ssh.connect('192.168.1.2', username='vinod', password='screct')

Another way is to use an SSH key:

import paramiko
import os
privatekeyfile = os.path.expanduser('~/.ssh/id_rsa')
mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile)
ssh.connect('192.168.1.2', username = 'vinod', pkey = mykey)

Running Simple Commands

Lets run some simple commands on a remote machine.

import paramiko
 
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('beastie', username='vinod', password='secret')
stdin, stdout, stderr = ssh.exec_command('df -h')
print stdout.readlines()
ssh.close()

“paramiko.AutoAddPolicy()” which will auto-accept unknown keys.

Using sudo in running commands:

import paramiko
 
cmd    = "sudo /etc/rc.d/apache2 restart"
 
ssh    = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('beastie', username='vinod', password='secret')
stdin, stdout, stderr = ssh.exec_command(cmd)
stdin.write('secret\n')
stdin.flush()
print stdout.readlines()
ssh.close()

Secure File Transfer Using SFTPClient

SFTPClient is used to open an sftp session across an open ssh Transport and do remote file operations.

An SSH Transport attaches to a stream (usually a socket), negotiates an encrypted session, authenticates, and then creates stream tunnels, called Channels, across the session. Multiple channels can be multiplexed across a single session (and often are, in the case of port forwardings).

First we will create a Transport

import paramiko
import os
privatekeyfile = os.path.expanduser('~/.ssh/id_rsa')
mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile)
username = 'vinod'
transport.connect(username = username, pkey = mykey)

Now we can start the SFTP client:

sftp = paramiko.SFTPClient.from_transport(transport)

Now lets pull a file across from the remote to the local system:

remotepath='/var/log/system.log'
localpath='/tmp/system.log'
sftp.get(remotepath, localpath)

Now lets push a file to remote system:

remotepath='/var/www/images/file.png'
localpath='/tmp/file.png'
sftp.put(remotepath, localpath)

Finally, close the SFTP connection and the transport:

sftp.close()
transport.close()

Happy SSHing