I often need to produce high level traffic for testing purposes.This can be http, ldap or SOAP traffic etc.
To achieve that I have written two python scripts that combined they produce high traffic using python’s multithreading and multiprocessing capabilities. The below code can work on any machine (even with only one CPU).

There is an excellent post on web about Multiprogramming, Multitasking, Multithreading and Multiprocessing. So please read this first if those terms are a bit confusing to you – as it was for me.

Python Multithreading script

Now let’s see our first script called request.py

The first python script accepts two arguments. The first argument is the number of threads we want and the second argument is the duration we want each thread to run.

The main procedure implements the multithreading. Each thread starts and executes until a event is sent to the thread when timer -controlled by duration parameter – expires. Each thread executes the operation procedure.

The oparation procedure is where your specific code comes in. Here just for example we print the pid of the process and the instance of thread running. So you can replace the highlighted line with your own code to be executed.

#!/usr/bin/python

import threading
import logging, sys
import time
import argparse
import os


def operation(thr,stop_event):
	while(not stop_event.is_set()):
		print 'pid: ' + str(os.getpid()) + ' ' + 'thr: ' + str(thr) + '\n'



def main(args):
	threads,duration=args
	threads_1 = []
	events = []
	for thr in range(threads):
		e = threading.Event()
		t = threading.Thread(target=operation, args=(thr,e))
		events.append(e)
		threads_1.append(t)
			
	for t in threads_1: t.start()
	

	time.sleep(duration)
	for e in events: e.set()
	
	
	alive = []
	for t in threads_1:
		alive.append(t.isAlive())
	while any(alive)==True:
		alive = []
		for t in threads_1:
			alive.append(t.isAlive())
			
	print "All threads finished..."


if __name__ == '__main__':
	if len(sys.argv) == 1: #No arguments passed, go for default values
		threads = 10
		duration = 60
		args = threads,duration
		main(args)
	else:
		args = sys.argv[1],sys.argv[2]
		main(args)

 

 

Python Multiprocessing script

So far the above script can run as standalone. It will create some threads that will run for a pre-defined duration. This will only utilize on CPU of the system.

The script below called multiprocess.py will create a pool and execute concurrently request.py as many times as CPU’s in the system, creating one process per CPU.

#!/usr/bin/python

from request import main
import multiprocessing
from multiprocessing import Pool
import os
import sys
import argparse


def check_positive(value):
	ivalue = int(value)
	if ivalue <= 0:
		 raise argparse.ArgumentTypeError("Only positive int values allowed")
	return ivalue

parser = argparse.ArgumentParser(description='Multiprocess requests')
parser.add_argument('-t','--threads', help='Threads', required=True, type=check_positive)
parser.add_argument('-d','--duration', help='Test Duration (seconds)', required=True, type=check_positive)
args = parser.parse_args()

cpu_count = multiprocessing.cpu_count()

test_duration = args.duration
thr = args.threads
duration = [test_duration] * cpu_count
threads = [thr] * cpu_count

#print threads
#print duration


if __name__ == '__main__':
	pool = Pool(cpu_count) 
	result = pool.map(main, zip(threads, duration))

 

You can now run the code by executing:

./multiprocess.py -t 10 -d 60

x processes will be created (where x is the number of your CPU’s) and each process will run for 60 seconds creating 10 threads each.