I often need to produce high level traffic for testing purposes.This can be http, ldap or SOAP traffic etc.
To achieve that I have written two python scripts that combined they produce high traffic using python’s multithreading and multiprocessing capabilities. The below code can work on any machine (even with only one CPU).
There is an excellent post on web about Multiprogramming, Multitasking, Multithreading and Multiprocessing. So please read this first if those terms are a bit confusing to you – as it was for me.
Python Multithreading script
Now let’s see our first script called
The first python script accepts two arguments. The first argument is the number of threads we want and the second argument is the duration we want each thread to run.
The main procedure implements the multithreading. Each thread starts and executes until a event is sent to the thread when timer -controlled by duration parameter – expires. Each thread executes the operation procedure.
The oparation procedure is where your specific code comes in. Here just for example we print the pid of the process and the instance of thread running. So you can replace the highlighted line with your own code to be executed.
#!/usr/bin/python import threading import logging, sys import time import argparse import os def operation(thr,stop_event): while(not stop_event.is_set()): print 'pid: ' + str(os.getpid()) + ' ' + 'thr: ' + str(thr) + '\n' def main(args): threads,duration=args threads_1 =  events =  for thr in range(threads): e = threading.Event() t = threading.Thread(target=operation, args=(thr,e)) events.append(e) threads_1.append(t) for t in threads_1: t.start() time.sleep(duration) for e in events: e.set() alive =  for t in threads_1: alive.append(t.isAlive()) while any(alive)==True: alive =  for t in threads_1: alive.append(t.isAlive()) print "All threads finished..." if __name__ == '__main__': if len(sys.argv) == 1: #No arguments passed, go for default values threads = 10 duration = 60 args = threads,duration main(args) else: args = sys.argv,sys.argv main(args)
Python Multiprocessing script
So far the above script can run as standalone. It will create some threads that will run for a pre-defined duration. This will only utilize on CPU of the system.
The script below called
multiprocess.py will create a pool and execute concurrently
request.py as many times as CPU’s in the system, creating one process per CPU.
#!/usr/bin/python from request import main import multiprocessing from multiprocessing import Pool import os import sys import argparse def check_positive(value): ivalue = int(value) if ivalue <= 0: raise argparse.ArgumentTypeError("Only positive int values allowed") return ivalue parser = argparse.ArgumentParser(description='Multiprocess requests') parser.add_argument('-t','--threads', help='Threads', required=True, type=check_positive) parser.add_argument('-d','--duration', help='Test Duration (seconds)', required=True, type=check_positive) args = parser.parse_args() cpu_count = multiprocessing.cpu_count() test_duration = args.duration thr = args.threads duration = [test_duration] * cpu_count threads = [thr] * cpu_count #print threads #print duration if __name__ == '__main__': pool = Pool(cpu_count) result = pool.map(main, zip(threads, duration))
You can now run the code by executing:
./multiprocess.py -t 10 -d 60
x processes will be created (where x is the number of your CPU’s) and each process will run for 60 seconds creating 10 threads each.