Python Threads and Queues

August 11, 2009

Introduction

Threads are a technique by which multiple paths of execution can be run in one program at "the same time." Actually the threads do not run at the same physical time. But they run concurrently and asynchronously. This can be a great help in program design. If two threads can be independent of each other, it frees the programmer from the need to know which steps have priority.

Also, some things take a lot of "system" time. For, instance writing to disk can take hundreds of times longer than adding two numbers. If one thread is "blocked" waiting for disk I/O to complete, another thread can be busy doing math.

Threads can be different blocks of code, multiple "copies" of the same block of code, or a combination of these.

Python has thread support on any OS which has pthreads or Posix Threads. Threads are actually more portable than other methods of running multiple processes. For instance, as of Python 2.5, os.fork() is not supported on Windows.

Let us look at a simple threading program, and its output.

#!/usr/bin/env python
# sample1.py 

import thread
from threading import *
from time import sleep

def counter():
    for i in range(3):
      print thread.get_ident(),i
      # sleep(.01)

## MAIN ##
thrlist = []
for i in range(3):
    thrlist.append(Thread(target=counter))
for thr in thrlist:
    thr.start()
    # thr.join()

print 'PROGRAM ENDING'


PROGRAM ENDING
-1211495536 0
-1211495536 1
-1211495536 2
-1219888240 0
-1219888240 1
-1219888240 2
-1228280944 0
-1228280944 1
-1228280944 2

We import thread and threading. Primarily we use the the higher level threading module, but we will also access thread module functions.

In the MAIN section we create three threads of the function 'counter' using Thread(). And we append these to a list. The "for each" thread in 'thrlist' we call start(). The function counter() simply loops three times and prints the thread identity and the counter 'i'.

Now let us uncomment the sleep() statement. We get the following output:

PROGRAM ENDING
-1210754160 0
-1219146864 0
-1227539568 0
-1210754160 1
-1219146864 1
-1227539568 1
-1210754160 2
-1219146864 2
-1227539568 2

Each thread is still counting. but the sleep() allowed each thread time to take control and execute. Therefore now the output is interleaved, with each thread having a turn.

Notice that PROGRAM ENDING is printing out at the beginning rather than the end. This is because the main loop is executing faster than the threads can start running. If there are things which you need to do after a thread or threads have completed, then you can use join() to block until the completion of a thread. If we uncomment the join() statement we get:

-1211245680 0
-1211245680 1
-1211245680 2
-1219638384 0
-1219638384 1
-1219638384 2
-1211245680 0
-1211245680 1
-1211245680 2
PROGRAM ENDING

Python Threads and Queues
Python Threads and Queues - Page 2