I have a piece of code that queries a DB and returns a set of IDs. For each ID, I need to run a related query to get a dataset. I would like to run the queries in parallel to speed up the processing. Once all the processes are run, then I build a block of text and write that to a file, then move to the next id.
Process object is not iterable(on line 9).
Here is what I have so far:
from helpers import *
idSet = getIDset(10)
for id in idSet:
ds1 = multiprocessing.Process(target = getDS1(id))
ds1list1, ds1Item1, ds1Item2 = (ds1)
ds2 = multiprocessing.Process(target = getDS2(id))
ds3 = multiprocessing.Process(target = getDS3(id))
ds4 = multiprocessing.Process(target = getDS4(id))
ds5 = multiprocessing.Process(target = getDS5(id))
movefiles = multiprocessing.Process(moveFiles(srcPath = r'Z://', src = ds1Item2 , dstPath=r'E:/new_data_dump//'))
## is there a better way to get them to start in unison than this?
## how do I know all processes are finished before moving on?
page = +ds1+'\n' \
I usually keep my "processes" in a list.
plist =  for i in range(0, 5) : p = multiprocessing.Process(target = getDS2(id)) plist.append(p) for p in plist : p.start() ... do stuff ... for p in plist : p.join() # <---- this will wait for each process to finish before continuing
Also I think you have an issue with creating your Process. "target" is supposed to be a function. Not the result of a function as it seems you have it (unless your function returns functions).
It should look like this:
p = Process(target=f, args=('bob',))
Where target is the function, and args is a tuple of arguemnts passed like so:
def f(name) : print name