python - How to detect multiprocessing.Pipe is full? -
problem description: doing multiprocessing in python , using multiprocessing.pipe() communicate between processes. have been searching lot still couldn't find way detect whether pipe full or not. example below, writepipe process keeps putting number 2 different pipes (odd & even) , readpipe process continuously read these 2 pipes. however, speed of read odd pipe faster pipe full. @ moment, writepipe process blocked while readpipe process still keep waiting read odd pipe causes deadlock.
my question: there way can detect pipe full can stop putting number full pipe while still running , put number pipe still has spaces?
from multiprocessing import process, pipe def writepipe(sendnumberodd, sendnumbereven): = 0 while true: if % 2 == 0: sendnumbereven.send(i) else: sendnumberodd.send(i) += 1 def readpipe(recvnumberodd, recvnumbereven): counteven = 0 while true: counteven += 1 print(counteven, recvnumbereven.recv()) countodd = 0 while countodd < 50: countodd += 1 print (countodd, recvnumberodd.recv()) if __name__ == '__main__': recvnumberodd, sendnumberodd = pipe(duplex=false) recvnumbereven, sendnumbereven = pipe(duplex=false) write = process(target=writepipe, args=(sendnumberodd, sendnumbereven)) read = process(target=readpipe, args=(recvnumberodd, recvnumbereven)) write.start() read.start() sendnumberodd.close() sendnumbereven.close()
proposal untested
class connection(multiprocessing.connection): def __init__(self, maxsize=0): self.__maxsize = maxsize self.size = 0 self.__lock = multiprocessing.lock def send(self, obj): self.__lock: self.size += sizeof(obj) super().send(obj) def recv(self): _recv = super().recv() self.__lock: self.size -= sizeof(_recv) return _recv def full(self): if self.__maxsize > 0: return self.size >= self.__maxsize return none def pipe(maxsize=0, duplex=true): return connection(maxsize), connection(maxsize)
implement poll() check if data ready.
python » documentation: poll([timeout])
return whether there data available read.
for example:
if recvnumbereven.poll(): counteven += 1 print(counteven, recvnumbereven.recv()) alternative use wait(...) both
multiprocessing.connection.wait(object_list, timeout=none)
wait till object in object_list ready. returns list of objects in object_list ready.
Comments
Post a Comment