forked.set_trace() rendering multiple GUIs in terminal that break each other called from Multiprocess, is there a better way to do this?

[aaronjolson]

I have some code for taking in stream data from kafka (two different topics, 1 topic per process).

from confluent_kafka import DeserializingConsumer
from multiprocessing import Process, Manager, Lock


def process_one(lock, event_store):

    consumer = DeserializingConsumer(
        {
            ...
        }
    )
    consumer.subscribe([topic])
    iterations = 0
    while True:
        message = consumer.poll(1)
        iterations += 1
        if iterations == 100:
            iterations = 0

            from pudb import forked
            forked.set_trace()
        if message is None:
            print(f'message is None')
            continue

        data = message.value()
        lock.acquire()
        if data['Foo']['Bar'] in event_store:
            event_store[data['Foo']['Bar']]['baz'].append(data)
        else:
            event_store[data['Foo']['Bar']] = {'baz': [], 'boo': []}
            event_store[data['Foo']['Bar']]['baz'].append(data)
        lock.release()

        print()
        print(data)


def process_two(lock, event_store):
    consumer = DeserializingConsumer(
        {
            ...
        }
    )

    consumer.subscribe([topic])
    print("Now subscribing to shipexec stream!", flush=True)
    iterations = 0
    while True:
        message = consumer.poll(1)
        iterations += 1
        if iterations == 100:
            iterations = 0
            from pudb import forked
            forked.set_trace()
        if message is None:
            continue

        data = message.value()

        lock.acquire()
        if data['Baz']['Bar'] in event_store:
            event_store[data['Bar']['Foo']]['boo'].append(data)
        else:
            event_store[data['Bar']['Foo']] = {'baz': [], 'boo': []}
            event_store[data['Bar']['Foo']]['boo'].append(data)
        lock.release()

        print()
        print(data)


def main():
    manager = Manager()
    lock = Lock()
    event_store = manager.dict()

    c1 = Process(target=process_one, args=(lock, event_store))
    c2 = Process(target=process_two, args=(lock, event_store))

    c1.start()
    c2.start()
    c1.join()
    c2.join()


if __name__ == '__main__':
    main()

I run this in a docker container. I exec into the docker container and run this script.
There is an issue there when one of process hits the set_trace(), the second process keeps going, eventually it hits the set_trace() as well and a second instance of pudb tries to spawn in the same terminal and it renders over top of the previous one and things get really screwed up. Is there a better way to do this?

[mvanderkamp]

I think you'll need to get the debuggers running in different terminals. You could try debugging from a separate terminal or remote debugging. This could get a bit more tricky inside a docker container, it's not something I've personally tried.