Event racing condition

[krishnak]

I have an asyncio method which sets this

responseQueue.append(msg)
                    responseQueueEvent.set()
                    responseQueueEvent.clear()

The method which is waiting on responseQueueEvent is as below, the responseQueue object is a global variable
There are occassions where the event is set but the queue is empty at the waiting method, it is not that the msg variable is empty. The append function appears to be completing after Event.set() is called which results in the waiting function to proceed with an empty list. How do I avoid this

The same code works OK 80% of the time.

try:
               await asyncio.wait_for(responseQueueEvent.wait(),5)
               print("No time out leaving as response received")
               if len(responseQueue)>0:
                   print("queue has values")

           except asyncio.TimeoutError:  
               print("No response received")
               continue

[karfas]

What exactly are these responseQueue and responseQueueEvent objects ?
The asyncio queue found at https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py has a different interface and it's own event handling.
However, I don't see the the benefit from using an event for the queue. The task who reads from the queue could wait (maybe using asyncio.sleep(0)) until at least one entry is in the queue.

[krishnak]

It makes sense if the waiter has timed out and there is no one to clear a stale event.

[peterhinch]

@jimmo My understanding is that it is valid to immediately clear an Event if you can guarantee that the waiting task(s) are paused waiting on the Event at the time it is set. If the waiting task hasn't reached the Event at the time it is cleared, then events will be missed. That said, I too would recommend clearing it in the waiting task.

The symptom we have here is that the Event seems to be being set when the queue is empty: too many triggers rather than too few. On the basis of the code presented this doesn't seem to make sense. Nor do the symptoms relating to globals and GC. Something else is going on. Are interrupts or threaded code involved anywhere?

I can see two options to try to debug this:

1. Use print statements to show queue contents when the event is set and when it is awaited.
2. Alternatively write a minimal test case and gradually extend it until it becomes the full solution.

[krishnak]

Hello Peter

There are no threads or interrupts, just asyncio. When the bug occured - the print statements printing the queue will print expected values - but this will be always printed AFTER the code exits the anothermethodwhichiswaiting - inside this method the queue will be always empty.
However the place where it gets filled, i.e process message - the valid values get filled but it gets printed only after the waiting method exits.

I have tried to set high time outs like 10 seconds - irrespective of that, the queue value always gets printed afterwards. So my theory of GC sounds illogical.

My code has undergone major transformation in the last 24 hours and I have moved away from this problem. I can try to reproduce this problem at a later date and if reproduce-able, I shall update. For time being please treat this as resolved.

[jimmo]

@krishnak just a quick note that there seems to be a misunderstanding here about how global works.

You only need to use the global keyword if your code assigns to a variable. Your code doesn't assign to responseQueue -- it calls .append() and presuably .pop().

[krishnak]

After re-reading your comment, I understood what you are meaning, it appears to be a redundant declaration at the start of each function.

When the issue occured, I had declared myqueue = [] at the start of the file, I shifted that inside the main method which starts the async tasks. This immediately resolved the issue. To recreate the problem I assigned another queue with my current code similarly at the start of the code when I encountered the problem. I am not able to repoduce the problem.

At that time of my issue, I had a simple code which operated on a single frequency, it has now transformed in to multi frequency hoping code with couple of locks. Hence if at all the issue occurs it should occur now. I appreciate the effort you all show here to resolve the reported issue. I don't want you to waste your time on this unless I am able to reproduce it again.