[Solved] How can I execute a function on a CPU core, and get a callback when it has completed?

P i Asks: How can I execute a function on a CPU core, and get a callback when it has completed?
How can I execute a function on a CPU core, and get a callback when it has completed?


Context​

I’m recieving a stream:

Code:
symbols = ['ABC', 'DFG', ...]  # 52 of these

handlers = { symbol: Handler(symbol) for symbol in symbols }

async for symbol, payload in lines:  # 600M of these
    handlers[symbol].feed(payload)

I need to make use of multiple CPU cores to speed it up.

handler['ABC'] (e.g.) holds state, but it’s disjoint from the state of (e.g.) handler['DFG']

Basically I can’t have 2 cores simultaneously operating e.g. handler['ABC'].


My approach so far​

I have come up with the following solution, but it’s part pseudocode, as I can’t see how to implement it.

Code:
NCORES = 4
symbol_curr_active_on_core = [None]*NCORES

NO_CORES_FREE = -1
def first_free_core():
    for i, symbol in enumerate(symbol_curr_active_on_core):
        if not symbol:
            return i
    return NO_CORES_FREE

for symbol, payload in lines:
    # wait for avail core to handle it

    while True:
        sleep(0.001)
        if first_free_core() == NO_CORES_FREE:
            continue
        if symbol in symbol_curr_active_on_core:
            continue
        core = first_free_core()
        symbol_curr_active_on_core[core] = symbol

        cores[core].execute(
            processor[symbol].feed(payload),
            on_complete=lambda core_index: 
                symbol_curr_active_on_core[core_index] = None
        )

So my question is specifically: How to convert that last statement into working Python code?

Code:
        cores[core].execute(
            processor[symbol].feed(payload),
            on_complete=lambda core_index: 
                symbol_curr_active_on_core[core_index] = None
        )

PS More generally, is my approach optimal?

Ten-tools.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, Ten-tools.