DevHeads IoT Integration Server•3mo ago

Improving Gesture Recognition Consistency on ESP32 with TinyML

Hello, I'm working on a gesture recognition project using TinyML on an ESP32 with an accelerometer (MPU6050). My goal is to detect specific gestures (e.g., wave, swipe) using machine learning. I trained a model using Edge Impulse and successfully deployed it onto the ESP32. However, when I run the inference code, I get inconsistent results, and sometimes the output is incorrect even when performing the same gesture. Occasionally, the ESP32 throws the following error:

E (1155) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
 - IDLE (CPU 0)
 - ml_task
Tasks currently running:
CPU 0: ml_task
CPU 1: IDLE

E (1155) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
 - IDLE (CPU 0)
 - ml_task
Tasks currently running:
CPU 0: ml_task
CPU 1: IDLE

Here is the inference code I'm running on the ESP32:

from machine import Pin, I2C
from mpu6050 import MPU6050
import time
import numpy as np
from tinyml_model import predict_gesture  # Edge Impulse generated model

# Initialize MPU6050
i2c = I2C(0, scl=Pin(22), sda=Pin(21))
mpu = MPU6050(i2c)

# Function to collect data from MPU6050
def get_sensor_data():
    accel = mpu.accel
    return np.array([accel.x, accel.y, accel.z])

# Main loop for gesture recognition
while True:
    try:
        data = get_sensor_data()
        gesture = predict_gesture(data)  # Run inference on collected data

        if gesture == "wave":
            print("Wave gesture detected!")
        elif gesture == "swipe":
            print("Swipe gesture detected!")
        else:
            print("No gesture detected")

        time.sleep(0.5)  # Delay to avoid rapid re-inference

    except Exception as e:
        print("Error:", e)
        time.sleep(1)

from machine import Pin, I2C
from mpu6050 import MPU6050
import time
import numpy as np
from tinyml_model import predict_gesture  # Edge Impulse generated model

# Initialize MPU6050
i2c = I2C(0, scl=Pin(22), sda=Pin(21))
mpu = MPU6050(i2c)

# Function to collect data from MPU6050
def get_sensor_data():
    accel = mpu.accel
    return np.array([accel.x, accel.y, accel.z])

# Main loop for gesture recognition
while True:
    try:
        data = get_sensor_data()
        gesture = predict_gesture(data)  # Run inference on collected data

        if gesture == "wave":
            print("Wave gesture detected!")
        elif gesture == "swipe":
            print("Swipe gesture detected!")
        else:
            print("No gesture detected")

        time.sleep(0.5)  # Delay to avoid rapid re-inference

    except Exception as e:
        print("Error:", e)
        time.sleep(1)

I suspect the issue might be related to timing or resource limitations on the ESP32. How can I fix the watchdog error and improve the consistency of gesture detection? Should I adjust the sampling rate, modify the inference loop, or implement additional optimizations for TinyML on the ESP32?

3 Replies

ZacckOsiemo•3mo ago

There are two issues here likely to be addressed individually - Your Model's accuracy - Task starvavation caused by the ml_task, seems its not yielding in time or stuck, your watchdog might also be misconfigured, also maybe you have work in your idle task that isn't yielding

wafa_ath•3mo ago

Thanks! I think you're right about the model accuracy. And for task starvation, how can I make sure ml_task yields properly and set up the watchdog to avoid triggering?

ZacckOsiemo•3mo ago

We've had quite a few session on rtoses and how to use them, they should come in handy in this situation.

Gaming

Programming

Improving Gesture Recognition Consistency on ESP32 with TinyML