GUI b
DELAY 1000
STRING 1111
STRING 2222
STRING 3333
STRING 4444
STRING 5555
STRING 6666
STRING 7777
STRING 8888
STRING 9999
STRING 0000
STRING 0123
STRING 0124
STRING 0125

но есть проблема, устройство блокируется. Можем ли мы решить эту проблему и как.
Прошивка - bruce
Устройство M5 Stick Plus2

To my surprise, I made some changes, repeated my process - but the update never happened.

That's when I found out the name of the binary file was changed from:
m5stick_c_plus to m5stack_stickc_plus

I haven't found any documentation on the name of the binary, but I assume it related to the board (and mine is still: M5StickCPlus).

I had been using library 2.0.4 but then upgraded to 2.0.7
and now I see the version changed to: 0.4.3
And the web page says to use the M5Unified version. For me that's
a bridge too far right now.

Arduion is version 1.8.19

Is there any documentation on the versions and how the binary names are created? I did just see this (thanks). https://docs.m5stack.com/en/platform

I especially need to have a binary that is the same memory configuration as my 2.0.4 M5 and ESP32 since the OTA has to work for 100's of installed versions.

thanks.

thanks,

Introduction

Translate AirPods through the iPhone app to Express, then through python-shell to pymycobot, and finally synchronize with myCobot to synchronize the rotation angle of AirPods with the posture of myCobot 🤖.

Although it's unclear if there's a demand, I will provide a rough outline of the source code.

Project Structure

This project primarily employs technologies such as headphone-motion, web servers, the express framework of node.js, python-shell, and pymycobot. Here is a brief introduction to these technologies:
1.Headphone-Motion: Headphone Motion is a project that utilizes specific technology to track and utilize user head movements. Although the specifics may depend on the implementation method and platform used (such as iOS), it mainly demonstrates how to capture head movement data by connecting to the device's headphones (especially those smart headphones with built-in sensors). A more intuitive effect of this can be seen in the Headphone Motion Unity Plugin made by GitHub user anastasiadeana. It is capable of real-time tracking of the user's head movements, including tilting and rotating, which is a core technology of this project.

2.Web Server: There are many types of servers, which provide data, services, or applications to other applications or devices. Servers perform tasks such as processing data requests, hosting websites, storing information, running business applications, and so on. In this project, the web server mainly handles receiving head movement data from the iOS application and passing these data to the script controlling the mycobot robot arm.

3.Express-Node.js: Express is a fast, open, minimalistic web application framework for Node.js. It is designed for building web applications and APIs. It allows developers to set up middleware to respond to HTTP requests in a very fast and simple way, making the development of web applications quicker and easier.
https://github.com/expressjs/express

4.Pymycobot-Python: pymycobot is a Python library specifically designed for controlling the myCobot robot arm. This library provides a series of functions and interfaces that allow developers to communicate and control the myCobot robot arm directly through Python scripts. Using pymycobot, developers can write code to control the robot arm's movements, adjust its posture, execute preset action sequences, etc., making it widely applicable in education, research, automation, and more.

iOS App

This application is based on an app I created previously, utilizing react-native-headphone-motion to access the sensors in AirPods through a React Native iOS application.

It's an interesting project where, when you are detected bending your head down for a long time while wearing AirPods, a lamenting sound will remind you.
https://twitter.com/i/status/1745588902981931443
https://blog.kimizuka.org/entry/2024/01/16/105208

One point to note is the change I made: I added a process to send POST requests to the web server in onDeviceMotionUpdates. Additionally, to avoid overburdening the server with POST requests on every update, I set it up to send a request at a minimum interval of 500ms.

App.tsx

useEffect(() => {
  const delay = 500;
  const handleDeviceMotionUpdates = onDeviceMotionUpdates((data) => {
     // If the time since the last request is less than 500ms, return
    if (Date.now() - lastUpdateTimeRef.current < delay) {
      return;
    }

    // Post sensor values to the Web server
    axios.post(String(process.env.API_URL), { 
      pitch: data.attitude.pitchDeg || 0,
      roll: data.attitude.rollDeg || 0,
      yaw: data.attitude.yawDeg || 0
    }).then(() => {
      lastUpdateTimeRef.current = Date.now();
    }).catch((err) => {
      console.error(err);
      lastUpdateTimeRef.current = Date.now();
    });

    setPitch(data.attitude.pitch);
    setPitchDeg(data.attitude.pitchDeg);
    setRoll(data.attitude.roll);
    setRollDeg(data.attitude.rollDeg);
    setYaw(data.attitude.yaw);
    setYawDeg(data.attitude.yawDeg);
    setGravityX(data.gravity.x);
    setGravityY(data.gravity.y);
    setGravityZ(data.gravity.z);
    setRotationRateX(data.rotationRate.x);
    setRotationRateY(data.rotationRate.y);
    setRotationRateZ(data.rotationRate.z);
    setUserAccelerationX(data.userAcceleration.x);
    setUserAccelerationY(data.userAcceleration.y);
    setUserAccelerationZ(data.userAcceleration.z);
  });

  return () => {
    handleDeviceMotionUpdates.remove();
  };
}, []);

In the POST request, I used axios, which can send asynchronous HTTP requests to REST endpoints and handle responses.
Therefore, it's also necessary to add module imports.

import axios from 'axios';

Code

import axios from 'axios'; // Added to simplify the POST request
import React, {
  useEffect,
  useRef, // Added to maintain a 500ms interval
  useState,
} from 'react';
import {Button, SafeAreaView, StyleSheet, Text} from 'react-native';
import {
  requestPermission,
  onDeviceMotionUpdates,
  startListenDeviceMotionUpdates,
  stopDeviceMotionUpdates,
} from 'react-native-headphone-motion';

const API_URL = 'http://localhost:3000'; // Fill in the URL to POST

export default function App() {
  const lastUpdateTimeRef = useRef<number>(0); // Added to keep track of the last update time
  const [pitch, setPitch] = useState(0);
  const [pitchDeg, setPitchDeg] = useState(0);
  const [roll, setRoll] = useState(0);
  const [rollDeg, setRollDeg] = useState(0);
  const [yaw, setYaw] = useState(0);
  const [yawDeg, setYawDeg] = useState(0);
  const [gravityX, setGravityX] = useState(0);
  const [gravityY, setGravityY] = useState(0);
  const [gravityZ, setGravityZ] = useState(0);
  const [rotationRateX, setRotationRateX] = useState(0);
  const [rotationRateY, setRotationRateY] = useState(0);
  const [rotationRateZ, setRotationRateZ] = useState(0);
  const [userAccelerationX, setUserAccelerationX] = useState(0);
  const [userAccelerationY, setUserAccelerationY] = useState(0);
  const [userAccelerationZ, setUserAccelerationZ] = useState(0);

  useEffect(() => {
    const delay = 500; // Store the update interval in a variable
    const handleDeviceMotionUpdates = onDeviceMotionUpdates(data => {
      if (Date.now() - lastUpdateTimeRef.current < delay) {
        // Return if the update interval is not met
        return;
      }

      // Post sensor values to the Web server
      // Update lastUpdateTimeRef regardless of success or failure
      // For some reason, await was not used
      axios
        .post(String(API_URL), {
          pitch: data.attitude.pitchDeg || 0,
          roll: data.attitude.rollDeg || 0,
          yaw: data.attitude.yawDeg || 0,
        })
        .then(() => {
          lastUpdateTimeRef.current = Date.now();
        })
        .catch(err => {
          console.error(err);
          lastUpdateTimeRef.current = Date.now();
        });

      setPitch(data.attitude.pitch);
      setPitchDeg(data.attitude.pitchDeg);
      setRoll(data.attitude.roll);
      setRollDeg(data.attitude.rollDeg);
      setYaw(data.attitude.yaw);
      setYawDeg(data.attitude.yawDeg);
      setGravityX(data.gravity.x);
      setGravityY(data.gravity.y);
      setGravityZ(data.gravity.z);
      setRotationRateX(data.rotationRate.x);
      setRotationRateY(data.rotationRate.y);
      setRotationRateZ(data.rotationRate.z);
      setUserAccelerationX(data.userAcceleration.x);
      setUserAccelerationY(data.userAcceleration.y);
      setUserAccelerationZ(data.userAcceleration.z);
    });

    return () => {
      handleDeviceMotionUpdates.remove();
    };
  }, []);

  return (
    <SafeAreaView style={styles.container}>
      <Button
        title={'requestPermission'}
        onPress={async () => {
          await requestPermission();
        }}
      />
      <Button
        title={'startListenDeviceMotionUpdates'}
        onPress={async () => {
          await startListenDeviceMotionUpdates();
        }}
      />
      <Button
        title={'stopDeviceMotionUpdates'}
        onPress={async () => {
          await stopDeviceMotionUpdates();
        }}
      />
      <Text>{lastUpdateTimeRef.current}</Text>
      <Text>{`pitch: ${pitch}`}</Text>
      <Text>{`pitchDeg: ${pitchDeg}`}</Text>
      <Text>{`roll: ${roll}`}</Text>
      <Text>{`rollDeg: ${rollDeg}`}</Text>
      <Text>{`yaw: ${yaw}`}</Text>
      <Text>{`yawDeg: ${yawDeg}`}</Text>
      <Text>{`gravityX: ${gravityX}`}</Text>
      <Text>{`gravityY: ${gravityY}`}</Text>
      <Text>{`gravityZ: ${gravityZ}`}</Text>
      <Text>{`rotationRateX: ${rotationRateX}`}</Text>
      <Text>{`rotationRateY: ${rotationRateY}`}</Text>
      <Text>{`rotationRateZ: ${rotationRateZ}`}</Text>
      <Text>{`userAccelerationX: ${userAccelerationX}`}</Text>
      <Text>{`userAccelerationY: ${userAccelerationY}`}</Text>
      <Text>{`userAccelerationZ: ${userAccelerationZ}`}</Text>
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    justifyContent: 'center',
    backgroundColor: 'white',
  },
});

Actually, it would be more convenient to specify the API_URL directly in the application, but I implemented it this way considering speed.

Web Server

I set up a local server on my Mac. To operate myCobot, I went through several setup steps, mainly to adapt to Mac computers. These steps included installing drivers for the robot arm, updating the firmware of mycobot 280, and other operations, all detailed in this article.
https://blog.kimizuka.org/entry/2021/08/10/131812
I think creating a web server with Python would be smoother, but based on my skill set, using Node.js is the fastest method. So, I plan to quickly set up the server using Express. Communication with myCobot is conducted through Python, so for this part, I decided to use python-shell to implement it.

require('dotenv').config(); // Used to pass the port of myCobot from outside
const express = require('express');
const { PythonShell } = require('python-shell'); // Used for communicating with myCobot
const app = express();
const http = require('http').Server(app);

const duration = 100; // If the delay (500ms) set on the application side is too small, it will cause problems

app.use(express.json());
app.post('/', (req, res) => {
  try {
    const angles = [0, 0, 0, 0, 0, 0];

    // For myCobot's joint information, refer to page 13 of https://www.elephantrobotics.com/wp-content/uploads/2021/03/myCobot-User-Mannul-EN-V20210318.pdf
    // The array stores the 6 joints in order from the bottom up
    // Each joint has a definite range of motion; ensure not to exceed this range    
    angles[0] = Math.max(-90, Math.min(req.body.yaw || 0, 90)); // J1
    angles[3] = Math.max(-90, Math.min(req.body.pitch || 0, 90)); // J4
    angles[5] = Math.max(-175, Math.min(req.body.roll || 0, 175)); // J6

    // myCobot connected via USB receives instructions from Python
    PythonShell.runString(
      `from pymycobot.mycobot import MyCobot; MyCobot('${ process.env.MY_COBOT_PORT }').send_angles([${ angles }], ${ duration })`,
      null,
      (err) => err && console.error(err)
    );
  } catch (err) {
    console.error(err);
  }
  res.send(200);
});

try {
  const angles = [0, 0, 0, 0, 0, 0];

  // Resets posture at startup
  PythonShell.runString(
    `from pymycobot.mycobot import MyCobot; MyCobot('${ process.env.MY_COBOT_PORT }').send_angles([${ angles }], ${ duration })`,
    null,
    (err) => err && console.error(err)
  );
} catch(err) {
  console.error(err);
}

http.listen(3000, '0.0.0.0');

Because it's necessary to execute pymycobot through PythonShell, the pymycobot directory must be placed at the same level as app.js.
https://github.com/elephantrobotics/pymycobot
Once everything is prepared and the PC is connected to myCobot,
you can start the web server and pass the pitch, roll, and yaw values received through POST requests to myCobot.
Although this time the sensor values of AirPods are sent from an iPhone application via POST, the source of the POST could be from anywhere. Therefore, I think setting up such a server could be potentially useful in the future.

source code:
https://github.com/kimizuka/mycobot-express/tree/example/airpods

Summary

This project showcases the integration of headphone motion detection with robotic control, illustrating the potential of combining human movement data with robotics. By capturing head motion through smart headphones and translating it into commands for a robotic arm, it enhances human-robot interaction and introduces innovative applications in automation and assistive technologies. This approach opens up new avenues for intuitive control mechanisms in robotics, making technology more accessible and interactive.
Finally, thanks again to Kimizuka for sharing this case with us.Feel free to explore further or share your insights with us.

I am a freelancer specializing in machine learning and robotics technology. My passion began during a course in artificial intelligence in college, which inspired me to explore new methods of human-machine interaction. In particular, for the operation of robotic arms, I have always wanted to simplify their complexity to make them more intuitive and easier to use.
The inspiration for this project stems from my love for innovative technology and the pursuit of improving the ways humans interact with machines. My goal is to develop a gesture-based robotic arm control system that allows non-professionals to operate it with ease. For this purpose, I chose Google's MediaPipe library for gesture recognition and used mycobot 320 m5 as the experimental platform.

Technical Overview

Google MediaPipe

MediaPipe is an open-source cross-platform framework developed by Google, specifically designed for building various perception pipelines. This framework offers a wealth of tools and pre-built modules, enabling developers to easily build and deploy complex machine learning models and algorithms, especially in the field of image and video analysis.

A notable feature of MediaPipe is its support for real-time gesture and facial recognition. It can efficiently process video streams and identify and track human gestures and facial features in real-time. This capability makes it incredibly useful in interactive applications, augmented reality (AR), virtual reality (VR), and robotics.

You can try the gesture recognition online feature without needing to install anything.
MediaPipe Studio
Its easy-to-use API and comprehensive documentation make it easier to integrate this framework, making it very suitable for use in the fields of machine learning and computer vision.

pymycobot

pymycobot is a Python API for serial communication and control of the mycobot robotic arm. This library is designed to facilitate developers in controlling the mycobot robotic arm using the Python language. It offers a series of functions and commands that allow users to control the movements and behavior of the robotic arm through programming. For example, users can use the library to get the angles of the robotic arm, send angle commands to control the movement of the arm, or get and send the coordinates of the robotic arm.
The only standard for using this library is that it must be used with the mycobot series of robotic arms, which are specifically adapted for the mycobot.

Product Introduction

myCobot 320 M5stack

The myCobot 320 M5 is a six-axis collaborative robotic arm developed by Elephant Robotics for users. It has a working radius of 350mm and a maximum load capacity of 1000g. The robotic arm is suitable for an open ROS simulation development environment and includes forward and inverse kinematics algorithms. It supports multiple programming languages, including Python, C++, Arduino, C#, and JavaScript, and is compatible with Android, Windows, Mac OSX, and Linux platforms. The versatility of the myCobot 320 M5 makes it suitable for a variety of development and integration applications.

2D Camera

A 2D camera that can be mounted on the end of the mycobot320, communicating via a USB data cable. It can present the view seen from the end of the robotic arm.

Development Process

Project Architecture

I have divided this project primarily into three functional modules:
Gesture Recognition: This module is mainly used for the recognition of gestures, capable of returning information about what the gesture is, such as a thumbs-up, etc.
Robotic Arm Control: This main function is used for setting the motion control of the robotic arm, including coordinate control, angle control, and so on.
Program Logic: This is used to handle the logic of the program's operation, setting confirmation times for gestures, resetting recognition times, etc. These will be detailed further in subsequent sections.

Compilation Environment

Operating System: Windows 11
Programming Language: Python 3.9+
Libraries:opencv,pymycobot,mediapipe,time

Gesture Recognition

To perform gesture recognition, we first need to obtain a camera image. Here, we use the OpenCV library to access the camera feed.

import cv2

# Get camera stream, default camera - 0, external cameras in order - 1, 2, 3
cap = cv2.VideoCapture(1)

# Continuously acquire camera footage
while cap.isOpened():
    #Get the current image screen
    ret, frame = cap.read()
    # Convert BGR image to RGB
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)    
    # display screen on computer
    cv2.imshow('gesture control',frame)
    # Press the 'q' key to exit to avoid an infinite loop
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break     

With this, the image capture from the camera is successful. Next, we use MediaPipe for gesture recognition.

import mediapipe as mp

# Initialize the MediaPipe Hands module
mp_hands = mp.solutions.hands
hands = mp_hands.Hands()
mp_draw = mp.solutions.drawing_utils


# Process the image and detect hands
result = hands.process(rgb_frame)

if result.multi_hand_landmarks:
    for hand_landmarks in result.multi_hand_landmarks:
        mp_draw.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)

The output after recognizing a gesture is precise in identifying each joint on the hand and names each joint point. MediaPipe Hands provides 21 key points (landmarks) for the hand, collectively depicting the structure of the hand, including the wrist and the joints of each finger. Taking the thumb as an example, there are four joints, which from bottom to top are CMC, MCP, IP, TIP.

cmc: Carpometacarpal Joint
mcp:Metacarpophalangeal Joint
ip:Interphalangeal Joint
tip:tip

Having these landmarks alone is not enough; we need to set a method to recognize specific gestures. For example, if we want to recognize a thumbs-up gesture, we analyze that during a thumbs-up, the tip of the thumb is at the highest point above the entire palm. This makes it much easier. As long as we determine that the tip of the thumb is higher than the tips of all other fingers in the image, then the gesture is identified as a thumbs-up. (Other methods of analysis can also be used.)
Generally, we can obtain three attributes for each joint: X, Y, Z, representing the position of that joint in the image.

# Get the attributes of the thumb tip
thump_tip = hand_landmarks.landmark[mp.hands.HandLandmark.THUMB_TIP]

# Get the height of the thumb tip
thump_tip.y

# Determine thumbs up gesture
def is_thump_up(hand_landmarks):
    thumb_tip = hand_landmarks.landmark[mp_hands.HandLandmark.THUMB_TIP]
    index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP]
    # Determine which joint is higher.
    if thumb_tip.y < index_tip.y:
        return True
    
    return False

If you want other gestures, you can also set a special identification method based on the characteristics of the hand shape. At this point, gesture recognition is completed.

Robotic Arm Motion Control

Initially, my idea was that when the camera recognizes a gesture, it would send a control command to the robotic arm. Let's start with a simple action, setting the robotic arm to perform a nodding motion.
The pymycobot library offers many functions that are very convenient for controlling the robotic arm.

from pymycobot.mycobot import Mycobot
import time
# connect robot arm
mc = Mycobot(port,baud)

#Control the movement of the robotic arm using angles
mc.send_angles([angles_list],speed)

#Control the movement of the robotic arm using coordinates
mc.send_coords([coords_list],speed,mode)

# Nodding action
def ThumpUpAction(self):
        self.mc.send_angles([0.96, 86.22, -98.26, 10.54, 86.92, -2.37], 60)
        time.sleep(1.5)
        for count in range(3):
            self.mc.send_angles([0.79, 2.46, (-8.17), 4.3, 88.94, 0.26], 70)
            time.sleep(1)
            self.mc.send_angles([(-3.6), 30.32, (-45.79), (-46.84), 97.38, 0.35], 70)
            time.sleep(1)
        self.mc.send_angles([0.79, 2.46, (-8.17), 4.3, 88.94, 0.26], 70)
        time.sleep(1)
        self.mc.send_angles([0.96, 86.22, -98.26, 10.54, 86.92, -2.37], 60)

To enhance the readability and modifiability of the overall code, it's beneficial to create a robotic arm class for easy calling and modification.

class RobotArmController:

        def __init__(self,port):
        self.mc = MyCobot(port, 115200)
        self.init_pose = [0.96, 86.22, -98.26, 10.54, 86.92, -2.37]
        self.coords = [-40, -92.5, 392.7, -92.19, -1.91, -94.14]
        self.speed = 60
        self.mode = 0
        
        def ThumpUpAction(self):
            ...
            
        def OtherAction(self):
            ...

Program Logic Processing

During debugging, some issues arose. When recognizing gestures, continuous recognition meant that if a gesture was recognized 10 times in 1 second, 10 commands would be sent to the robotic arm. This was not what I initially envisioned.
Therefore, logical adjustments were needed. Here's how I addressed it:

# Set a 2-second timeframe to confirm the gesture. Only when a thumbs-up gesture is maintained for 2 seconds, the command to control the robotic arm is issued, using a control variable approach.

#init
#Variable to detect whether gesture exists
gesture_detected = False
#Variable that determines the timing after the gesture appears
gesture_start_time = None
# Set the variable 2s after the gesture appears
gesture_confirmation_time = 2

# When a specific gesture appears, gesture_start_time begins to count. During this period, continuous checks are made. If 2 seconds have passed, the gesture is confirmed, and then the corresponding robotic arm movement for that gesture is executed.

current_time = time.time()
if current_gesture:
    if not gesture_detected:
        gesture_detected = True
        gesture_start_time = current_time
    elif current_time - gesture_start_time > gesture_confirmation_time and not action_triggered:
        if current_gesture == "thumb_up":
            robotic arm action()

However, this is still not sufficient, as the hand maintaining the gesture for over 2 seconds would continue sending commands to the robotic arm. Here, we need to set a cooldown period to allow sufficient time for the robotic arm to complete its movement.

action_triggered = False
cooldown_start_time = None
cooldown_period = 2

# process gesture
                current_time = time.time()
                if current_gesture:
                    if not gesture_detected:
                        gesture_detected = True
                        gesture_start_time = current_time
                    elif current_time - gesture_start_time > gesture_confirmation_time and not action_triggered:
                        #Perform corresponding actions based on gestures
                        if current_gesture == "thumb_up":
                            print('good good')
                            mc.thum_up()
                        elif current_gesture == "palm_open":
                            print('forward')
                            mc.increment_x_and_send()
                        # You can add more gestures and corresponding action judgments
                        action_triggered = True
                        cooldown_start_time = current_time
                else:
                    gesture_detected = False
                    gesture_start_time = None
                    if action_triggered and current_time - cooldown_start_time > cooldown_period:
                        print('can continue')
                        action_triggered = False
                        cooldown_start_time = None

Video

https://youtu.be/9vOPKO_IG9M
Summary
This project demonstrates a method of using gesture recognition to control the myCobot 320, creating a new form of human-machine interaction. Although currently only a limited number of gestures and corresponding robotic arm movements have been implemented, it lays the groundwork for broader applications of robotic arms in the future. The innovative attempt to combine gestures with robotic arm control has not only improved my programming skills but also enhanced my problem-solving abilities, providing valuable experience for future related projects.

Introduction

This project demonstrates a simple pick-and-place operation using a robotic arm, employing GPT-4V and myCobot. The demonstration uses a method named SoM (Object Detection Method) to generate robotic movements through natural language. In simpler terms, the machine receives natural language instructions, locates the target, and then commands the robotic arm to perform the grabbing task. The highlight of this project is the combination of GPT-4V's image processing and the SoM object detection algorithm, enabling the robotic arm to move through natural language interaction with the machine.

Software

SoM

Set of Mark (SoM) is a method designed to enhance the visual understanding capabilities of large language models. After an image is processed with SoM, a series of markers are added to it. These markers can be recognized and processed by language models, aiding in more accurate identification and understanding of objects and content within the image. This enhancement allows language models to conduct more precise analysis and descriptions of elements in images, thereby improving their performance in visual tasks.

GPT-4V

The commonly discussed GPT is a large predictive model with which we can have conversational interactions. The new iteration, GPT-4V, is a multimodal large language model that not only processes textual information but now also handles image information. It is capable of image understanding, image generation, and image description. This large model combines the powerful natural language processing abilities of GPT-4 with current image analysis technologies, offering more efficient and accurate integrated capabilities in vision and language.
Below is an example provided by OpenAI:

User:
"These are frames from a video that I want to upload. Generate a compelling description that I can upload along with the video."

#GPT-4V Image description processing
GPT4:
"🐺 Survival of the Fittest: An Epic Tale in the Snow ❄️ - Witness the intense drama of nature as a pack of wolves face off against mighty bison in a harsh winter landscape. This raw footage captures the essence of the wild where every creature fights for survival. With each frame, experience the tension, the strategy, and the sheer force exerted in this life-or-death struggle. See nature's true colors in this gripping encounter on the snowy plains. 🦬"

Remember to respect wildlife and nature. This video may contain scenes that some viewers might find intense or distressing, but they depict natural animal behaviors important for ecological studies and understanding the reality of life in the wilderness.

Openai-whisper

https://openai.com/research/whisper
An automatic speech recognition (ASR) system developed by OpenAI. It utilizes advanced deep learning techniques to transcribe, translate, and understand human speech. Whisper is characterized by its ability to support multiple languages and its exceptional performance in recognizing voices with various accents and in noisy environments. Additionally, it can handle different audio qualities, making it suitable for various applications such as transcribing meeting records, automatically generating subtitles, and assisting in translations.

pymycobot

https://github.com/elephantrobotics/pymycobot/
pymycobot is a Python library for the mycobot robot series. MyCobot is a compact, multifunctional collaborative robotic arm suitable for education, research, and lightweight industrial applications. The PyMyCobot library provides a simple programming interface, enabling developers to control and program MyCobot robots for operations such as movement, grabbing, and sensing. This library supports multiple operating systems and development environments, facilitating its integration into various projects, especially in robotics and automation. By using Python, a widely-used programming language, pymycobot makes operating and experimenting with MyCobot robots more accessible and flexible.

Hardware

myCobot 280M5

The myCobot 280 M5 is a desktop-level compact six-axis collaborative robot produced by Elephant Robotics. Designed for compactness, it is suitable for education, research, and light industrial applications. The myCobot 280 M5 supports various programming and control methods, compatible with different operating systems and programming languages, including:
●Main and auxiliary control chips: ESP32
●Supports Bluetooth (2.4G/5G) and wireless (2.4G 3D Antenna)
●Multiple input and output ports
●Supports free movement, joint movement, Cartesian movement, trajectory recording, and wireless control
●Compatible operating systems: Windows, Linux, MAC
●Supported programming languages: Python, C++, C#, JavaScript
●Supported programming platforms and tools: RoboFlow, myblockly, Mind+, UiFlow, Arduino, mystudio
●Supported communication protocols: Serial port control protocol, TCP/IP, MODBUS
These features make the myCobot 280 M5 a versatile, user-friendly robot solution suitable for a variety of application scenarios.

myCobot Vertical Suction Pump V2.0

Operates on the principle of vacuum adhesion, providing 3.3V IO control, and can be extensively used in the development of various embedded devices.

Camera

Standard USB and LEGO interfaces. The USB interface can be used with various PC devices, and the LEGO interface can be conveniently fixed. It is applicable to machine vision, image recognition, and other applications.

mylangrobot Software Analysis
The specific workflow of the project described at the beginning is as follows:

1. Audio Input: Record audio instructions first.
2. Audio Processing: Use "openai-whisper" to process the audio and convert it into text.
3. Language Model Interaction: Use the GPT-4 model to process the converted text instructions and understand the user's commands.
4. Image Processing: Use GPT-4V and the enhanced image capability of SoM to process images and find the target mentioned in the instructions.
5. Robotic Arm Control: Control the robotic arm to grab the identified target.

Audio Processing
This function utilizes speech_recognition to capture audio data from the microphone, enabling the computer to recognize it.
Libraries used:

import io
import os
from enum import Enum
from typing import Protocol

import openai
import speech_recognition as sr
from pydub import AudioSegment
from pydub.playback import play

Define interfaces, capture user input, and provide output to the user.

class Interface(Protocol):
    def input(self, prefix: str = "") -> str:
        return prefix + self._input_impl()

    def _input_impl(self) -> str:
        ...

    def output(self, message: str) -> None:
        ...

Initialize the microphone for audio input and output.

class Audio(Interface):
    def __init__(self):
        self.r = sr.Recognizer()
        self.mic = sr.Microphone()
        # openai-whisper API key
        self.client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

Convert the input audio into text format for output.

 def _input_impl(self) -> str:
        print("Please tell me your command.")
        with self.mic as source:
            self.r.adjust_for_ambient_noise(source)
            audio = self.r.listen(source)

        try:
            return self.r.recognize_whisper(audio, language="japanese")

        except sr.UnknownValueError:
            print("could not understand audio")
        except sr.RequestError as e:
            print("Could not request results from Google Speech Recognition service; {0}".format(e))

The final return 'r' is the text format of the audio, which can be used for interaction with the GPT-4 model.

Image Processing and GPT-4 Language Interaction

When transmitting text to the GPT-4 model for interaction, images are sent along, so image processing and interaction are discussed together.
Libraries used for image processing:

import cv2
import numpy as np
import supervision as sv
import torch
from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

from .utils import download_sam_model_to_cache

Primarily uses the SamAutomaticMaskGenerator feature to mark and draw markers on detected targets.

#Convert image to RGB format
        image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
#Image processing, target detection and marker rendering
        sam_result = self.mask_generator.generate(image_rgb)
        detections = sv.Detections.from_sam(sam_result=sam_result)
        height, width, _ = image.shape
        image_area = height * width

        min_area_mask = (detections.area / image_area) > self.MIN_AREA_PERCENTAGE
        max_area_mask = (detections.area / image_area) < self.MAX_AREA_PERCENTAGE
        detections = detections[min_area_mask & max_area_mask]
        
        
        #Returns the result of the image and detected information
        labels = [str(i) for i in range(len(detections))]
        annotated_image = mask_annotator.annotate(scene=image_rgb.copy(), detections=detections)
        annotated_image = label_annotator.annotate(scene=annotated_image, detections=detections, labels=labels)
        return annotated_image, detections

This results in the following effect.
Note: The below function requires obtaining the GPT-4 API-Key for usage.
The resulting image is passed to the GPT-4 model, which requires some processing before use. Through GPT-4V, the image can be processed to return information about the image content and corresponding object information.

def prepare_inputs(message: str, image: np.ndarray) -> dict:
    # # Path to your image
    # image_path = "temp.jpg"
    # # Getting the base64 string
    base64_image = encode_image_from_cv2(image)

    payload = {
        "model": "gpt-4-vision-preview",
        "messages": [
            {"role": "system", "content": [metaprompt]},
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": message,
                    },
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}},
                ],
            },
        ],
        "max_tokens": 800,
    }

    return payload


def request_gpt4v(message: str, image: np.ndarray) -> str:
    payload = prepare_inputs(message, image)
    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
    res = response.json()["choices"][0]["message"]["content"]
    return res

Robotic Arm Control and Overall Integration
After image processing and GPT-4V model processing, the interpreted instructions generate target position information. This position information is passed to the robotic arm control system, which moves to the corresponding location to perform the grabbing action.
Key methods involved:
Move to the target object.

    def move_to_object(self, object_no: int, speed: Optional[float] = None) -> None:
        object_no = self._check_and_correct_object_no(object_no)
        print("[MyCobotController] Move to Object No. {}".format(object_no))
        detection = (
            np.array([-self._detections[object_no][0], -self._detections[object_no][1]]) + self.capture_coord.pos[:2]
        )
        print("[MyCobotController] Object pos:", detection[0], detection[1])
        self.move_to_xy(detection[0], detection[1], speed)

grab action
  def grab(self, speed: Optional[float] = None) -> None:
        print("[MyCobotController] Grab to Object")
        current_pos = self.current_coords().pos
        self.move_to_z(self.object_height + self.end_effector_height, speed)
        self._mycobot.set_basic_output(self._suction_pin, 0)
        time.sleep(2)
        self.move_to_z(current_pos[2], speed)

drop action
    def move_to_place(self, place_name: str, speed: Optional[float] = None) -> None:
        print("[MyCobotController] Move to Place {}".format(place_name))
        self._current_position = self.positions[place_name]
        self._mycobot.sync_send_angles(
            np.array(self._current_position) + self.calc_gravity_compensation(self._current_position),
            speed or self._default_speed,
            self._command_timeout,
        )
        print("Current coords: {}".format(self.current_coords()))

After each function is implemented, coordinate the entire process, streamline the workflow logic, and complete the task.
The specific code can be viewed in the operator.py file.

Example

Below is an example test to observe the project's outcome. The content involves a voice input saying "pick up the chocolate," and the robotic arm executes the task.
https://youtu.be/Eda1m7DnIhQ

Summary

This project demonstrates how to leverage advanced artificial intelligence and robotics technologies to accomplish complex automation tasks. By integrating voice recognition, natural language processing, image analysis, and precise robotic arm control, the project has successfully created a robotic system capable of understanding and executing spoken instructions. This not only enhances the naturalness and efficiency of robot-human interaction but also opens up new possibilities for robotic technology in various practical applications, such as automated manufacturing, logistics, assistive robots, and more.
Finally, thanks again to Shirokuma for sharing this case with us. If you have better examples, feel free to contact us!

Today, I am going to present a robotic arm model that I have independently designed and implemented. The core feature of this model is to achieve real-time gesture tracking - just a gentle drag with your hand, and the robotic arm can immediately follow your movements.

The reason why I wanted to create such a model is that in some dangerous environments, we can use robotic arms to replace manual work, thereby avoiding threats to human life.

You might ask, why not directly use remote keyboard control, joystick control, or APP control, but choose to manually drag? I believe that only manual operation can satisfy our need for precision to the greatest extent. Therefore, I decided to start trying to make this model and have initially completed the entire demo.

I hope that through this demo, I can demonstrate the infinite possibilities of robotic arms to everyone, and at the same time, I hope to inspire everyone's infinite longing for future technology.

Robotic Arm

The mechArm 270 is a 6 DOF robotic arm, with a compact structure design that can fit into a backpack for easy transportation. Importantly, it has many open control APIs, which allows you to quickly start controlling the robotic arm using Python. There are no complicated operations, and it even supports graphical programming, which allows people who are not very familiar with code to quickly get started with controlling the robotic arm.

The mechArm is a desktop robotic arm with a structure that mimics industrial designs. Its maximum working radius is 270mm, it can carry a load of 250g, and its repeat positioning accuracy is controlled within ±0.5mm.

Project

After introducing the basic equipment, let's start with the record of how I created this demo.

Environment:
Operating system: Windows 11

Programming language: Python 3.9+

Python libraries: pymycobot, time

pymycobot is an open-source library for Elephant Robotics, specifically designed to control the robotic arm of Elephant Robotics. Here is an example of a simple control code.

Code：

#Main methods used

#Create objects to communicate with the robotic arm.
MyCobot(serial,baud)
# angles control robot,
send_angles([list_angles],speed)
# coords control robot
send_coords([list_coords],speed,mode)

Example:

import time
from pymycobot.mycobot import MyCobot

# create a object
mc = MyCobot("com7",115200)

# angles control
mc.send_angles([0,0,0,0,0,0],100)
time.sleep(1)
mc.send_angles([90,90,90,90,90,90],100)
time.sleep(1)

I briefly introduced how to use python to control mechArm. Isn’t it easy?

Problem Analysis

Before starting the project, it's important to set up a framework and understand the specific problems we need to solve. For this, I made a flowchart of the project. In the following, I will refer to the manually controlled robotic arm as R1, and the following motion robotic arm as R2.

Control Robotic Arm: As mentioned above, the robotic arm can be controlled using the methods provided by the pymycobot library.

Motion Control Methods: The R1 robotic arm can be dragged by hand, returning the current angle information of the robotic arm at all times. The R2 robotic arm controls based on the angle information received from R1.

Communication Between Robotic Arms: This step is quite important in the entire project. Once established, the robotic arm can easily implement information transmission.

Next, I will mainly explain the Motion Control Methods and Communication Between Robotic Arms.

Motion Control methods

1. Get real-time angle information

pymycobot provides the "get_angles()" method to obtain the angle information of the current robot arm.

# Can obtain the current angle information of the robotic arm in real time
get_angles()

# example
print("real-time angles:",mc.get_anlges())

result: real-time angles:[0,0,0,0,0,0]

# Continuously obtain the current angle
while True:
    angels = mc.get_angles()
    print(angles) 
    time.sleep(0.1) #Go to the next step every 0.1s

2. Set the Refresh Mode for the Robotic Arm

The refresh mode of the robotic arm mainly falls into two categories: interpolation mode and non-interpolation mode. These refer to the ways in which the end effector of the robotic arm is controlled during motion trajectory planning. If no mode is set, the robotic arm may not be able to perform the expected motion correctly, which may lead to the following consequences:

1. Unsmooth motion

2. Inaccurate motion

3. Discontinuous motion

Interpolation Mode: The interpolation mode can realize smooth and continuous trajectory planning, ensuring that the position and posture of the end effector of the robotic arm transition smoothly during the motion process.

Non-Interpolation Mode: The non-interpolation mode means that the robotic arm only focuses on specific target points during the motion process, without performing interpolation calculations. Under the non-interpolation mode, the position and posture of the robotic arm will jump directly between key points, without undergoing a smooth transition.

When multiple robotic arms use the interpolation mode for motion at the same time, there may be situations of waiting or queuing. Therefore, we choose to use the non-interpolation mode.

#Set refresh mode
set_fresh_mode(1/0) 
1：no interpolation
0：interpolation

mc.set_fresh_mode(1)

Our code that integrates the previous ones is as follows.

Code:

import time
from pymycobot.mycobot import MyCobot

mc = MyCobot("COM7", 115200)    #release arm
mb = MyCobot("COM11", 115200)   #move arm

mb.set_fresh_mode(1)    #no interpolation
time.sleep(1)
mc.release_all_servos() #release robot
time.sleep(1)
speed = 100
while True:
    angles = mc.get_angles()    #get release arm angles
    mb.send_angles(angles, speed)   #send angles to move arm
    time.sleep(0.1)
## Communication between robotic arms：
Our solution is to connect two robotic arms to the same PC and connect them through a serial port.

# build connection
from pymycobot.mycobot import MyCobot

mc = MyCobot("COM7", 115200)   
mb = MyCobot("COM11", 115200)

By using the most basic USB data cables for connection, we have two serial port numbers for the robotic arms on our computer, and we can send instructions to them separately.

https://www.youtube.com/watch?v=NByjgoqc2O4

Summary

From the content, it can be seen that although we can achieve about 70-80% synchronization, there are other factors that can cause significant delays. The reasons for the delays could be various, such as the speed of data processing and transmission, the reaction speed of the robotic arm, software optimization, hardware performance, etc. All of these are potential factors that can cause delays.

In addition, there is a significant limitation in that their communication is connected through serial ports. If the distance is a bit further, this method cannot be used, and its practicality is not strong. In the future, I will try to use wireless connections such as Bluetooth and WiFi to control the robotic arm.

This article will document my experience using myCobot 280 M5stack in ROS. Why use ROS? Because when it comes to robotics, it's hard to avoid the ROS operating system.
Today is our first time using this system. Today, I will start with an introduction to ROS, the configuration of the environment, and the use of mycobot280 in ROS.

ROS

ROS (Robot Operating System) is a flexible framework for writing robot software. It is a semi-open source project that provides researchers and developers with a set of tools and libraries to help them create various robot applications. ROS is mainly used for research and development of various robot systems, such as autonomous vehicles, service robots, industrial robots, drones, etc. It makes the development of robots more efficient and enhances the functionality and flexibility of robots. The two most notable modules in the ROS operating system are navigation and robotic arm control (moveit). Moveit allows users to quickly build robotic arm models and implement robotic arm control, such as modeling, kinematic solution, motion planning, obstacle avoidance, and more. Given such a powerful system, I want to use it to simulate some functions of the robotic arm, such as obstacle avoidance, path planning, etc.

Environment configuration

ROS primarily supports operating systems like Ubuntu (Linux) and Mac OS (with partial support). The most common usage is on the Ubuntu system. My computer is installed with Ubuntu 18.04 version. The corresponding ROS version for Ubuntu 18 is Melodic. Different versions of Ubuntu correspond to different ROS versions.

ubuntu16.04-kinetic
ubuntu18.04-melodic
ubuntu20.04-noetic

Since my computer has already had the system and ROS environment installed, I won’t record each step here. The main focus will be on the operations within ROS. For the installation of the Ubuntu system and the setup of the ROS environment, you can search on Google. The ROS official website also has detailed installation tutorials.
The scenario I’m using today involves controlling a robotic arm with MoveIt, so the demand for computer hardware is not high. Please note that if you are using features like SLAM radar navigation, you might need a higher-spec computer to handle the computational power. The standard to check whether ROS has been successfully installed is to open the terminal and type “roscore”. If the content that appears is the same as the image below, it means the installation was successful.

Let’s use the ROS!

Project

To create a robotic arm in ROS and use some path planning features, you need to perform the following steps:

• list itemCreate a URDF file, which is the model of the robotic arm.

• list itemLoad the URDF file into the ROS server.

• list itemUse RViz for visualization.

• list itemCreate a MoveIt configuration package.

• list itemUse MoveIt for motion planning.

Creating a URDF file

Creating a URDF file First, we need to create a URDF (Unified Robot Description Format) file for the robotic arm. This file is crucial as it describes the properties of the robotic arm, such as joints, links, and sensors, in the form of a 3D model. Since the product is the myCobot 280 M5Stack robotic arm from Elephant Robotics, Elephant Robotics has provided a URDF file for the myCobot 280 on Github, which describes some physical properties of the robotic arm.
How to generate a URDF file:
You need to use CAD software to create the model, ensuring that the 3D model you create matches the actual size of the robotic arm. Because the exported model needs to be in a format that ROS can accept, it is usually a DAE (Collada) file.

Load URDF to ROS server

Create a ROS package
To create a package to store the URDF file, execute the following code in the terminal:

catkin_create_pkg mycobot_description rospy std_msgs sensor_msgs

Here, ‘rospy’, ‘std_msgs’, and ‘sensor_msgs’ are dependency packages in ROS. Add the generated URDF file to the mycobot_description/urdf directory.

Create a launch file
In the mycobot_description/launch directory, create a file named load_urdf.launch and add the following content to it.

<launch>  <param name="robot_description" command="$(find xacro)/xacro.py $(find mycobot_description)/urdf/mycobot.urdf.xacro" /></launch>

Lanuch ROS file
Open the terminal and enter the following command:

roslaunch my_robot_description load_urdf.launch

After running, we can see the model we created in RViz.

Configure MoveIt

Start MoveIt Setup Assistant:

In a terminal, run the following command to start MoveIt Setup Assistant:

roslaunch moveit_setup_assistant setup_assistant.launch

Load URDF in MoveIt In the main interface of MoveIt Setup Assistant, click the “Create New MoveIt Configuration Package” button. Then, in the pop-up dialogue box, select my URDF file (in the ROS parameter robot_description). Click the “Load Files” button to load your URDF file. If the image as shown below appears, it means the file has been loaded successfully.

Configure Robot

Configure your robot according to your needs. Here are a few configurations briefly introduced:

• Self-Collisions: MoveIt will automatically calculate the self-collision matrix of your robot. You can
  click the “Regenerate Default Collision Matrix” button to generate a default self-collision matrix.
• Virtual Joints: You can define virtual joints for your robot. Virtual joints are often used to connect your robot to the world.
• Planning Groups: You can define the planning groups of your robot. Planning groups are a set of joints and links that need to move together. For example, you can create a planning group that includes all arm joints.
• Robot Poses: You can define preset poses for your robot. These preset poses can be reused in planning.

After the configuration is completed, a .launch file will be generated, and we can use the robotic arm in moveit by running it.

Let’s see how the moveit function performs path planning. By dragging the coordinates of the end of the robotic arm, we can make changes to the movement of the robotic arm, and also randomly generate paths.

Object obstacle avoidance

We add an obstacle in MoveIt for the robotic arm to avoid. We add a cube, so the robotic arm will navigate around it when it follows its trajectory.

In addition to this, MoveIt has many features. For instance, motion planning, which can perform path planning for multi-joint robots, avoid collisions, and also use various motion planning algorithms like RRT, EST, LBKPIECE, etc. It also includes collision detection, which can carry out complex 3D collision detection, including self-collision and environmental collision (obstacles). Furthermore, it offers simulation capabilities, and can be integrated with the Gazebo robot simulator for realistic physical simulation, among other features.

Summary

MoveIt provides a comprehensive set of tools for research into the motion planning and control of robotic arms, eliminating the need to build complex motion planning systems from scratch. This allows developers to focus more of their energy on implementing specific application functions, rather than on solving underlying motion planning and control problems. Overall, ROS and MoveIt provide strong support for the development of robotic technology. They not only reduce development difficulties and improve development efficiency, but also promote innovation and application in robotic technology. Whether you are a researcher in robotic technology or a developer in practical applications, ROS and MoveIt will be indispensable tools for you.

In the future, I will continue to explore more robotic arm projects and share my findings and achievements from time to time. If you find any errors in this article, feel free to communicate with me. If you like this article, your likes and comments will be the greatest support for me!

When we saw a video on YouTube of someone using a robotic arm to achieve object tracking, it deeply inspired us. We became very interested in this project and decided to independently develop a similar program.

myCobot 280 M5Stack

The robotic arm used for the operation is the myCobot280 M5Stack. This is a small 6-axis robotic arm produced by Elephant Robotics, with M5Stack-Basic as the microprocessor, ESP32 as the auxiliary control, and a UR collaborative structure. The myCobot280 M5Stack-Basic has a body weight of 800g, a payload of 250g, a working radius of 280mm, and a compact and portable design. Despite its small size, it is powerful and easy to operate, capable of collaborating with humans and working safely.

Process

The following image is a flowchart of the project development process.

Capture the target

Before beginning development, we conducted some research and experiments. We used a camera to capture images of objects and utilized the OpenCV library for recognition. We attempted various methods, but object recognition required machine learning for the target we wanted to identify, which would increase the project development time. Ultimately, we decided to use aruco codes for identification, which allowed us to quickly capture the aruco codes and proceed to the next stage of development.

Code：

def show_video_v2(self):
        # self.robot.init_robot()
        xyz = np.array([0,0,0])
        LIST = []
        num_count = 0
        list_len = 5
        # cmax = [180, 80, 240]
        # cmin = [130, -80, 200]
        cmax = [150, -150, 300]
        cmin = [-150, -250, 200]

        while cv2.waitKey(1) < 0:
            success, img = self.cap.read()
            if not success:
                print("It seems that the image cannot be acquired correctly.")
                break
            # transfrom the img to model of gray
            gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
            # Detect ArUco marker.
            corners, ids, rejectImaPoint = cv2.aruco.detectMarkers(
                gray, self.aruco_dict, parameters=self.aruco_params
            )

            if len(corners) > 0:
                if ids is not None:
                    # get informations of aruco
                    ret = cv2.aruco.estimatePoseSingleMarkers(
                        # '''https://stackoverflow.com/questions/53303730/what-is-the-value-for-markerlength-in-aruco-estimateposesinglemarkers'''
                        corners, 0.025, self.camera_matrix, self.dist_coeffs
                    )
                    # rvec:rotation offset,tvec:translation deviator
                    (rvec, tvec) = (ret[0], ret[1])
                    
                    (rvec - tvec).any()
                    xyz = tvec[0, 0, :] * 1000
                    rpy = rvec[0,0,:]

                    camera = np.array([xyz[0], xyz[1], xyz[2]])

                    if num_count > list_len:
                        target = model_track(camera)
                        print("target", target)

                        for i in range(3):
                            if target[i] > cmax[i]:
                                target[i] = cmax[i]
                            if target[i] < cmin[i]:
                                target[i] = cmin[i]

                        pose = np.array([-103, 8.9, -164])
                        coord = np.concatenate((target.copy(), pose), axis=0)

                        # q1 = math.atan(xyz[0] / xyz[2])*180/np.pi
                        mc.send_coords(coord,50,0)
                        
                        
                        # print('target', coord)
                        num_count = 1
                    else:
                        num_count = num_count + 1
                    

                    for i in range(rvec.shape[0]):
                        # draw the aruco on img
                        cv2.aruco.drawDetectedMarkers(img, corners)
            cv2.imshow("show_video", img)

Hand-eye calibration

Hand-eye calibration refers to the process of determining the position and orientation of the robot end effector (such as a mechanical arm) relative to the robot base coordinate system in the field of robotics. This process involves pairing the robot end effector with a camera and then determining its position and orientation in the robot base coordinate system by capturing its position and orientation in the camera's field of view.
Hand-eye calibration typically involves a series of movements between the robot end effector and the camera to collect enough data to calculate the transformation matrix between them. This transformation matrix describes the position and orientation of the robot end effector relative to the camera, which can be used to control the robot's motion and accurately perform the required tasks.
In "eye-to-hand" hand-eye calibration, the camera is considered a stationary observer ("eye"), while the robot end effector is considered a moving object in the camera's field of view ("hand"). As the robot end effector moves around the camera, a series of images are collected that contain information about the end effector's position and orientation at different locations and orientations. By analyzing these images, the position and orientation of the robot end effector relative to the camera can be calculated, completing the hand-eye calibration.
The following is the code for processing the coordinate transformation data.

#The function is used to calculate the similarity between cameras.
def calculate_similarity(camera):
    n = camera.shape[0]
    total_similarity = 0
    for i in range(n):
        for j in range(i+1, n):
            vector_a = camera[i]
            vector_b = camera[j]
            dot_product = np.dot(vector_a, vector_b)
            norm_a = np.linalg.norm(vector_a)
            norm_b = np.linalg.norm(vector_b)
            similarity = dot_product / (norm_a * norm_b)
            total_similarity += similarity
    return total_similarity/n
# The function is used to calculate the rate of change in similarity.
def similarity_change_rate(new_similarity):
    global prev_similarity
    if prev_similarity is None:
        prev_similarity = new_similarity
        return 0
    else:
        change_rate = (new_similarity - prev_similarity) / prev_similarity
        prev_similarity = new_similarity
        return change_rate

#The function is used to convert a rotation matrix to Euler angles.
def CvtRotationMatrixToEulerAngle(pdtRotationMatrix):
    pdtEulerAngle = np.zeros(3)

    pdtEulerAngle[2] = np.arctan2(pdtRotationMatrix[1, 0], pdtRotationMatrix[0, 0])

    fCosRoll = np.cos(pdtEulerAngle[2])
    fSinRoll = np.sin(pdtEulerAngle[2])

    pdtEulerAngle[1] = np.arctan2(-pdtRotationMatrix[2, 0], (fCosRoll * pdtRotationMatrix[0, 0]) + (fSinRoll * pdtRotationMatrix[1, 0]))
    pdtEulerAngle[0] = np.arctan2((fSinRoll * pdtRotationMatrix[0, 2]) - (fCosRoll * pdtRotationMatrix[1, 2]), (-fSinRoll * pdtRotationMatrix[0, 1]) + (fCosRoll * pdtRotationMatrix[1, 1]))

    return pdtEulerAngle
# The function is used to convert Euler angles to a rotation matrix.
def CvtEulerAngleToRotationMatrix(ptrEulerAngle):
    ptrSinAngle = np.sin(ptrEulerAngle)
    ptrCosAngle = np.cos(ptrEulerAngle)

    ptrRotationMatrix = np.zeros((3, 3))
    ptrRotationMatrix[0, 0] = ptrCosAngle[2] * ptrCosAngle[1]
    ptrRotationMatrix[0, 1] = ptrCosAngle[2] * ptrSinAngle[1] * ptrSinAngle[0] - ptrSinAngle[2] * ptrCosAngle[0]
    ptrRotationMatrix[0, 2] = ptrCosAngle[2] * ptrSinAngle[1] * ptrCosAngle[0] + ptrSinAngle[2] * ptrSinAngle[0]
    ptrRotationMatrix[1, 0] = ptrSinAngle[2] * ptrCosAngle[1]
    ptrRotationMatrix[1, 1] = ptrSinAngle[2] * ptrSinAngle[1] * ptrSinAngle[0] + ptrCosAngle[2] * ptrCosAngle[0]
    ptrRotationMatrix[1, 2] = ptrSinAngle[2] * ptrSinAngle[1] * ptrCosAngle[0] - ptrCosAngle[2] * ptrSinAngle[0]
    ptrRotationMatrix[2, 0] = -ptrSinAngle[1]
    ptrRotationMatrix[2, 1] = ptrCosAngle[1] * ptrSinAngle[0]
    ptrRotationMatrix[2, 2] = ptrCosAngle[1] * ptrCosAngle[0]

    return ptrRotationMatrix

Robotic Arm Control

After this step, object detection and control of the mechanical arm follow, which involves converting the coordinates of the recognized object into motion commands for the mechanical arm. The pymycobot library is used to control the mechanical arm.

#The function is used for visual tracking and calculating the target position.
def Visual_tracking280(coord, camera):
    pose_camera = camera[:3]
    angle_camear = camera[3:]
    r = CvtEulerAngleToRotationMatrix(angle_camear)
    # r = np.array([[1, 0, 0],
    #                  [0, 1, 0],
    #                  [0, 0, 1]])
    euler = np.radians(coord[3:])
    R = CvtEulerAngleToRotationMatrix(euler)
    offset = np.array([0, 0, -250])
    Roff = np.array([[1, 0, 0],
                     [0, -1, 0],
                     [0, 0, -1]])
    # Roff = np.array([[1, 0, 0],
    #                  [0, 1, 0],
    #                  [0, 0, 1]])
    vector = pose_camera + offset
    # print("R", R)
    # print("r", r)

    move_pos = np.dot(np.dot(R, r), Roff).dot(vector)
    pos = coord[:3] + move_pos
    # angle = np.array(CvtRotationMatrixToEulerAngle(np.dot(np.dot(R, r), Roff))) * 180/np.pi
    angle =  coord[3:]
    target = np.concatenate((pos, angle))
    return target
    
    #Calculate the target position based on the camera coordinates.
def model_track(camera):
    model_pos = np.array([-camera[0], -camera[2], -camera[1]])
    camera_pos = np.array([-37.5, 416.6, 322.9])
    target_pos = model_pos + camera_pos
    # print("model_pos", model_pos)
    # print("target_pos", target_pos)
    return target_pos

Finally, let's summarize the logical relationship of the project.

Let's take a look at how it performs.

It may be noticed that sometimes myCobot does not move. This is because its body was blocking the camera, preventing it from capturing the target object. When moving objects, it is important to ensure that the mechanical arm body does not block the camera.

This code is applicable to the entire myCobot280 series, including Pi, Jetson Nano, and other versions. The parameters may need to be adjusted based on the specific version being used.

People often compare the Jetson Nano with the Raspberry Pi. I have tried this program on two different robotic arms, and it is evident that the Jetson Nano version is much more responsive than the Raspberry Pi, owing to its superior computational power.There is a noticeable delay of approximately one second between them as observed by the naked eye.

Summary

During the debugging process, we found that the tracking effect was not very smooth and responsive. We adjusted the smoothness by controlling the detection cycle, but it was necessary to slowly move the tracked object to achieve better results. There are still some shortcomings, as the body of the mechanical arm may block the camera's field of view when the camera is fixed, making it impossible to proceed with the next tracking step. One solution we thought of is to move the camera to a position where it is not blocked (which would require recalculating the coordinates). If you have any better ideas, please feel free to communicate with us! Thank you for your patience.

PC: 0x04d25826
EXCVADDR: 0x04d25824

Decoding stack results
0x40090d71: prvIdleTask at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c line 3382
0x4008f5b6: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

So my workflow for now is:

#define HEADER_PIXEL(data,pixel) {
pixel[0] = (((data[0] - 33) << 2) | ((data[1] - 33) >> 4));
pixel[1] = ((((data[1] - 33) & 0xF) << 4) | ((data[2] - 33) >> 2));
pixel[2] = ((((data[2] - 33) & 0x3) << 6) | ((data[3] - 33)));
data += 4;
}

Write your own function rgb888to565 to compress the pixels into a uint16_t.

Draw a bitmap of the image as fast as you can:

#include <M5StickCPlus.h>
#include "1.h"
int pixel[3];
// pointer fu to preserve the start of .h data
char *datastart;
uint16_t *bitmap;

void setup() {
M5.begin();
M5.Lcd.setRotation(3);
bitmap = (uint16_t *)malloc(height * width * 2);
}

void loop() {
M5.Lcd.fillScreen(GREEN);
datastart = data;
for (int16_t y=0; y < height; y++) {
for (int16_t x=0; x < width; x++) {
HEADER_PIXEL(data, pixel);
bitmap[60*y + x] = rgb888to565(pixel[0], pixel[1], pixel[2]);
}
}
M5.Lcd.drawBitmap(0,0,width,height,bitmap);
data = datastart;
}

Or you can use the Sprite library, which works well.

However when I type in the letters, I still have to wait "too long" until the display is updated.

What can you recommend being the fastest way to display the new letters? Is there a possibility to partially update the display? I tried "M5.EPD.UpdateArea(0,60,540,400,UPDATE_MODE_GL16);" but nothing changes.

Thanks.

DiWa

CODE (needful part of it):

void Tastatur(){
canvas.fillCanvas(0);
Header();
canvas.setTextArea(10, 60, 520, 460);
canvas.setTextSize(48);
canvas.print(Eingabe);

//Raster
canvas.setTextSize(32);
canvas.drawRoundRect( 10, 430, 168, 80, 10, 15);
canvas.drawRoundRect(186, 430, 168, 80, 10, 15);
canvas.drawRoundRect(362, 430, 168, 80, 10, 15);
canvas.fillRoundRect( 11, 431, 166, 78, 9, 2);
canvas.fillRoundRect(187, 431, 166, 78, 9, 2);
canvas.fillRoundRect(363, 431, 166, 78, 9, 2);
canvas.drawString("wiki", 65, 455);
canvas.drawString("wo", 250, 455);
canvas.drawString("123", 415, 455);

//canvas.createRender(48, 256);
canvas.setTextSize(48);
canvas.drawRoundRect( 10, 518, 80, 80, 10, 15); canvas.drawString("a", 35, 538);
canvas.drawRoundRect( 98, 518, 80, 80, 10, 15); canvas.drawString("b", 123, 538);
canvas.drawRoundRect(186, 518, 80, 80, 10, 15); canvas.drawString("c", 211, 538);
canvas.drawRoundRect(274, 518, 80, 80, 10, 15); canvas.drawString("d", 299, 538);
canvas.drawRoundRect(362, 518, 80, 80, 10, 15); canvas.drawString("e", 387, 538);
canvas.drawRoundRect(450, 518, 80, 80, 10, 15); canvas.drawString("f", 475, 538);
canvas.drawRoundRect( 10, 606, 80, 80, 10, 15); canvas.drawString("g", 35, 626);
canvas.drawRoundRect( 98, 606, 80, 80, 10, 15); canvas.drawString("h", 123, 626);
canvas.drawRoundRect(186, 606, 80, 80, 10, 15); canvas.drawString("i", 216, 626);
canvas.drawRoundRect(274, 606, 80, 80, 10, 15); canvas.drawString("j", 305, 626);
canvas.drawRoundRect(362, 606, 80, 80, 10, 15); canvas.drawString("k", 387, 626);
canvas.drawRoundRect(450, 606, 80, 80, 10, 15); canvas.drawString("l", 475, 626);
canvas.drawRoundRect( 10, 694, 80, 80, 10, 15); canvas.drawString("m", 33, 714);
canvas.drawRoundRect( 98, 694, 80, 80, 10, 15); canvas.drawString("n", 123, 714);
canvas.drawRoundRect(186, 694, 80, 80, 10, 15); canvas.drawString("o", 211, 714);
canvas.drawRoundRect(274, 694, 80, 80, 10, 15); canvas.drawString("p", 299, 714);
canvas.drawRoundRect(362, 694, 80, 80, 10, 15); canvas.drawString("q", 387, 714);
canvas.drawRoundRect(450, 694, 80, 80, 10, 15); canvas.drawString("r", 475, 714);
canvas.drawRoundRect( 10, 782, 80, 80, 10, 15); canvas.drawString("s", 35, 802);
canvas.drawRoundRect( 98, 782, 80, 80, 10, 15); canvas.drawString("t", 123, 802);
canvas.drawRoundRect(186, 782, 80, 80, 10, 15); canvas.drawString("u", 211, 802);
canvas.drawRoundRect(274, 782, 80, 80, 10, 15); canvas.drawString("v", 299, 802);
canvas.drawRoundRect(362, 782, 80, 80, 10, 15); canvas.drawString("w", 387, 802);
canvas.drawRoundRect(450, 782, 80, 80, 10, 15); canvas.drawString("x", 475, 802);
canvas.drawRoundRect( 10, 870, 80, 80, 10, 15); canvas.drawString("y", 35, 890);
canvas.drawRoundRect( 98, 870, 80, 80, 10, 15); canvas.drawString("z", 123, 890);
canvas.drawRoundRect(186, 870, 80, 80, 10, 15); canvas.drawString("?", 211, 890);

canvas.drawRoundRect(275, 870, 80, 80, 10, 15); canvas.fillRoundRect(276, 871, 78, 78, 9, 2); canvas.drawString("_", 305, 890);
canvas.drawRoundRect(362, 870, 80, 80, 10, 15); canvas.fillRoundRect(363, 871, 78, 78, 9, 2); canvas.drawString("<-", 375, 890);
canvas.drawRoundRect(450, 870, 80, 80, 10, 15); canvas.fillRoundRect(451, 871, 78, 78, 9, 2); canvas.drawString("OK", 455, 890);

canvas.pushCanvas(0,0,UPDATE_MODE_GC16);

int touch = 0;
int Auswahl = 0;
int Leer_Stelle = 0;
int Zeichen = 0;

while(Auswahl == 0){
if(M5.TP.avaliable()){
if(!M5.TP.isFingerUp()){
M5.TP.update();
bool is_update = false;
for(int i=0;i<2; i++){
tp_finger_t FingerItem = M5.TP.readFinger(i);
if((point[i][0]!=FingerItem.x)||(point[i][1]!=FingerItem.y)){
is_update = true;
point[i][0] = FingerItem.x;
point[i][1] = FingerItem.y;
if (FingerItem.x >= 430 && FingerItem.x < 510 && FingerItem.y >= 360 && FingerItem.y <= 530) {
Serial.println("wiki");
canvas.setTextSize(32);
Eingabe = "Schreib einen wiki\nArtikel über\n";
canvas.pushCanvas(0,0,UPDATE_MODE_DU4);
}
if (FingerItem.x >= 430 && FingerItem.x < 510 && FingerItem.y >= 274 && FingerItem.y <= 352) {
Serial.println("wo");
canvas.setTextSize(32);
Eingabe = "wo finde ich ";
canvas.pushCanvas(0,0,UPDATE_MODE_GC16);
}
if (FingerItem.x >= 430 && FingerItem.x < 510 && FingerItem.y >= 98 && FingerItem.y <= 178) {
Serial.println("123");
Sonderzeichen();
}
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 450 && FingerItem.y <= 530) Eingabe = Eingabe + "a"; //Serial.print("a");
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 362 && FingerItem.y <= 442) Eingabe = Eingabe + "b"; //Serial.print("b");
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 274 && FingerItem.y <= 352) Eingabe = Eingabe + "c"; //Serial.print("c");
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 186 && FingerItem.y <= 266) Eingabe = Eingabe + "d"; //Serial.print("d");
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 98 && FingerItem.y <= 178) Eingabe = Eingabe + "e"; //Serial.print("e");
if (FingerItem.x >= 518 && FingerItem.x < 598 && FingerItem.y >= 10 && FingerItem.y <= 90) Eingabe = Eingabe + "f"; //Serial.print("f");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 450 && FingerItem.y <= 530) Eingabe = Eingabe + "g"; //Serial.print("g");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 362 && FingerItem.y <= 442) Eingabe = Eingabe + "h"; //Serial.print("h");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 274 && FingerItem.y <= 352) Eingabe = Eingabe + "i"; //Serial.print("i");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 186 && FingerItem.y <= 266) Eingabe = Eingabe + "j"; //Serial.print("j");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 98 && FingerItem.y <= 178) Eingabe = Eingabe + "k"; //Serial.print("k");
if (FingerItem.x >= 606 && FingerItem.x < 686 && FingerItem.y >= 10 && FingerItem.y <= 90) Eingabe = Eingabe + "l"; //Serial.print("l");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 450 && FingerItem.y <= 530) Eingabe = Eingabe + "m"; //Serial.print("m");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 362 && FingerItem.y <= 442) Eingabe = Eingabe + "n"; //Serial.print("n");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 274 && FingerItem.y <= 352) Eingabe = Eingabe + "o"; //Serial.print("o");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 186 && FingerItem.y <= 266) Eingabe = Eingabe + "p"; //Serial.print("p");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 98 && FingerItem.y <= 178) Eingabe = Eingabe + "q"; //Serial.print("q");
if (FingerItem.x >= 694 && FingerItem.x < 774 && FingerItem.y >= 10 && FingerItem.y <= 90) Eingabe = Eingabe + "r"; //Serial.print("r");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 450 && FingerItem.y <= 530) Eingabe = Eingabe + "s"; //Serial.print("s");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 362 && FingerItem.y <= 442) Eingabe = Eingabe + "t"; //Serial.print("t");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 274 && FingerItem.y <= 352) Eingabe = Eingabe + "u"; //Serial.print("u");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 186 && FingerItem.y <= 266) Eingabe = Eingabe + "v"; //Serial.print("v");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 98 && FingerItem.y <= 178) Eingabe = Eingabe + "w"; //Serial.print("w");
if (FingerItem.x >= 782 && FingerItem.x < 862 && FingerItem.y >= 10 && FingerItem.y <= 90) Eingabe = Eingabe + "x"; //Serial.print("x");
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 450 && FingerItem.y <= 530) Eingabe = Eingabe + "y"; //Serial.print("y");
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 362 && FingerItem.y <= 442) Eingabe = Eingabe + "z"; //Serial.print("z");
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 274 && FingerItem.y <= 352) Eingabe = Eingabe + "?"; //Serial.print("z");
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 186 && FingerItem.y <= 266) { //Serial.print("Leer");
Eingabe = Eingabe + " ";
Leer_Stelle = Eingabe.length() - 1;
}
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 98 && FingerItem.y <= 178) { //Serial.print("<-");
Eingabe_len = Eingabe.length();
if (Eingabe_len >= 1) Eingabe = Eingabe.substring(0, Eingabe_len - 1);
canvas.fillRect(0, 52, 540, 360, 0);
}
if (FingerItem.x >= 870 && FingerItem.x < 950 && FingerItem.y >= 10 && FingerItem.y <= 90) {
Auswahl = 1;
Serial.println("OK"); Serial.println();
}
int Zeilenlaenge = 20;
canvas.setTextSize(48);
canvas.setTextArea(10, 60, 520, 460);
Zeichen = Zeichen + 1;
if (Zeichen > Zeilenlaenge){
Eingabe.setCharAt(Leer_Stelle, '\n');
canvas.fillRect(0, 60, 540, 360, 0);
Zeichen = Eingabe.length() - Leer_Stelle;
}
canvas.print(Eingabe);
canvas.pushCanvas(0,0,UPDATE_MODE_DU4);
}
}
}
}
}
Eingabe.replace('\n', ' ');
Serial.println(Eingabe);
monitorWiFi();
canvas.setTextArea(10, 60, 520, 460);
canvas.drawRect(100, 300, 330, 80, 15); canvas.fillRect(101, 301, 328, 78, 2);
canvas.drawString("processing ...", 110, 320);
canvas.pushCanvas(0,0,UPDATE_MODE_GL16);
Serial.println(openAI_text(Eingabe));
}

My is not working too.

Pinout is the same as stated on https://docs.m5stack.com/en/unit/unit_cam

But in Homeassistant ESPHome generates these errors:

[12:03:45][C][ledc.output:180]: LEDC Output:
[12:03:45][C][ledc.output:181]: Pin GPIO2
[12:03:45][C][ledc.output:182]: LEDC Channel: 2
[12:03:45][C][ledc.output:183]: PWM Frequency: 1000.0 Hz
[12:03:45][C][ledc.output:184]: Phase angle: 0.0°
[12:03:45][C][ledc.output:185]: Bit depth: 16
[12:03:45][C][esp32_camera:048]: ESP32 Camera:
[12:03:45][C][esp32_camera:049]: Name: Timer-CAM1
[12:03:45][C][esp32_camera:050]: Internal: YES
[12:03:45][C][esp32_camera:052]: Data Pins: D0:32 D1:35 D2:34 D3:5 D4:39 D5:18 D6:36 D7:19
[12:03:45][C][esp32_camera:053]: VSYNC Pin: 22
[12:03:45][C][esp32_camera:054]: HREF Pin: 26
[12:03:45][C][esp32_camera:055]: Pixel Clock Pin: 21
[12:03:45][C][esp32_camera:056]: External Clock: Pin:27 Frequency:20000000
[12:03:45][C][esp32_camera:060]: I2C Pins: SDA:25 SCL:23
[12:03:45][C][esp32_camera:062]: Reset Pin: 15
[12:03:45][C][esp32_camera:080]: Resolution: 640x480 (VGA)
[12:03:45][E][esp32_camera:123]: Setup Failed: ESP_ERR_NOT_FOUND
[12:03:45][E][component:082]: Component esp32_camera is marked FAILED

However I have spend enough time trying to make it work. So I will not conplain now that it is working.

Thanks again

I am looking for a sample in arduino-esp32.
Bluetooth profiles have a2dp and spp. I would like to run this a2dp and spp at the same time. Have you seen any sample code anywhere that achieves this functionality?
If anyone knows of any, could you please let me know?
As for my specific story, I am running the ESP32-A2DP library and BluetoothSerial at the same time and using two M5 ATOM Lite devices to synchronize GIF animation and MP3 background music to be played back synchronously.

Thank you in advance.

Or in other words, there is no way to directly connect D- and D+ from the USB-C breakout to the M5PicoStamp.

Hi Felix,

thanks alot for that information.
Was hoping this converter could be easy be done by some ICs and restistors maybe but would be too difficult for me yet.
So i think i will get another controller with built in or a cojntroller with onboard USBC connection.
Makes no sense to spend much time and money if the device will be procudes 100 times each with MCU and a serial converter.

Your help was great!!!

Best regards
Hannes