cnnmmd

cnnmmd_xoxxox_tlkfig

Clients: Chat client (Microcontroller: Bare metal (M5Stack Basic) + Figure)

dependence:

・

cnnmmd_xoxxox_libmcu

overview

Chat client that works with Figure:

・

Language used: C++ (Arduino)

・

Development environment: PlatformIO

・

Operating environment: Microcontroller (bare metal) + LCD + speaker: M5Stack Basic: M5STACK-K001-V27

・

Operating environment: Microcomputer (bare metal): M5Stack ATOM Lite: M5STACK-C008

・

Operating environment: Microphone: M5Stack: M5STACK-U089

・

Operating environment: Video camera: M5Stack: M5STACK-U174-B

・

Operating environment: Motor (servo motor: 180 degrees) x 2: M5Stack: M5STACK-A076-B

・

Operating environment: LED (3 colors)

・

Operating environment: Frame/link mechanism/gear: LEGO technic

Lip sync uses three-color LEDs to express the opening and closing of the mouth. The video camera uses object recognition to recognize the human (user) face, and the figure's head is rotated by a motor to track it.

Installation: Manual: Software side

*

This is the code for the camera/motor unit -- the code for the microphone/speaker is listed there, as it overlaps with another plugin (cnnmmd_xoxxox_tlkmcu).

Follow these steps to use the code found in the plugin, compile it, and transfer it to your device (these steps are CLI based, so adapt them accordingly if you are using a GUI):

◯

Configuration: Client side (tlkfig)

・

For: M5Stack ATOM Lite

# Setting $dirtop='...' # Top folder for this tool $dirapp="${dirtop}/cnnmmd/export/app/xoxxox/appmcu" #
Folder for this tool, for this application
$ dirmcu='...' # Top folder of the microcontroller development environment (PlatformIO)
$ dirprj_fig='prj002' # Microcontroller project folder: for camera/motor unit
$ devusb_fig='/dev/...' # Microcontroller serial device
$ bpsusb='115200' # Transfer baud rate
$ alias pio='${HOME}/.platformio/penv/bin/pio' #
Alias name of the executable file for the microcontroller development environment (PlatformIO) # Initial: Individual settings
$ cat     
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
$ mkdir -p
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act
$cp      
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
\
          
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
$ vi      
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
...
const char* keynet = "..."; // Wireless LAN ID 
const char* pwdnet = "..."; // Wireless LAN password...

# Initial: Reflect the individual settings of this tool
$ cd ${dirtop}/cnnmmd/manage/bin

$ ./remove.sh && ./append.sh

# Initial: Initialize the microcontroller project ~ Install libraries
$ mkdir ${dirmcu}/${dirprj_fig}
$ cd ${dirmcu}/${dirprj_fig}

$ pio project init --board m5stack-atom
$ pio pkg install -l 'bblanchon/ArduinoJson'
$ pio pkg install -l 'arminjo/ServoEasing@^3.4.0'
$ pio pkg install -l 'madhephaestus/ESP32Servo@^3.0.6'

# Initial: Refer to/transcribe the resources of this tool
$ cd ${dirmcu}/${dirprj_fig}/src
$ rm params.h main.cpp
$ ln -s ${dirapp}/src_act/params_actfig.h params.h
$ ln -s ${dirapp}/src_act/main_actfig.cpp main.cpp

# Operation: Compile
$ cd ${dirmcu}/${dirprj_fig}
$ pio run

# Operation: Transfer to device
$ cd ${dirmcu}/${dirprj_fig}
$ pio run --target upload --upload-port $devusb_fig} # Transfer (code)
$ pio device monitor --port $devusb_fig} --baud ${bpsusb} #
Check (via serial)

◯

Check: Client side: Video camera ()

To check the address assigned to your camcorder, follow these steps:

・

Development environment: Arduino IDE

> tools
 
> boards -> esp32 -> m5unitcams3
 
> usb cdc on boot -> enabled
 
> psram -> opi psram

> tools
 
> port: /dev/...

> tools
 
> serial monitor # Check the assigned address here

Check that you can actually access the assigned address:

・

Setting screen: http://...

・

Output video (stream): http://...:81/stream

◯

Configuration: Server side

# Initial: Individual setting: Server side (corresponding to the client's camera/motor unit)

$ cat     
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/cnf/xoxxox/cnffig.json
$ mkdir -p
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox
$cp      
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/cnf/xoxxox/cnffig.json
\
          
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox/cnffig.json
$ vi      
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox/cnffig.json
...
  "urlact": "http://.../", # Motor address "urlcam": "http://...:81/stream", # Camera address...

# Initial: Reflect the individual settings of this tool
$ cd ${dirtop}/cnnmmd/manage/bin

$ ./remove.sh && ./append.sh

*

The code for this plugin is written for PlatformIO, but as of now (March 2025), it is difficult to develop the above video camera in this environment (this appears to be due to differences in thinking between the PlatformIO community and Espressif) - however, M5Stack has released code for the Arduino IDE, and we have verified that it works (the video camera in this video was compiled and run using that code as is).

Installation: Manual: Hardware side

The two servo motors (180 degrees) are positioned so that they are perpendicular to each other (rotation of the neck is up and down and left and right - there is no tilting of the head).

The video camera is positioned so that its movement matches the head - this code rotates the servo motors (the head) so that the face seen by the camera is always in the center of the frame (screen).

execution

Start the necessary servers and run the workflow.

Next, the microcontroller is turned on - the motors begin to rotate in sync with the object recognition by the camera.