≡ menu
× menu

cnnmmd

overview

A chat client that works with figures: [※1]

・
Language used: C++ (Arduino)
・
Development environment: PlatformIO
・
Operating environment: Microcontroller (bare metal) + LCD + speaker: M5Stack Basic: M5STACK-K001-V27
・
Operating environment: Microcomputer (bare metal): M5Stack ATOM Lite: M5STACK-C008
・
Operating environment: Microphone: M5Stack: M5STACK-U089
・
Operating environment: Video camera: M5Stack: M5STACK-U174-B
・
Operating environment: Motor (servo motor: 180 degrees) x 2: M5Stack: M5STACK-A076-B
・
Operating environment: LED (3 colors)
・
Operating environment: Frame/link mechanism/gear: LEGO technic

Lip syncing uses three-color LEDs to express the opening and closing of the mouth. Object recognition using a video camera recognizes the user's face, and the figure's head is rotated by a motor to track the face.

*1
The part model number is Switch Science's product code.

Installation: Manual: Software side

*
This is the code for the camera/motor unit - the code for the microphone/speaker is shared with another plugin (cnnmmd_xoxxox_tlkmcu), so it's listed there.

Follow these steps to use the code found in the plugin, compile it, and push it to your device (these steps are CLI-based, so adapt them accordingly if you're using a GUI):

◯
Configuration: Client side (tlkfig)
・
Applicable to: M5Stack ATOM Lite
# Setting
$ dirtop='...' # Top folder of this tool
$ dirapp="cnnmmd/export/app/xoxxox/appmcu" # Folder of this tool for this app
$ dirmcu='...' # Top folder of microcontroller development environment (PlatformIO)
$ dirprj_fig='prj002' # Microcontroller project folder: for camera/motor unit
$ devusb_fig='/dev/...' # Microcontroller serial device
$ bpsusb='115200' # Transfer baud rate
$ alias pio='${HOME}/.platformio/penv/bin/pio' # Alias name of the executable file for the microcontroller development environment (PlatformIO) # Initial: Individual settings
$ cat      cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
$ mkdir -p cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act
$cp       cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
\
          
cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
$ vi       cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_act/params_actfig.h
...
const char* keynet = "..."; // Wireless LAN ID
const char* pwdnet = "..."; // Wireless LAN password...
# Initial: Reflect the individual settings of this tool
$ cd cnnmmd/manage/bin
$ ./remove.sh && ./append.sh
# Initial: Initialize the microcontroller project ~ Install libraries
$ mkdir ${dirmcu}/${dirprj_fig}
$ cd ${dirmcu}/${dirprj_fig}
$ pio project init --board m5stack-atom
$ pio pkg install -l 'bblanchon/ArduinoJson'
$ pio pkg install -l 'arminjo/ServoEasing@^3.4.0'
$ pio pkg install -l 'madhephaestus/ESP32Servo@^3.0.6'
# Initial: Reference/transcribe the resources of this tool
$ cd ${dirmcu}/${dirprj_fig}/src
$ rm params.h main.cpp
$ ln -s ${dirapp}/src_act/params_actfig.h params.h
$ ln -s ${dirapp}/src_act/main_actfig.cpp main.cpp
# Operation: Compile
$ cd ${dirmcu}/${dirprj_fig}
$ pio run
# Operation: Transfer to device
$ cd ${dirmcu}/${dirprj_fig}
$ pio run --target upload --upload-port $devusb_fig} # Transfer (code)
$ pio device monitor --port $devusb_fig} --baud ${bpsusb} # Check (via serial)
◯
Check: Client side: Video camera ()

To find the address assigned to your camcorder, follow these steps:

・
Development environment: Arduino IDE
> tools
 
> boards -> esp32 -> m5unitcams3
 
> usb cdc on boot -> enabled
 
> psram -> opi psram
> tools
 
> port: /dev/...
> tools
 
> serial monitor # Check the assigned address here

Check that you can actually access the assigned address:

・
Settings page: http://...
・
Output video (stream): http://...:81/stream
◯
Configuration: Server side
# Initial: Individual setting: Server side (corresponding to the client's camera/motor unit)
$ cat      cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/cnf/xoxxox/cnffig.json
$ mkdir -p cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox
$cp       cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/cnf/xoxxox/cnffig.json \
          
cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox/cnffig.json
$ vi       cnnmmd/import/custom_appmcu_appfig/export/cnf/xoxxox/cnffig.json
...
  "urlact": "http://.../", # Motor address "urlcam": "http://...:81/stream", # Camera address...
# Initial: Reflect the individual settings of this tool
$ cd cnnmmd/manage/bin
$ ./remove.sh && ./append.sh
*
The code for this plugin is written for PlatformIO, but as of now (March 2025), it is difficult to develop the video camera mentioned above in this environment (apparently due to differences in thinking between the PlatformIO community and Espressif). However, M5Stack has released the code for the Arduino IDE, and we have verified that it works (the video camera in this video was compiled and run using that code as is).

Installation: Manual: Hardware side

Two servo motors (180 degrees) are positioned so that they are perpendicular to each other (rotation is up and down and left and right of the neck - there is no tilting of the head).

The video camera is positioned so that its movement matches the head - this code rotates the servo motors (= head) so that the face recognized by the camera is always centered in the frame (screen).

execution

Start the necessary servers and run the workflow.

Next, the microcontroller is powered on - the motors begin to rotate in time with the camera's object recognition.