cnnmmd

cnnmmd_xoxxox_tlkmcu

Clients: Chat client (microcontroller: bare metal (M5Stack Basic))

dependence:

・

cnnmmd_xoxxox_libmcu

overview

Chat client on a microcontroller (bare metal):

・

Language used: C++ (Arduino)

・

Development environment: PlatformIO

・

Operating environment: Microcontroller (bare metal) + LCD + speaker: M5Stack Basic: M5STACK-K001-V27

・

Operating environment: Microphone: M5Stack: M5STACK-U089

Lip sync only periodically switches the image around the mouth (while speech is being generated (playback)) (to reduce the load). [※1]

The character images use the BMP format (because switching images is slow when using JPEG) - for this reason, the images are pre-installed on the microcontroller (they can also be transferred over a network, but BMP images are large in size and take a long time to start up).

*1

However, it often takes a considerable amount of time between the end of one turn of conversation and the start of the next process.

Installation: Manual

*

This is code for a microphone/speaker unit - however, it shares some common parts with another plugin (cnnmmd_xoxxox_tlkfig), so you only need to run it where necessary.

Follow these steps to use the code found in the plugin, compile it, and transfer it to your device (these steps are CLI based, so adapt them accordingly if you are using a GUI):

◯

Configuration: Client side (tlkmcu/tlkfig)

・

Target: M5Stack Basic

・

Reference: For microphone unit (PDM): https://github.com/m5stack/M5-ProductExampleCodes/blob/master/Unit/PDM

# Setting $dirtop='...' # Top folder for this tool $dirapp="${dirtop}/cnnmmd/export/app/xoxxox/appmcu" #
Folder for this tool, for this application
$ dirmcu='...' # Top folder of the microcontroller development environment (PlatformIO)
$ dirprj_mcu='prj001' # Microcontroller project folder: for microphone/speaker unit
$ devusb_mcu='/dev/...' # Microcontroller serial device
$ bpsusb='115200' # Transfer baud rate
$ alias pio='${HOME}/.platformio/penv/bin/pio' #
Alias name of the executable file for the microcontroller development environment (PlatformIO) # Initial
$ mkdir ${dirmcu}

# Initial: Individual setting: In the following case: tlkmcu

$ mkdir -p
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk
$cp      
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkmcu/export/app/xoxxox/appmcu/src_tlk/params_swtimg.h
\
          
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk/params_swtimg.h
$ vi      
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk/params_swtimg.h
...
const char* wifiid = "..."; // Wireless LAN ID 
const char* wifipw = "..."; // Wireless LAN password const char* srvadr = "..."; // Address of intermediate connector...

# Initial: Individual setting: In the following case: tlkfig

$ mkdir -p
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk
$cp      
${dirtop}/cnnmmd/import/cnnmmd_xoxxox_tlkfig/export/app/xoxxox/appmcu/src_tlk/params_swtled.h
\
          
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk/params_swtled.h
$ vi      
${dirtop}/cnnmmd/import/custom_appmcu_appfig/export/app/xoxxox/appmcu/src_tlk/params_swtled.h
...
const char* wifiid = "..."; // Wireless LAN ID 
const char* wifipw = "..."; // Wireless LAN password const char* srvadr = "..."; // Address of intermediate connector...

# Initial: Reflect the individual settings of this tool
$ cd ${dirtop}/cnnmmd/manage/bin

$ ./remove.sh && ./append.sh

# Initial: Initialize the microcontroller project ~ Install libraries
$ mkdir ${dirmcu}/${dirprj_mcu}
$ cd ${dirmcu}/${dirprj_mcu}

$ pio project init --board m5stack-core-esp32
$ pio pkg install --library 'm5stack/M5Stack'

$ cd ${dirmcu}/${dirprj_mcu}/lib
$ mkdir PDM
$ cd ${dirmcu}/${dirprj_mcu}/lib/PDM

$ curl -O
https://raw.githubusercontent.com/m5stack/M5-ProductExampleCodes/master/Unit/PDM/fft.h
$ curl -O
https://raw.githubusercontent.com/m5stack/M5-ProductExampleCodes/master/Unit/PDM/fft.cpp

# Initial: View/copy resources for this tool
$ cd ${dirmcu}/${dirprj_mcu}/lib
$ ln -s ${dirapp}/lib/AppTlk AppTlk

$ cd ${dirmcu}/${dirprj_mcu}
$ mkdir data
$ cd ${dirmcu}/${dirprj_mcu}/data
$ cp ${dirapp}/data_tlk/*.bmp .

# Operation
$ cd ${dirmcu}/${dirprj_mcu}/src
$ rm params.h main.cpp

# Operation: In the following case: tlkmcu

$ ln -s ${dirapp}/src_tlk/params_swtimg.h params.h
$ ln -s ${dirapp}/src_tlk/main_swtimg.cpp main.cpp

# Operation: In the following case: tlkfig

$ ln -s ${dirapp}/src_tlk/params_swtled.h params.h
$ ln -s ${dirapp}/src_tlk/main_swtled.cpp main.cpp
$ ln -s ${dirapp}/src_tlk/main_swtled_try002.cpp main.cpp # Test (lights up)

# Operation: Compile
$ cd ${dirmcu}/${dirprj_mcu}
$ pio run

# Operation: Transfer to device
$ cd ${dirmcu}/${dirprj_mcu}
$ pio run --target uploadfs # Transfer (image) *Only when there is an image update
$ pio run --target upload --upload-port $devusb_mcu} # Transfer (code)
$ pio device monitor --port $devusb_mcu} --baud ${bpsusb} #
Check (via serial)

execution

Start the necessary servers and run the workflow.

Next, turn on the microcontroller. The following message will appear on the screen (LCD). Press the right button (of the three buttons on the case) to start speaking. The speech will end when the time (number of seconds) specified in the configuration file has elapsed: [※C]

> prc: ini

※C

At the beginning of a conversation or when switching speakers, it takes some time (possibly minutes on a CPU) for the speech synthesis engine to load the dictionary and individual models -- while the speech synthesis node display is active, models are loading or speech data is being generated.