cnnmmd

For example, you can create an image using the following steps: Create/generate a 2D image of the character - Change the expression around the mouth (using inpaint, etc.) - Mask the background (using Rembg, etc.) - Make the mask transparent (alpha). [※1]

The created image will be placed (by default) in the following location/with the following filename:

> cnnmmd/import_custom/<folder>/export/web/xoxxox/img
 
> <image>_c.png # Image with closed mouth
> <image>_o.png # Image with open mouth

Change the following variable to match the image filename (if you're using ComfyUI conversation flow, change it from the following node):

・

Xoxxox_SwtImg

> adrchr = "/xoxxox/img/<c>_<m>.png" #
Image path (no change necessary if you do not change the default path)
> imgchr = "<image>" # Common name of the image (the part that corresponds to "<c>" in the path)

Apply the changes you made to the tool itself:

$ ./remove.sh && ./append.sh

If you have already started the servers, restart them (if you are using the ComfyUI conversation flow, restart the following three servers):

$ docker restart xxoxxox_appmid
$ docker restart xxoxxox_appcmf
$ docker restart xxoxxox_appweb

*1

The Mask Alpha node is also available in this tool:

・

Xoxxox_TrnBak

The text output by the language generation node is analyzed by the sentiment analysis node - and the results are matched to image filenames for 2D character expressions:

・

Xoxxox_GenTxt

・

Xoxxox_CnvSen

・

Xoxxox_SenTxt

The image file name is sent from the polling sender node to the specified endpoint (specified by a three-digit number):

・

Xoxxox_DlyGet

Change the following client-side variables to match the specified endpoint (for ComfyUI conversation flows, change them from the following node):

・

Xoxxox_SwtImg

> pthswc = "gppNNN" # Path of the receiving endpoint (match the number with the polling sending node)

We'll switch roles between turns - we'll use a memory node to remember the response from the previous turn:

・

Xoxxox_SetMem

・

Xoxxox_GetMem

We use a group of nodes that extract binary data from data keys: [※1][※2]

・

Xoxxox_SetTxt / Xoxxox_GetTxt …… Receiving/transferring text data

・

Xoxxox_SetImg / Xoxxox_GetImg …… Receiving/delivering image data

・

Xoxxox_SetAud / Xoxxox_GetAud …… Receiving/transferring audio data

*1

All binary data flows through the relay server and is connected by data keys - no binary data flows to the GUI workflow side (ComfyUI).

*2

Using this tool from ComfyUI doubles the data flow, and both are server/client configurations, which may seem needlessly redundant -- but the cloud side of ComfyUI's APIs should also be configured similarly to this tool (by outsourcing heavy data conversion and tedious server adjustments, ComfyUI can focus on generating and processing content) -- so you could also say that the cloud's functionality has become more compact, allowing you to more freely manipulate the flow...

Use a group of nodes for storing files: [※1]

・

Xoxxox_SetDir / Xoxxox_GetDir …… Store/get file (single file)

・

Xoxxox_SetDis / Xoxxox_GetDis …… Store/get files (multiple files)

The relay server container shares the following folder on the host (PC) with the following path - you can use this folder as a temporary location for data:

> cnnmmd/storage # Host side (PC side)
> /opt/storage # container side

*1

When storing multiple files in a folder, the data key (in ascending order of creation time) is used as the file name.

Currently, the web client app supports receiving a single trigger that changes the face (including the conversation flow on ComfyUI):

・

cnnmmd_xoxxox_tlkweb

・

cnnmmd_xoxxox_tlkelc

The following client app plugins can send and receive multiple triggers:

・

cnnmmd_xoxxox_tlkvam

Apply the patch file (*_custom.yml) to the relay server startup file and increase the number of endpoints to 4 or more:

・

cnnmmd_xoxxox_appmid

The number of endpoints is specified as follows:

command: bash -c "python /opt/common/bin/xoxxox/srvmid.py --numset
<n> --numget <n>"

Here's a sample patch file:

> export_option-opt002/cnf/docker_appmid_custom.yml

For standard output, we use the speech output/language output/emotion classification nodes:

・

cnnmmd_xoxxox_ttsply

・

cnnmmd_xoxxox_tttsim

・

cnnmmd_xoxxox_sensim

Use a speech recognition node with empty string output: [※1]

・

cnnmmd_xoxxox_sttnil

*1

For example, if there is any vocalization, such as "yes" or "uh," it will detect it and move on to the next node.

Use the CLI workflow.

For Python, we have prepared a library that simplifies operation, but basically, all you need to do is exchange JSON/binary data with the relay server endpoints - so it can also be operated with other languages (such as JavaScript). [※1]

The following plugins have examples for Python:

・

cnnmmd_xoxxox_mgrpyt_con_try_001

・

cnnmmd_xoxxox_mgrpyt_web_tlk_wsp_vox_lcp_001

*1

The GUI workflow is just a wrapper around the CLI version of the workflow.

Currently, the following servers use app authentication (API KEY):

・

cnnmmd_xoxxox_imgdll

・

cnnmmd_xoxxox_imgnai

・

cnnmmd_xoxxox_sttoaw

・

cnnmmd_xoxxox_ttsnjv

・

cnnmmd_xoxxox_tttgpt

・

cnnmmd_xoxxox_tttnai

・

cnnmmd_xoxxox_sengpt

To change the path to the authentication file, apply a patch file (docker_*.yml) to your container's startup file (docker_*_custom.yml) - in the patch file, change:

env_file:
- ${HOME}/sec/key/.envath_*.txt

Apply the patch file (docker_*.yml) to your container's startup file (docker_*_custom.yml) - in the patch file, change the following:

Ports:
- <port_number>:80

Change the following items (according to your customization instructions) in the following files:

・

cnnmmd_xoxxox_appmid

・

cnnmmd/export/cnf/xoxxox_cnfsrv_appmid.json

"xoxxox_appmid_rem" : "http://localhost:10001"

・

cnnmmd_xoxxox_appweb

・

cnnmmd/export/cnf/xoxxox_cnfsrv_appweb.json

"xoxxox_appweb_rem" : "http://localhost:10002"

◯

Building the environment (using the provided container images)

Server apps that support both CPU and GPU provide both container images.

To use a GPU-optimized container image, specify the corresponding registry in the patch file - the GPU-optimized registry is commented out in the original file. Copy this (uncommented) into the patch file: [1]

・

Registry specification: Copy source: cnnmmd/manage/cnf/cnfenv.txt

#<container_image> <registry>-arc<number>

・

Registry specification: Copy destination: cnnmmd/manage/cnf/cnfenv_custom.txt

<container_image> <registry>-arc<number> #
Remove the comment out

・

Reference: cnfcmm

◯

Building an environment (creating a container image in your own environment)

You can also build your own image using the image build file for GPU builds and the registry entry for " +":

・

Registry specification: cnnmmd/manage/cnf/cnfenv_custom.txt

<container_image> +

・

Build configuration file: PLUGIN/export_option-arNNN/docenv_env*.txt

・

Reference: cnfcmm

◯

Launching the environment

You will also need to apply a patch file to your container's startup file - sample GPU patch files are provided in the following folders for each plugin:

・

Patch file: docker_*_custom.yml

・

Sample: export_option-arcNNN

*

If you have already pulled/created a CPU container image, remove it:

・

docker rmi...

Use a custom dependency file -- any name followed by "- ..." followed by the plugins it depends on -- and then start it by name like a normal plugin (starting the servers is handled automatically by the dependent plugins):

・

cnnmmd/manage/cnf/depend_custom.txt

・

Reference: cnfcmm

Get and place the given custom node:

・

howcmf

Tool resources are divided by managing container networks, container groups, and port groups for each user and allocating them using configuration files and proxy servers. [*1]

Even with this configuration, all users can use the same GUI workflow (because individual server names and port numbers are written in the configuration file). [*2]

*1

Authenticating with a proxy server and creating an account for each user will further strengthen security.

*2

The number of tools that can actually be run depends on the constraints of server resources (CPU and GPU).

All executables/configuration files for the tool can be overwritten from any custom folder: [※1]

・

showlist

*1

You can create as many custom folders as you like, so you can choose which ones apply depending on your situation and needs.

Publish your custom folder in Git format.

・

showlist

Add the plugin address to the repository list patch file:

・

showlist