cnnmmd

composition

This tool is a reuse of the data conversion/server adjustment module for media production (see separate article).

The module itself is designed to connect multimodal (document/music/audio/image/video, etc.) server applications and diverse client applications with common elements, with the goal of improving its scalability and usability.

*

It's not specifically a system for talking to characters - the chat tool is just one workflow that uses one endpoint of the system (each conversation corresponds to one turn in the flow).

A server essentially runs as one server app in one container (the container can be located anywhere, locally or in the cloud).

The functionality of your server is determined by the engine you use - there are models that run locally or in the cloud (where you have full control) and those that use online services via APIs.

Intermediate connectors relay interactions between servers and between servers and clients -- communication takes place through endpoints on the intermediate connectors (you can create as many endpoints as your resources allow).

All data that flows through the intermediate connector is binary (even text) - data format conversion is done here, so the server application/client side can minimize the code implementation, and data storage/retrieval in temporary memory or files is also done here.

A workflow is created by connecting the functions (function nodes) of the intermediate connectors and specifying the flow of a repeating pattern (= one turn). The function nodes of a workflow have a one-to-one correspondence with the server applications on the container.

○

Data flow control (workflow)

・

CLI-style workflow (Python) - Control can be sequential, branched, and iterative

・

GUI-based workflow (ComfyUI): You can visually grasp the flow, but control is only sequential (however, there is a reasonable degree of freedom due to automatic return to the first function node, closed conditional branching by switch nodes, and memory function across turns).

○

Data manipulation by intermediate connectors (workflow function nodes)

・

Data export/import to file

・

Data output/input to memory (stored between turns)

・

Streaming data (music/video)

・

Data conversion (music/image/audio)

・

Polling for sending/receiving data (long polling)

*

By using containers (Docker), content and applications can be divided and fixed as independent packages. By connecting them over a network (TCP/IP), the configuration can be changed flexibly (loose coupling).

*

In fact, the actual entities of each node in the workflow can be anywhere -- nodes (apps) that can run on a PC run locally, nodes (apps) that require a GPU run in the cloud, nodes (apps) that require a service call an API, and so on -- and of course, the entities of intermediate connectors and GUIs (ComfyUI) can also be anywhere (in the workflow, apps are just labels (their addresses are written in the configuration file), so the workflow can be used as is even if the container/app placement changes).

Structure: Folders

All resources (workflow function nodes/engines/servers/configuration files, etc.) can be dynamically expanded to allow users to maximize functionality.

Therefore, the global namespace of tools is also based on the user (the creator's name).

The main body of this tool (export) is the expanded plugin group (import). Adding, deleting, starting and stopping the plugin group is managed collectively with a script (manage).

> ${dirtop}/cnnmmd

 
> export # The main body of this tool - folders/files extracted from plugins
> import # Plugins
> manage # Plugin management
> cnf # Configuration files
> cnfsrc.txt # Correspondence file between plugins and repositories
> cnfsrc_custom.txt # The patch file (overwrites duplicates)
   
> bin # Plugin management scripts
> lib # Plugin management libraries
> storage # For storing temporary files (shared between intermediate connectors, containers and hosts)

Structure: Folder: Archive (archive of plugins)

A plugin has a folder for extraction (export) and a folder (mangae) for storing its management scripts. You can add multiple custom folders (custom_*) to modify all the files of the tool (all plugin files).

> ${dirtop}/cnnmmd/import

 
> cnnmmd_<author>_<plugin> # Plugin
> export # Plugin deployment folder (this folder/file group will be deployed to the main body of this tool)
   
> manage # Management scripts for individual plugins (update-delete/start-stop)
   
> app<...> # Applications required to run the plugin
> cnnmmd_custom_<...> # Custom folders
> export # Folder for expanding custom files (These folders/files will be expanded into the main body of this tool - overwriting)
   
> manage # Management scripts
> cnf # Configuration files
> latest.txt # File indicating whether overwriting is allowed

Configuration: Folder: Deployment (Deployment of plugins)

Directly below the tool itself (first level) are the configuration files (cnf), executables (bin), static and dynamic libraries (lib, prc, dyn), and folders for content and applications -- all of which are basically the plugin author's folder at the second level.

> ${dirtop}/cnnmmd/export

 
> cnf # individual: configuration file
> bin # individual: program: executable -- but no path
> lib # individual: program: library (python): static
> prc # individual: program: library (python): dynamic (engines)
 
> dyn # Individual: Program: Library (python): Dynamic: Meta-program (a set of function node libraries)
 
> web # Individual: Content: For the web
> app # General: Applications - Resources mainly for client applications

> ${dirtop}/cnnmmd/export/cnf

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> envdir.txt # Shared paths (shared libraries, etc.)
   
> cnfnet.json # Common network settings (TCP/IP: SSL, etc.)
   
> docenv_<...>.txt # Image creation file for containers (Docker)
> docker_<...>.yml # Startup file for containers (Docker)
> docker_<...>_custom.yml # Patch file (additions/changes)
   
> config_(stt|tts|ttt|sen|img|...)<model>.json # Various AI
Configuration file for engine
> xoxxox_cnfsrv_(stt|tts|ttt|sen|img|...)<model>.json #
Correspondence between server label and entity (address) (<label_server>: <address_server>)
)
 
> xxoxxox_cnfcmf_(stt|tts|ttt|sen|img|...)<model>.json #
Selection items for GUI apps (ComfyUI) (lst(stt|tts|ttt|sen|img|...)_nod,
lst(stt|tts|ttt|sen|img|...)_cnf)

> ${dirtop}/cnnmmd/export/bin

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> flwscr_<...>.py # Workflows (CLI)
   
> srv<...>.py # various servers
> clt<...>.py # various clients (via the internet)
   
> cli<...>.py # Various clients (reading the engine directly)

> ${dirtop}/cnnmmd/export/lib

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)

> ${dirtop}/cnnmmd/export/prc

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> engine_(stt|tts|ttt|sen|img|...)<model>.py # Engines

> ${dirtop}/cnnmmd/export/dyn

 
> <author> # Author folder
> __init__.py
   
> <...>.py
 
> xoxxox # Folder by author (author: xoxxox)

> ${dirtop}/cnnmmd/export/web

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> (doc|img|vce) # Content group (document/image/audio)
   
> lib # Shared libraries (javascript)

> ${dirtop}/cnnmmd/export/app

 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> appcmf
     
> doc # Workflows (GUI)
       
> flwcmf_<...>.json
     
> lib # Custom nodes

Configuration: Folder: Deployment (Deployment of plugins): Individual: For chat

> ${dirtop}/cnnmmd/export/cnf
 
> xoxxox_cnfspk_tts<model>.json # List of speakers (for speech synthesis engine)