cnnmmd

composition

This tool is designed with connectivity, scalability, and convenience in mind to connect multimodal (document/music/voice/image/video, etc.) server applications with a variety of client applications: [*1]

◯

Connectivity

・

Separating the operating environments of the server groups / Standardizing operations for adding, deleting, starting, and stopping (using containers)

・

Centralize communication between servers and between servers and clients (introduce a relay server)

◯

Scalability

・

Separation of server communication functions and engine/dynamic addition of engine (dynamic library)

・

Dynamically add a relay server handler (dynamic library + dynamic code generation)

・

Plugin independence (separating global namespace by author name)

◯

Convenience

・

Standardize the operations of acquiring, deleting, starting, and stopping plugins (introduce management module)

・

You can create flows that support complex processes (sequential, branching, and repetition) (CLI version of the workflow creation environment)

・

Visually create workflows (GUI workflow creation environment)

*1

The chat tool is also an extension of this tool -- although currently, chat uses one endpoint on the relay server, with a simple (one-way/synchronous) flow that repeats one-on-one conversation per turn -- but true AI + VR/MR chat will be a many-to-many asynchronous stream communication, so the tool will also need to support that configuration (the relay server will be a stack that shares status, and the servers will asynchronously add and retrieve status from the stack -- in this case, a GUI workflow creation environment like ComfyUI will no longer be able to handle it with a DAG, and a different UI will be required).

flow

The server basically runs as one server application on one container (the container can be located anywhere, such as locally or remotely). [*1]

The functionality of the server is determined by the engine it uses (which can be any dynamic library model running locally or remotely, or consuming services via an API).

The relay server relays communication between servers and between servers and clients. All data that flows through it is in binary format, and it also converts this data and stores it in temporary memory or files (these functions, or relay handlers, can be added as desired as dynamic libraries and dynamic code). [*2]

All communication takes place through a set of endpoints on the relay server (you can create as many endpoints as your resources allow).

A workflow is created by connecting the handlers of the relay server as nodes and specifying the flow of a repeating pattern (= 1 turn) (basically, the nodes of a workflow correspond one-to-one with the servers on the container). [*3]

○

Data flow control (workflow)

・

CLI-style workflow (Python) - Control can be sequential, branched, and repeated

・

GUI-based workflow (ComfyUI): You can visually grasp the flow, but control is only sequential (however, there is a certain degree of freedom thanks to features such as automatic return to the first node, closed conditional branching using switch nodes, and memory functions that span turns).

○

Data manipulation by relay servers (relay handlers/workflow nodes)

・

Data export/import to file

・

Data output/input to memory (stored between turns)

・

Streaming data (music/video)

・

Data conversion (music/image/audio)

・

Data sending/receiving polling (long polling)

*1

By using containers (Docker), content and applications can be divided and fixed as independent packages. By connecting them over a network, the configuration can be flexibly changed (loose coupling).

*2

Data conversion is also performed on the relay server side, so code implementation on the server and client sides can be kept to a minimum.

*3

With this configuration, the actual entities of each node in the workflow are placed in the appropriate location depending on their purpose -- nodes (apps) that can be run on a PC are launched locally, nodes (apps) that require a GPU are run remotely, nodes (apps) that require a service call an API, and so on -- and of course, the actual entities of the relay server and GUI (ComfyUI) can also be located anywhere (in the workflow, apps are just labels (their addresses are written in the configuration file), so the workflow can be used as is even if the container/app placement changes).

Structure: Folder

All resources (relay handlers, engines, servers, configuration files, etc.) are dynamically scalable to allow users to maximize functionality.

Therefore, the global namespace of tools is also based on users (creator names).

The main body of this tool (export) is the expanded folder of the plugins (import). Adding, deleting, starting, and stopping the plugins is managed collectively using a script (manage).

> cnnmmd
 
> export # The main body of this tool - folders/files extracted from plugins
> import # Plugins
> import_cuntom # Custom folders (overwrite the main body with these files)
 
> manage # Manage plugins
> cnf # Configuration files
> cnfenv.txt # Corresponding file for container images/registries
> cnfenv_custom.txt # Patch file (overwrites duplicates)
     
> cnfsrc.txt # Corresponding file for plugins/scripts/repositories
> cnfsrc_custom.txt # The patch file (overwrites any duplicates)
     
> depend.txt # Dependencies of plugins
> depend_custom.txt # Patch file
> bin # Plugin management scripts
> lib # Plugin management libraries
> storage # For storing files (shared between the relay server container and the host)

Each patch file can be written in the following format:

・

cnfenv_custom.txt

<container_image> <registry> # Container image/registry correspondence

<registry> : - # Do not pull this container image <registry> : + # Build a container image locally <registry> : ... # Pull a container image from this registry

・

cnfsrc_custom.txt

<plugin> <script> <branch>(| <repogitory>)
# Plugin/script/repository support

<script> : - # Do not run a shell script <script> : + # Run a shell script <branch> : - # Do not get this plugin <branch> <repogitory> : ... ... #
Get the plugin from this branch in this repository

・

depend_custom.txt

<plugin> # Plugin / any name - <plugin> # Dependent plugins (can overlap with other items - in that case, the dependencies will be deeper from top to bottom)
- ...

Structure: Folder: Archive (archive of plugins)

All plugins are placed in a designated folder (import). A plugin has an export folder and a management script folder (mangae). Users can add multiple custom folders (import_custom) that modify all files in the tool (all plugin files).

> cnnmmd/import
 
> <...> # Folder for plugins
> export # Folder for extracting files (This folder/files will be extracted (overwritten) to the tool itself)
   
> manage # Management scripts
> bin
       
> create.sh|delete.sh|launch.sh|finish.sh #
Add (update) - Delete / Start - Stop (automatically start from the main unit)
   
> app<...> # Server-side applications required to run plugins

> cnnmmd/import_custom
 
> <...> # Custom folder
> export # Folder for expanding files (This folder/files will be expanded (overwritten) into the tool itself)
   
> manage # Management scripts
> bin
       
> create.sh|delete.sh|launch.sh|finish.sh #
Add (update) - Delete / Start - Stop (automatically start from the main unit)

Configuration: Folder: Deployment (Deployment of plug-ins)

Directly below the tool itself (first level) are folders for configuration files (cnf), executable files (bin), static and dynamic libraries (lib, prc, dyn), and content and applications (web, app) - the second level of all folders is essentially the plugin author's folder.

> cnnmmd/export
 
> cnf # Individual: Configuration file
> bin # Individual: Program: Executable file - but path is not set
> lib # Individual: Program: Library (Python): Static
> prc # Individual: Program: Library (Python): Dynamic (Implementation of engines)
 
> dyn # Individual: Program: Library (python): Dynamic + dynamic code generation (implementation of relay server handlers)
 
> web # Individual: Content: For web
> app # General: Apps - Resources mainly for server-side applications

> cnnmmd/export/cnf
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> envdir.txt # Shared paths (shared libraries, etc.)
   
> cnfnet.json # Common network settings (TCP/IP: SSL, etc.)
   
> docenv_<...>.txt # Image creation file for containers (Docker)
> docker_<...>.yml # Startup file for containers (Docker)
> docker_<...>_custom.yml # Patch file (additions/changes)
   
> config_(stt|tts|ttt|sen|img|...)<model>.json # Various AI
Configuration file for engine
> xoxxox_cnfsrv_(stt|tts|ttt|sen|img|...)<model>.json #
Correspondence between server label and entity (address) (<label_server>: <address_server>)
)
 
> xxoxxox_cnfcmf_(stt|tts|ttt|sen|img|...)<model>.json #
Selection items for GUI application (ComfyUI) (lst(stt|tts|ttt|sen|img|...)_nod,
lst(stt|tts|ttt|sen|img|...)_cnf)

> cnnmmd/export/bin
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> flwscr_<...>.py # Workflows (CLI)
   
> srv<...>.py # various servers
> clt<...>.py # various clients (via the internet)
   
> cli<...>.py # Various clients (directly read the engine)

> cnnmmd/export/lib
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)

> cnnmmd/export/prc
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> engine_(stt|tts|ttt|sen|img|...)<model>.py # Engines

> cnnmmd/export/dyn
 
> <author> # Author folder
> __init__.py
   
> <...>.py
 
> xoxxox # Folder by author (author: xoxxox)

> cnnmmd/export/web
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> (doc|img|vce) # Content group (document/image/audio)
   
> lib # Shared library (javascript)

> cnnmmd/export/app
 
> <author> # Folder by author
> xoxxox # Folder by author (author: xoxxox)
   
> appcmf
     
> doc # Workflows (GUI)
       
> flwcmf_<...>.json

Configuration: Folder: Deployment (Deployment of plugins): Individual: For chat

> cnnmmd/export/cnf
 
> xoxxox_cnfspk_tts<model>.json # List of speakers (for speech synthesis engine)