cnnmmd

cnnmmd_xoxxox_tlkvam

Clients: Chat client (VR/MR app (Virt-A-Mate))

Implementation/Rights (Source Code/License): [※E]

・

https://github.com/cnnmmd/cnnmmd_xoxxox_tlkvam

※E

This code is available to the public so you can see how it works (it's still being cleaned up for collaboration).

dependence:

overview

Chat client for the VR/MR app (Virt-A-Mate):

・

Language used: C# (MVRScript)

・

Operating environment: Virt-A-Mate ( https://www.patreon.com/meshedvr )

・

Operating environment: Virtual Desktop ( https://www.vrdesktop.net )

・

Operating environment: Quest

・

Setup: Manual: Chat

*

If individual settings are required before installation, modify the file according to the steps below.

Copy the following folders to the following folders on the app side (VaM):

# Transcribed from:
> cnnmmd/export/app/xoxxox
 
> appvam

# Post to:
> <dirvam>\custom\scripts\*

Load the following as a plugin into any PERSON in your app:

> <person>
> plugins
> add plugin
> select file
  *\appvam\doc\AppTlk.clist

*

For lip sync, you can use, for example, the following plug-ins:

・

https://hub.virtamate.com/resources/realtime-lipsync.1286/

Setting: Manual: Action: Receive trigger

*

If individual settings are required before installation, modify the file according to the steps below.

To receive triggers from the server, load the following as a plugin into any atom in your app:

> <person>
> plugins
> add plugin
> select file
  *\appvam\doc\AppRcv.cslist

This plugin receives a string of button ID (UIButton#N, ...) and executes that button. [*1]

*1

For example, if you convert the results of emotion analysis into each button ID and send it, the character will react in accordance with the emotion of the response.

Setting: Manual: Action: Send Trigger

*

If individual settings are required before installation, modify the file according to the steps below.

To send triggers to the server, load the following as a plugin into any atom in your app:

> <person>
> plugins
> add plugin
> select file
  *\appvam\doc\AppSnd.cslist

This plugin will act as a trigger action with the following settings (when activated, it will send empty data to the specified endpoint): [*1]

> ATOM
> button trigger
 
> settings
   
> receiver atom: ATOM
   
> receiver: plugin#N_Xoxxox.SndAct
   
> receiver target: flgbtn
   
> check to send true when triggered / uncheck to send false
when triggered: <yes>

*1

At the moment this is experimental (we don't yet have the server side functionality to handle the triggers sent properly).

Setting: Setting

The individual configuration files are as follows (the contents of the files that specify addresses are also shown): [*A] [*B]

# Individual settings:
>
cnnmmd/import/cnnmmd_xoxxox_tlkvam/export/app/xoxxox/appvam/bin
 
> Params.cs
    - public static string srvadr = "127.0.0.1" // Relay server address

※A

To modify a plugin file, copy it to a custom folder in the same location, modify the copied file, and then update the main (export) version as needed:

# Plugin side:
> cnnmmd/import/${plugin}/export/.../${target}
# Custom folder side:
> cnnmmd/import_custom/${custom}/export/.../${target}

$ cd cnnmmd/manage/bin
$ ./remove.sh && ./append.sh

execution

Start the necessary servers and run the workflow.

Next, load/reload the plugin in your app - and the conversation will begin.

※C

At the beginning of a conversation or when switching speakers, the speech synthesis engine takes some time (possibly minutes on a CPU) to load the dictionary and individual models - while the speech synthesis node display is active, models are being loaded or speech data is being generated.