Godot LLM template

A template/demo to show how to use the godot-llm addon

Text generation

Similarity

Image to text

Vector database

How to use

You may simply download the Android apk in the release page to try it out for Android devices.

For Windows/Linux:

1. Get the Godot LLM Template from the Godot asset library, or clone this project: git clone https://github.com/Adriankhl/godot-llm-template.git
2. Get Godot LLM directly from the godot asset library, or download the addon from godot-llm and place it to the addons folder
3. Download models
4. Run the project with Godot or export an Android apk (requires the Manage External Storage permission)

Download models

• Text generation recommendation: Meta-Llama-3-8B-Instruct-Q5_K_M.gguf or Phi-3-mini-4k-instruct-Q2_K.gguf for low-end devices such as Android phone
• Similarity/Vector database recommendation: mxbai-embed-large-v1.Q5_K_M.gguf
• Image to text recommendation: llava-phi-3-mini-int4.gguf, also download the llava-phi-3-mini-mmproj-f16.gguf from the same page

Folder and file structure:

├── addons
│   └── godot_llm
│       ├── bin
│       │   ├── libgodot_llm.android.debug.aarch64.so
│       │   ├── libgodot_llm.android.release.aarch64.so
│       │   ├── libgodot_llm.linux.debug.x86_64.so
│       │   ├── libgodot_llm.linux.release.x86_64.so
│       │   ├── libgodot_llm.windows.debug.amd64.dll
│       │   └── libgodot_llm.windows.release.amd64.dll
│       └── godot_llm.gdextension
├── db.gd
├── db.tscn
├── icon.svg
├── icon.svg.import
├── image_to_text.gd
├── image_to_text.tscn
├── LICENSE
├── main.gd
├── main.tscn
├── media
├── project.godot
├── README.md
├── similarity.gd
├── similarity.tscn
├── text_generation.gd
└── text_generation.tscn

You can now open the application with Godot

The application

You will see 4 modes: Text Generation, Similarity, Image to Text, and Vector Database.

In each mode, you need to click Model to point to the correct model gguf file. Addition, in Image to Text mode, you need to click Mmproj to point to your *mmproj*gguf file.

Text Generation

You will see there are 3 generation mode: Simple, Instruct, and Interactive.

Simple mode

Just edit the prompt and click Generate. Click Stop to stop the generation.

You choose None schema or Person schema, if you choose None schema, the model will generate the information of a character with Json format.

Instruct mode (currently this is disabled)

Click Start first with empty prompt, then modify the prompt and click Continue when it is available to talk to the AI. This is an interactive mode. Whenever you see the Continue button is available, either input something in the prompt and click Continue to send the input, or simply click Continue to let the AI talk itself.

Interactive mode

Similar to the instruct mode, except that it relies on a good initial prompt and additoinal reverse prompt and input suffix settings to generate a smooth conversation. The preset prompt should work decently well, click the Start button with the preset prompt and start talkin to the AI.

Similarity

Input prompts to compute the Embedding, or input two prompts to calculate their Similarity

Image to Text

Select an Image from your device. Alternative, click From View to capture your game screen as the image. You can Compute Base64 encoding for your image, or decode a base64 string back to Compute Image. Edit the prompt, and generate text from Image, Base64, or View (your game screen).

Vector Database

Click Metadata: id, year to define the metadata of the tables, then click Create tables to create a some tables for embedding storage and retrieval. Click Store text to split the document into chunks and store into the database, you may also click Split text to see the chunks. Click Retrieve to retrive three most similar text chunks with the upper left prompt, which satisfy the sql-where-clause in the middle right. You may also turn on "Verbose stdout", modify the upper prompt to a valid sql statement, and click execute to run the sql statement directly, you will see the result in godot log.