The quantized GGUF weights baked into this image

Unsloth Kimi K3 GGUF

<h2 id="kimi-k3-on-8x-mi300x">Kimi K3 on 8x MI300X</h2>
<p>This 1-Click image serves <strong>Kimi K3</strong> (Moonshot AI), a frontier-scale Mixture-of-Experts model, on a single 8x AMD Instinct MI300X node. Inference runs on <strong>llama.cpp built with the ROCm/HIP backend (gfx942)</strong> against <strong>ROCm 7.14</strong>, using Unsloth&#39;s <strong>Dynamic 2-bit <code>UD-Q2_K_XL</code></strong> quant. That quant is chosen deliberately: at 861 GB it fits in the node&#39;s aggregate 1536 GB of HBM while leaving room for a usable KV cache, where the next quant up (UD-Q4_K_XL, 1509 GB) would load but leave almost none.</p>
<h3 id="weights-are-baked-in-so-it-is-ready-on-first-boot">Weights are baked in, so it is ready on first boot</h3>
<p>All 19 <code>UD-Q2_K_XL</code> shards plus the vision projector are <strong>already in the image</strong> under <code>/opt/models/Kimi-K3-GGUF</code>. There is no first-boot download and no runtime dependency on Hugging Face being reachable. The build fails loudly if the bake is incomplete, so a published image always carries the full payload. Expect a couple of minutes for llama.cpp to load the weights onto the 8 GPUs after boot; the login banner tells you when the API is ready.</p>
<h3 id="two-ways-in-a-public-url-or-a-private-ssh-tunnel">Two ways in: a public URL, or a private SSH tunnel</h3>
<p><code>llama-server</code> exposes an OpenAI-compatible API (chat completions plus a browser chat UI) bound to <code>127.0.0.1</code>. On boot the droplet also brings up an ephemeral Cloudflare tunnel and prints the resulting public URL in the login banner, along with an <strong>API key generated uniquely for that droplet</strong>. No key is baked into the image, so nothing secret is published. Inference requests without the key are rejected. If you would rather not expose anything publicly, ignore the tunnel and reach the API over an SSH port-forward instead.</p>
<h3 id="thinking-only-so-budget-tokens-accordingly">Thinking-only, so budget tokens accordingly</h3>
<p>K3 always reasons before answering. Reasoning tokens count against <code>max_tokens</code>, so a small budget can be consumed entirely inside the thinking trace and return empty <code>content</code>. Use 2048 or more, and set depth with <code>reasoning_effort</code>.</p>
<h3 id="multimodal">Multimodal</h3>
<p>The image builds <code>llama-mtmd-cli</code> and loads the MoonViT-V2 vision projector, so image inputs work through the same API.</p>
<h3 id="8x-only">8x only</h3>
<p>The snapshot carries <code>min_disk</code> 2046 GB because of the baked weights, so DigitalOcean will only restore it onto the 8x MI300X SKU. That is also the only configuration with enough VRAM to serve this quant.</p>

Kimi K3 (UD-Q2_K_XL GGUF)

llama.cpp (Unsloth fork, HIP backend)

ROCm

Kimi K3 (Moonshot AI) served by llama.cpp on ROCm 7.14, using Unsloth's Dynamic 2-bit UD-Q2_K_XL quant sized to fit 8x AMD Instinct MI300X. The weights are baked into the image, so the OpenAI-compatible API comes up on first boot with no download step.

AMD Developer Cloud

Kimi K3 2-bit on 8x MI300X

Focus on building great things, while you use these 1-Click Apps to build your business.

These 1-Click Apps make it simple to deploy and manage your clients' infrastructure. Save time and resources, while reducing headaches and passing the value onto your clients.

Apps for agencies and website development shops

Work more effectively from home or remotely with these 1-Click Apps.

Work more effectively from home or remotely

Stop spending hours editing YAML and Helm charts. Instead, use these popular, preconfigured solutions to add apps to your Kubernetes cluster in minutes instead of days.

1-Click Apps for managing Kubernetes environments

Stay focused on teaching and learning, instead of managing infrastructure. 

1-Click Apps for students and educational institutions

Secure your network with a VPN, protect your database, and more, with this selection of 1-Click Apps.

SaaS Add-Ons make your DigitalOcean infrastructure better! Check out the full list of SaaS Add-Ons here!  

New Feature Alert: Introducing Software License Subscriptions on DigitalOcean Marketplace!

We now support Droplet 1-Click updates via API. See documentation for the details

Check out Tabnine, the AI coding assistant that accelerates & simplifies software development.

Description

Software Included

Getting started after deploying Kimi K3 2-bit on 8x MI300X