TarqaAITarqaAI
PartnerStart free
Tarqa.AI
TARQAAI DOCSAPI v1.0 · OpenAI-compatible
All systems operational
60+ modelsRAGGPU tunnels

GPU WebSocket Tunnels

Beta

Run open-weight models on your own hardware, call them through the standard TarqaAI API

What is a GPU tunnel?

The tunnel agent is a small Node.js daemon you run on a machine with a GPU. It opens an outbound WebSocket connection to TarqaAI — no inbound ports or firewall changes needed — and relays chat requests to a model server already running locally (Ollama, LM Studio, or vLLM). Your team then calls that model through the normal TarqaAI chat API, at $0 cloud inference cost.

Beta — install from source

The tunnel agent (packages/tunnel-agent, CLI name tarqa-tunnel) isn't published to npm yet. The setup below uses the real, currently-working flow: register the agent in the dashboard, then run it from source on your GPU machine.

Setup flow

1

Register a tunnel agent

In the dashboard, go to GPU Tunnel & Hosted Models → Register Agent, and give it a name, OS, GPU info, and total memory. This registers the machine — not a model yet.
2

Download the config and run the agent

The dashboard shows your Agent ID and a downloadable tarqa-tunnel.config.json pre-filled with your workspace and agent details. Save it, then start the agent from source on the GPU machine.
BASH
cd packages/tunnel-agent
npm install
npm run build
node dist/src/cli.js start --config ~/.tarqa/tarqa-tunnel.config.json
3

Register a hosted model

Once the agent shows Online, click Register Model. Pick the agent, choose from its auto-detected local models (or enter one manually), and set the local endpoint — e.g. http://localhost:11434 for Ollama.
4

Select it in chat

The model appears in the chat model picker as hosted-<id>, tagged with a tunnel badge, and routes every request over the WebSocket tunnel to your machine and back.

Calling the tunneled model

Once online, a hosted model behaves like any other model in /v1/chat/completions — just use its hosted-<id> model ID (the ID shown in the dashboard, not the raw local model tag):

BASH
curl -X POST "https://api.tarqaai.com/api/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hosted-<your-model-id>",
    "messages": [{"role": "user", "content": "Hello from local GPU!"}]
  }'

Security

  • Outbound-only. The agent only opens outbound connections to TarqaAI — nothing listens for inbound traffic on your machine, so no port forwarding or firewall changes are required.
  • Paired at registration. Each agent is bound to your account by a certificate thumbprint set when you register it in the dashboard; the connection is rejected unless the agent presents a match.
  • Per-model visibility. Hosted models default to private. You can share one with your team or make it public to your whole workspace from the model's settings.
Who can use GPU tunnels

Available to Pro plan users and to anyone in a Team or Enterprise workspace. Local GPU inference has no TarqaAI token cost — use it for high-volume workloads, fine-tuned models, or data that has to stay on-premise.