Home / Work / Coding Agent Server on a Free GPU

Coding Agent Server on a Free GPU

Runs Qwen2.5-Coder-7B on a Colab GPU with llama.cpp and exposes it through a token-protected private tunnel, so local coding tools (Aider, Continue, Cline) get a fast open-weight model without paid APIs.

Status
Working notebook
Repository
Private
Tech Stack
Qwen2.5-Coder-7B-Instruct (Q4_K_M GGUF), llama-cpp-python with CUDA, FastAPI, Google Colab GPU, private tunnel

Overview

A Colab notebook that turns a free notebook GPU into a private coding-model server for local tools such as Aider, Continue and Cline.

The Problem

  • A capable coding model needs a strong GPU that a student laptop doesn't have.
  • Paid coding APIs add up quickly during heavy use.

How It Works

  1. Check the GPU runtime (L4 / A100 / T4).
  2. Build llama-cpp-python with CUDA support.
  3. Download the 4.7 GB GGUF model once to Google Drive so later sessions skip it.
  4. Start a FastAPI server with an OpenAI-style endpoint, protected by a bearer token.
  5. Open a private tunnel and point local tools at the tunnel URL.

Highlights

  • Existing tools work unchanged because the endpoint follows the OpenAI format.
  • New tunnel URL and token every session keep it private.
  • Restoring the server in a new session takes three cells.