jllmproxy

A local, auth-less Ollama-compatible server that proxies to any OpenAI-compatible provider (DigitalOcean GenAI, OpenAI, Groq, OpenRouter, local vLLM, …). Pure Java 21, zero dependencies.

What it does

Tools built for Ollama (IDE plugins, CLIs, apps) talk to localhost:11434 using the Ollama API. jllmproxy listens there and forwards every call to your configured cloud provider, translating the request and the response in both directions — including streaming. Your API key stays local; nothing else is required to use it.

1. Get it — Docker

docker run

docker run -d --name jllmproxy -p 11434:11434 \
  -v $PWD/jllmproxy.json:/app/jllmproxy.json:ro \
  -v $PWD/data:/app/data \
  ghcr.io/arons/jllmproxy:latest

docker compose

Create a compose.yaml file:

services:
  jllmproxy:
    image: ghcr.io/arons/jllmproxy:latest
    ports:
      - "11434:11434"
    volumes:
      - ./jllmproxy.json:/app/jllmproxy.json:ro
      - ./data:/app/data
    restart: unless-stopped

Then start it with:

docker compose up -d

2. Configure

Create a jllmproxy.json file and mount it at /app/jllmproxy.json as shown above:

{
  "providers": [
    {
      "name": "DigitalOcean",
      "url": "https://inference.do-ai.run/v1",
      "key": "doo_v1_your-api-key-here"
    }
  ]
}

Only the first provider entry is used (v1 limitation). The url must be the OpenAI-compatible base URL, including the /v1 part when the provider uses it.

Pricing (optional)

Add a pricing array to track costs per request (USD per million tokens):

"pricing": [
  { "providers": ["DigitalOcean"], "input_per_million": 0.5, "output_per_million": 1.5 },
  { "providers": ["OpenAI"], "model": "gpt-4", "input_per_million": 30, "output_per_million": 60 }
]

An empty providers list matches all providers; a rule without model is the fallback for models without an exact match.

3. Configure the run

Extra arguments can be appended to the docker run command (after the image name). Defaults:

usage: jllmproxy [--config <file>] [--port <n>] [--host <addr>] [--timeout-seconds <n>] [--stats-retention-days <n>]

options:
  --config                path to config file (default: ./jllmproxy.json)
  --port                  local port (default: 11434)
  --host                  bind address (default: 127.0.0.1)
  --timeout-seconds       upstream request timeout (default: 300)
  --stats-file            path of the stats jsonl file (default: ./stats.jsonl);
                          point it inside a mounted directory when running in Docker
  --stats-retention-days  delete stats older than N days, 0 = keep forever (default: 90)

The server binds to 127.0.0.1 by default (local only, no auth). If you expose it on your network with --host 0.0.0.0, anyone who can reach the port can use your provider key — use a firewall or a reverse proxy.

4. Verify

curl http://localhost:11434/api/version
curl http://localhost:11434/api/tags          # lists provider models
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.3-70b-instruct",
  "messages": [{"role":"user","content":"hello"}],
  "stream": true
}'

Then point any Ollama client (e.g. the GitHub Copilot alternative plugin, ollama-compatible IDE extensions) at http://localhost:11434 and select a model from your provider.

Supported endpoints

Endpoint	Method	Description
`/api/chat`	POST	Chat, streaming (`"stream": true`) and non-streaming
`/api/generate`	POST	Plain prompt completion
`/api/embeddings` / `/api/embed`	POST	Embeddings
`/api/tags`	GET	List models (from the provider)
`/api/show`	POST	Minimal model info
`/api/version`	GET	Version (reports 0.6.4 for client compatibility)
`/v1/chat/completions`, `/v1/models`, `/v1/completions`, `/v1/embeddings`	POST/GET	Passthrough in OpenAI format (auth injected)
`/api/stats`	GET	Usage & cost aggregates (totals, per provider/model, daily) and raw entries

Usage statistics

Every proxied request is recorded to stats.jsonl in the working directory (one JSON line per request: timestamp, provider, model, endpoint, tokens, duration, cost). Open http://localhost:11434/dashboard.html for the dashboard: KPI cards, daily charts (requests, tokens, cost), provider/model filters and breakdown tables. Auto-refreshes every 30 seconds.

By default the file is stats.jsonl in the working directory (/app/data/stats.jsonl in the Docker image, see --stats-file). Mount the /app/data directory as shown in the Docker examples above — the app creates the file itself on first run. Do not mount a single stats.jsonl file path: if the host file does not exist, Docker creates a directory at the mount point instead.

Ollama options mapping

When a client sends Ollama options, they are translated to OpenAI parameters:

Ollama	OpenAI
`temperature`	`temperature`
`top_p`	`top_p`
`num_predict`	`max_tokens`
`stop`	`stop`
`seed`	`seed`

Troubleshooting

Client says the version is bad — /api/version reports 0.6.4; if your client requires a higher minimum, open an issue.
Model list is empty — check the provider url (must end at /v1 for OpenAI-style APIs) and that the key is valid; upstream errors are logged to stderr.
404 on /api/show — the requested model must exist in the provider's model list; run curl http://localhost:11434/api/tags to see valid names.
Debugging — the server logs every request body and upstream call (URI, status, error bodies) to stderr.

Docker image: ghcr.io/arons/jllmproxy