simbarag/docs/immich-mcp/design.md

Immich MCP: technical design

Status: proposed · Owner: Ryan · Date: 2026-10-13

1. Summary

Give Simba (and Claude Code) a small set of tools for the household Immich library so a request like "my Maine trip, here are the photos" turns into either a link to the album that already exists or a confirmed, one-step album creation. The tools live in a stdio MCP server we write and keep in this repository (immich_mcp/). raggr loads it the same way it loads Homebox and BookStack; Claude Code can register the same server with claude mcp add.

Work ships in two phases:

• Phase 1, read-only. Find albums, search photos by date, place and person, CLIP search, and spot photos missing from an existing album.
• Phase 2, writes. Create an album and add photos to an album, behind a server-side switch (IMMICH_ENABLE_WRITES=true) and a count check that stops the model from writing something the user did not confirm.

There are no delete, archive, trash, metadata-edit or sharing tools in either phase.

2. Goals and non-goals

Goals

1. "Do I already have an album for X?" returns the album name, photo count, date span and a clickable link, in one tool call.
2. "Photos from my Maine trip" with no album returns a summary the user can check (how many, which days, which towns, a few sample filenames) without moving hundreds of asset IDs through the model's context.
3. "Make an album for it" creates the album with exactly the photos the user saw in that summary, or does nothing and explains why.
4. "Is the Maine album complete?" lists photos taken during the album's dates that are not in it, such as camera imports or a partner's phone, and can add them after confirmation.
5. The same server works from Claude Code on the laptop.
6. Tool results stay small enough for the local Qwen model's 32k context: each result is at most about 2,500 characters, and all six tool schemas together cost at most about 1,500 tokens.

Non-goals

• Deleting, trashing, archiving, hiding, or editing assets or albums. Removing photos from an album is out too.
• Showing images inline in chat. The desktop client's photo artifact (docs/desktop-client.md) can build on this later using the asset links these tools return.
• Shared links, inviting album users, face naming, tagging.
• Replacing scripts/index_immich.py, the one-off script that describes Simba photos into the vector store. It is left untouched; retiring it is an open question (§13).
• Non-admin access. Photos are household-private, so like Homebox and BookStack these tools exist only on the admin agent.

3. Example conversations (acceptance scenarios)

These double as the manual verification script for the final ticket.

A. The album already exists

User: my maine trip, here are the photos Simba → immich_find_albums(query="maine") → [{"name": "Maine 2026", "asset_count": 412, "start": "2026-08-02", "end": "2026-08-09", "url": "https://photos.example/albums/…"}] Simba: You already have Maine 2026: 412 photos, Aug 2–9. [link]

B. No album yet

User: find my maine trip photos immich_find_albums("maine") → [] immich_search_photos(state="Maine") → total 398; clusters: Aug 2–9 2026 (391), Jul 14 2023 (7); top places Bar Harbor, Portland Simba: I found 391 photos from Aug 2–9, 2026, mostly Bar Harbor and Portland, plus 7 from a day in July 2023. Want an album for the August trip? User: yes, call it Maine 2026 immich_search_photos(taken_after="2026-08-02", taken_before="2026-08-09") → total 455 (this adds the drive up and photos with no GPS) Simba: Counting everything from Aug 2–9, not only photos tagged in Maine, gives 455 (64 have no location or were taken on the way). Create Maine 2026 with all 455? User: yes immich_create_album(name="Maine 2026", criteria={taken_after, taken_before}, expected_count=455) → created, 455 added, link

C. The album exists but is incomplete

User: is the maine album missing anything? immich_find_albums("maine") → Maine 2026, Aug 2–9 immich_album_gaps(album_id=…) → 37 photos taken Aug 1–10 are not in the album (mostly Aug 5, Canon EOS R6) Simba: 37 photos from those days aren't in it, mostly camera shots from Aug 5. Add them? immich_add_to_album(album_id, criteria={taken_after: 2026-08-01, taken_before: 2026-08-10}, expected_new_count=37)

D. Content search

User: pictures of simba in the snow immich_smart_search(query="orange cat in snow", person="Simba") → top 10, with dates and links

4. Approach and alternatives

Option	Pros	Cons
A. Our own stdio MCP server in this repo (chosen)	One implementation serves raggr and Claude Code; we control tool names, descriptions and output size; write safety is enforced in code we own; no new runtime dependency (mcp 1.30 is already locked through langchain-mcp-adapters); fits the Homebox/BookStack loading pattern	We maintain it (≈500 lines plus tests)
B. Third-party Immich MCP server (several exist: lidless-labs immich-mcp, barryw/ImmichMCP, okopok/immich-mcp, …), pinned and allowlisted like Homebox/BookStack	No code to write	Not evaluated in depth. Generic API-wrapping servers return per-asset JSON (a 400-photo trip won't fit in 32k) and add assets by ID list (the model would have to copy hundreds of UUIDs). Our key, which needs album write permissions, would go to a young third-party package. Tool descriptions we can't edit cost context.
C. Plain LangChain tools in utils/immich_service.py (like Mealie/YNAB)	Simplest wiring	Simba only; Claude Code would need a second implementation

Option A keeps option C's control and adds reuse from Claude Code. Most of the work goes into compact summaries and write guards, and no off-the-shelf server gives us either.

5. Architecture

            ┌───────────── raggr (admin agent) ─────────────┐
            │ app.py lifespan                                │
            │   load_immich_tools() ─┐                       │
            │                        │ langchain-mcp-adapters│
            │   wrap_admin_only()    │ (new stdio session    │
            │   prefix immich_*      │  per tool call)       │
            └────────────────────────┼───────────────────────┘
                                     │ stdio
 Claude Code ── claude mcp add ──────┤
                                     ▼
            ┌──────── immich_mcp (python -m immich_mcp) ─────┐
            │ server.py   FastMCP tools (6)                  │
            │ criteria.py PhotoCriteria → Immich search body │
            │ summarize.py clusters / places / samples       │
            │ client.py   ImmichClient (httpx, method+path   │
            │             allowlist, no DELETE)              │
            └────────────────────────┬───────────────────────┘
                                     │ HTTPS, x-api-key
                                     ▼
                              Immich server (/api)

Process lifetime. MultiServerMCPClient.get_tools() without a session creates tools that open a new stdio session for every call (langchain_mcp_adapters/tools.py). The server therefore must be stateless between calls: no in-memory selections, caches or tokens. Write tools take the same criteria as the preview and resolve them again, and the expected_count check (§8) catches any drift. Starting a process costs roughly 0.5–1 s per call. That is acceptable for this use, and IMM-04 measures it.

6. Package layout

New top-level package (copied into the image by a new COPY immich_mcp ./immich_mcp line, because the Dockerfile copies directories explicitly):

immich_mcp/
  __init__.py        # docstring only
  __main__.py        # `python -m immich_mcp` → server.main()
  config.py          # env parsing (§10), ImmichConfig dataclass
  client.py          # ImmichClient: async httpx, allowlisted endpoints only
  criteria.py        # PhotoCriteria (pydantic), date/time-zone handling, request bodies
  summarize.py       # pure functions: clusters, places, samples, histogram
  server.py          # FastMCP("immich"), tool definitions, JSON output shaping
tests/unit/
  test_immich_client.py
  test_immich_criteria.py
  test_immich_summarize.py
  test_immich_server.py      # tools via an in-memory MCP session
  test_immich_mcp.py         # raggr loader + admin gate (mirrors test_homebox_mcp.py)

raggr-side changes:

• utils/mcp_tools.py: load_stdio_tools(..., args: list[str] | None = None, cwd: str | None = None). Today args is hard-coded to [] and no cwd is passed. StdioConnection already supports both. Homebox and BookStack keep the defaults.
• utils/immich_mcp.py: IMMICH_TOOL_ALLOWLIST, select_immich_tools, load_immich_tools() (mirrors utils/homebox_mcp.py). The command defaults to sys.executable (the app's venv interpreter) with args=["-m", "immich_mcp"] and cwd set to the repo root (Path(__file__).resolve().parents[1]), so -m finds the package whatever the parent's working directory is. The subprocess env is only the IMMICH_* variables plus PATH (load_stdio_tools behaviour), so no other secrets leak into it. IMMICH_MCP_COMMAND overrides the command for debugging. Note the two names: utils/immich_mcp.py is raggr's loader, and top-level immich_mcp/ is the server. Neither imports the other.
• app.py: add await load_immich_tools() to admin_tools.
• blueprints/conversation/tool_routing.py: add a "photos" group with the terms immich photo photos picture pictures pic pics album albums trip trips vacation, map the prefix ("immich_", "photos"), and add "immich" to the admin prefix tuple in ToolRouter.__init__.
• blueprints/conversation/tool_labels.py: ("immich_", "Looking through photos…").
• raggr-frontend/src/components/ActivityTimeline.tsx: ["immich", Images] (lucide).
• blueprints/conversation/prompts.py: one sentence in INTEGRATIONS (§9).
• blueprints/integrations/__init__.py: admin row ("Immich", ("IMMICH_URL", "IMMICH_API_KEY")).
• config/validation.py: IMMICH_URL and IMMICH_PUBLIC_URL in the URL list; IMMICH_ENABLE_WRITES in the true/false list.
• .env.example, docker-compose.yml: env vars (§10). IMMICH_URL and IMMICH_API_KEY already exist in .env.example for the indexer, so reuse them.

7. Tool specifications

Every tool returns compact JSON (json.dumps(..., separators=(",", ":"), ensure_ascii=False)), at most MAX_RESULT_CHARS = 2500. Dates in output are calendar dates (YYYY-MM-DD): a photo's date is its own local day (§7.4), and album start/end (UTC timestamps in Immich) are converted to IMMICH_TIMEZONE. If a result would go over the cap, lists are cut down (samples first, then places, then clusters) and "trimmed": true is set; the result is never cut mid-JSON. Links are built from IMMICH_PUBLIC_URL: albums at /albums/{id}, photos at /photos/{id}. Errors come back as {"error": "<plain sentence>"} and never raise, so the model can tell the user what went wrong.

7.1 Shared input: PhotoCriteria

Field	Type	Meaning
taken_after	YYYY-MM-DD	inclusive, local day start
taken_before	YYYY-MM-DD	inclusive, local day end (converted to the next day 00:00 local, exclusive)
state	str	reverse-geocoded state/province, e.g. "Maine"
city	str	e.g. "Bar Harbor"
country	str	e.g. "United States of America"
person	str	Immich person name, resolved via GET /search/person?name=; several matches → error that lists them
media	"photo" | "video" | "all"	default "all"
favorites_only	bool	default false

The fields are flat tool parameters, never a nested object. Every tool that takes criteria declares these eight parameters directly. A nested criteria object would produce $defs/$ref in the tool schema, which the local model's llama-server grammar handles poorly, and it costs more schema tokens. In the examples in §3, criteria={…} is shorthand for passing these parameters.

For immich_search_photos and both write tools, at least one field must be set, so a request can never match the whole library. (immich_smart_search is already scoped by its query.) taken_before < taken_after is an error. Place names are matched case-insensitively: before searching, the server looks the value up in GET /search/suggestions?type=state|city|country and uses Immich's exact spelling. If nothing matches, it returns the three closest suggestions (difflib.get_close_matches) instead of searching.

7.2 Phase 1: read-only

immich_find_albums(query: str = "", limit: int = 10) Lists albums (GET /albums, both owned and shared-with-me) and ranks them by case-insensitive substring match, then difflib ratio ≥ 0.6. An empty query returns the most recently updated. Output per album: id, name, asset_count, start, end, shared, owned (false for albums shared with the key's user; adding to those may fail), url. Why match locally: Immich's name filter is an exact match, and "maine" should find "Maine 2026" and "Acadia / Maine".

immich_search_photos(criteria…, samples: int = 5)

1. POST /search/statistics gets the exact total.
2. Pages POST /search/metadata (size 1000, withExif: true) up to MAX_SCAN = 5000 assets.
3. Returns total, scanned, truncated, first, last, clusters (§7.4, at most 8), top_places (city/state with counts, at most 5), no_location (count with no city/state), by_media (photo/video counts), samples (evenly spaced across the scan: date, city, filename, url), and criteria (the normalized criteria, echoed so the model can pass it straight to a write tool).

immich_smart_search(query: str, criteria… (all optional), limit: int = 10) POST /search/smart (CLIP), with the same filters. Returns the top limit (at most 25) results in rank order: date, city, filename, url. There is no total, because smart search ranks results rather than filtering them. Use it to find content ("in the snow", "sunset at the beach"), not to collect a trip.

immich_album_gaps(album_id: str, pad_days: int = 1) Loads the album (GET /albums/{id} for startDate/endDate). It then collects the IDs already in the album (POST /search/metadata with albumIds=[id], paged) and the photos in [start − pad, end + pad] (local dates), and returns the search summary from §7.2 computed over the photos not in the album. It also returns criteria for that window and missing_count, ready for immich_add_to_album. Returns an error if the album has no dates (it's empty). missing_count must be computed exactly as immich_add_to_album computes expected_new_count; both go through one shared helper, resolve_missing(album_id, criteria). If the window has more than MAX_SCAN photos, missing_count is marked truncated, and a later add with that count fails the check. That's the safe direction.

7.3 Phase 2: writes (only registered when IMMICH_ENABLE_WRITES=true)

immich_create_album(name: str, expected_count: int, <criteria fields>, description: str = "")

1. If any album already has this exact name (case-insensitive), return an error that names it and suggests immich_add_to_album. Never create a duplicate.
2. Resolve the criteria to asset IDs (full paging, no MAX_SCAN limit, but refuse if total > MAX_WRITE = 2000).
3. If len(ids) != expected_count, return {"error": …, "actual_count": n} and write nothing.
4. POST /albums with albumName, description, and assetIds for the first 500. Add the rest with PUT /albums/{id}/assets in chunks of 500.
5. Return id, name, added, url.

immich_add_to_album(album_id: str, expected_new_count: int, <criteria fields>)

1. Resolve the criteria to IDs (same MAX_WRITE limit) and subtract the IDs already in the album.
2. If the count of new IDs differs from expected_new_count, return an error and write nothing.
3. PUT /albums/{id}/assets in chunks of 500. Count per-ID results from BulkIdResponseDto: success, duplicate, other errors (with reasons).
4. Return added, already_present, failed (at most 5 reasons), url.

7.4 Trip clustering (summarize.cluster_by_day)

Group scanned assets by the photographer's local day: the date part of localDateTime, taken as-is. Immich stores it as a "timezone-agnostic timestamp" of the wall-clock time, so converting it would shift days. If localDateTime is missing, fall back to fileCreatedAt converted to IMMICH_TIMEZONE. Sort the days and start a new cluster wherever the gap between consecutive days is more than gap_days = 2 (one quiet day in the middle of a trip does not split it). Each cluster reports start, end, count, and its top city/state. Clusters are sorted by count descending; keep 8. This is how "my Maine trip" becomes "Aug 2–9, 2026" rather than "every photo ever taken in Maine". It's a pure function, tested with hand-built fixtures.

8. Write safety

These layers stack; any one of them blocks an unwanted write.

1. Not registered by default. Write tools are defined only when IMMICH_ENABLE_WRITES=true. Phase 1 can be deployed and used before anyone decides to allow writes.
2. Admin-only. The tools are compiled only into app.admin_agent, and each call re-checks is_admin through wrap_admin_only (utils/mcp_tools.py).
3. Confirmed count. expected_count / expected_new_count must equal the number of photos the server resolves at write time. The model can only know the number by having shown a search result, and the prompt (§9) tells it to get a yes from the user first. If the library changed in between, the check fails.
4. Bounded size. MAX_WRITE = 2000 per call.
5. No destructive endpoints in the client. ImmichClient exposes named methods only, and its request helper accepts only the methods and path patterns in ALLOWED_ENDPOINTS. A unit test asserts that no DELETE, PATCH, or /trash, /assets (bulk) write is reachable.
6. A scoped API key. Immich API keys carry permissions. The key needs only album.read, asset.read, asset.statistics, person.read, and for Phase 2 album.create and albumAsset.create. Do not give it all.
7. No duplicate names. See immich_create_album step 1.
8. Writes add and never remove. At worst a write adds wrong photos to an album. That is fixable in the Immich UI and loses no data.

9. Prompt and routing

Append to INTEGRATIONS in blueprints/conversation/prompts.py:

If immich_* tools are available, use them for photos and albums: call immich_find_albums before searching or creating; for a trip, search by place, read the date clusters, then search the whole date range so photos without GPS are included; show the count and dates and get an explicit yes before immich_create_album or immich_add_to_album, passing the confirmed count.

Routing: the "photos" group (§6) makes these tools discoverable in TOOL_ROUTING_MODE=dynamic. In static mode they are always bound for admins, like Homebox.

10. Configuration

Variable	Required	Default	Notes
IMMICH_URL	yes	–	Base URL the server uses (LAN/compose network). Already in .env.example.
IMMICH_API_KEY	yes	–	Scoped key (§8.6). Already in .env.example.
IMMICH_PUBLIC_URL	no	IMMICH_URL	Base for links shown to the user.
IMMICH_TIMEZONE	no	TZ, then America/New_York	Local-day interpretation (§7.1, §7.4).
IMMICH_ENABLE_WRITES	no	false	Registers the Phase 2 tools.
IMMICH_MCP_COMMAND	no	sys.executable -m immich_mcp	Override for debugging.

All of them go in .env, .env.example and the raggr service environment in docker-compose.yml (project rule). If IMMICH_URL or IMMICH_API_KEY is unset, load_immich_tools() logs at info level and returns [], like Homebox.

Claude Code registration (documented in IMM-07):

claude mcp add immich \
  -e IMMICH_URL=http://192.168.1.5:2283 -e IMMICH_API_KEY=… \
  -e IMMICH_PUBLIC_URL=https://photos.… -e IMMICH_ENABLE_WRITES=true \
  -- uv run --directory ~/Programs/raggr python -m immich_mcp

11. Immich API compatibility

Verified against the upstream OpenAPI spec at 3.3.0-rc.0:

• Endpoints used: GET /server/version, GET /albums, GET /albums/{id}, POST /albums, PUT /albums/{id}/assets, POST /search/metadata, POST /search/smart, POST /search/statistics, GET /search/suggestions, GET /search/person.
• Deprecation. In v3.2.0 Immich deprecated the flat search fields (takenAfter, takenBefore, state, city, country, personIds, albumIds, type, page, …) in favour of a nested filter object (SearchFilter, with DateFilter {gte, lt}, StringFilterNullable {eq, in}, IdsFilter {any, all, none}) and cursor paging. The flat fields still work in 3.3, and they are the only option before 3.2. Mapping: takenAfter/takenBefore → filter.takenAt {gte, lt}, state/city/country → {eq}, personIds → {all}, type → {eq}, isFavorite → {eq: true}, albumIds → {any}, and in album_gaps, {none}.
• Decision. Build request bodies in a single function, criteria.to_search_body(criteria, version). Use the flat fields below 3.2 and filter/cursor from 3.2 on, choosing by GET /server/version (read once per process; a process lives for one call, so this is one extra request per call, or IMMICH_API_STYLE=legacy|filter to skip it). Golden-body tests pin both shapes. The filter style also gives album_gaps a server-side albumIds: {none: [id]}, so it doesn't need to diff locally.
• Production's Immich version is not yet known (it was unreachable from the design machine). IMM-01 records it. If it is below 2.0, stop and revisit this design.

12. Testing

• Pure units: criteria validation, local-day to UTC conversion (DST edges, inclusive taken_before), request-body golden tests for both API styles, clustering fixtures (single trip, quiet day inside, two trips, empty), sample spacing, output caps (a large fixture stays at most 2500 chars).
• Client: httpx.MockTransport. Checks the endpoint allowlist (DELETE raises), paging stops at MAX_SCAN, auth header, and that HTTP errors map to readable error strings.
• Tools: call the FastMCP server in-process with mcp.shared.memory.create_connected_server_and_client_session against the mock transport. Covers scenarios A–D from §3, the expected_count mismatch (no write happens), the duplicate-name refusal, write tools absent when the flag is off, and the schema token budget (sum of len(json.dumps(schema)) / 4 is at most 1500).
• raggr wiring: mirror tests/unit/test_homebox_mcp.py (allowlist, admin gate, missing env returns [], prefix), plus test_tool_routing.py cases for the photos group and admin hiding.
• Live (manual, IMM-08): scenarios A–D against production with a scoped key.

13. Open questions

1. Production Immich version and whether /search/statistics and the filter style are available (IMM-01).
2. Partner photos. Immich search covers partners with "show in timeline" enabled. Can the key's user add a partner's asset to their own album? If not, add returns per-ID errors, which we report. Verify in IMM-01.
3. Time zone of typed dates. Clustering uses each photo's own local day (§7.4), but taken_after/taken_before filter on UTC takenAt, interpreted in IMMICH_TIMEZONE. For a trip three hours west, a photo at 11 pm local on the last day falls outside the window. album_gaps pads by a day, and users can widen a range. A precise fix (filtering on localDateTime) is not in the API. Accepted for now.
4. Retire scripts/index_immich.py? Once smart search is live, the vector-store copy of Simba photo descriptions may no longer be needed. Decide after Phase 1 has run for a while.
5. Sharing. Should "send the Maine album to Mom" create a shared link? Not now; it would need sharedLink.create and a separate design.

14. Rollout

1. Merge Phase 1 with IMMICH_ENABLE_WRITES unset. Deploy and run scenarios A, B (read part), C (read part) and D.
2. Create the Phase 2 key permissions, set IMMICH_ENABLE_WRITES=true, redeploy, and run B and C end to end on a throwaway album name. Delete the test album by hand in Immich.
3. Register the server in Claude Code.