Run the Obsidian sync off the event loop and delete chunks by containment #84

2026-10-13T03:41:37-04:00

ryan commented

2026-10-13 03:41:37 -04:00

The periodic Obsidian sync (OBSIDIAN_CONTINUOUS_SYNC, every 60s by default) walked the vault, statted and read files, and ran synchronous SQLAlchemy queries and deletes directly on the event loop. While it ran, every active chat stream froze.

The vault walk, file reads, cache load and deletes run in asyncio.to_thread. Embedding stays async.
Deletes use JSONB containment (cmetadata @> CAST(:filter AS jsonb)), which langchain_postgres's ix_cmetadata_gin index (jsonb_path_ops) supports. The old cmetadata->>key = value couldn't use it. A sync pass deletes all stale files in one transaction.
Deleted files leave the index cache only after their chunks are deleted, so a failed delete is retried on the next sync instead of leaving orphaned chunks searchable.

Not done, deliberately:

There's no vector index on embedding, so similarity search is a brute-force scan. That's fine up to roughly tens of thousands of chunks; consider HNSW past ~100k (needs a migration).
The admin full-reindex path (index_obsidian_documents) still walks the vault on the loop. It's manual, not periodic.

Review: Sonnet worker plus two review rounds. Round 1 found the cache-before-delete ordering bug, which now has a regression test, and asked for the GIN and HNSW findings. Round 2 was clean.

558 unit tests, Ruff.

🤖 Generated with Claude Code

The periodic Obsidian sync (`OBSIDIAN_CONTINUOUS_SYNC`, every 60s by default) walked the vault, statted and read files, and ran synchronous SQLAlchemy queries and deletes directly on the event loop. While it ran, every active chat stream froze. - The vault walk, file reads, cache load and deletes run in `asyncio.to_thread`. Embedding stays async. - Deletes use JSONB containment (`cmetadata @> CAST(:filter AS jsonb)`), which langchain_postgres's `ix_cmetadata_gin` index (`jsonb_path_ops`) supports. The old `cmetadata->>key = value` couldn't use it. A sync pass deletes all stale files in one transaction. - Deleted files leave the index cache only **after** their chunks are deleted, so a failed delete is retried on the next sync instead of leaving orphaned chunks searchable. **Not done, deliberately:** - There's no vector index on `embedding`, so similarity search is a brute-force scan. That's fine up to roughly tens of thousands of chunks; consider HNSW past ~100k (needs a migration). - The admin full-reindex path (`index_obsidian_documents`) still walks the vault on the loop. It's manual, not periodic. **Review:** Sonnet worker plus two review rounds. Round 1 found the cache-before-delete ordering bug, which now has a regression test, and asked for the GIN and HNSW findings. Round 2 was clean. 558 unit tests, Ruff. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ryan added 1 commit 2026-10-13 03:41:37 -04:00

Run the Obsidian sync off the event loop and delete chunks by containment

CI / backend (pull_request) Successful in 42s

Details

CI / frontend (pull_request) Failing after 7s

Details

CI / backend (push) Successful in 41s

Details

CI / frontend (push) Failing after 7s

Details

e16ade4725

The periodic sync walked the vault, read files and ran synchronous database
deletes on the event loop, freezing active chat streams while it ran. Those
blocking steps now run in threads. Deletes use JSONB containment, which the
cmetadata GIN index (jsonb_path_ops) supports, and are batched into one
transaction per pass. Deleted files leave the index cache only after their
chunks are deleted, so a failed delete is retried on the next sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryan merged commit 0a50557d3d into main

2026-10-13 03:41:38 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ryan/simbarag#84