Phase 2, data agent: library, change log, notifier, net, fs, clipboard, shell
Status 2026-11-18. Handlers: plugin-host/src/imports/ (files named below). Core change log and batch: core/src/changes/, core/src/migrations/change-log.ts. Tests: plugin-host/test/data/.
Design sources: library.md, lifecycle.md, phase2-core.md (“For the stage 2 agents”), plugin-api/wit/library.wit and lifecycle.wit.
What was built
Core: change log, event stream, atomic batch
The README decision “Zinc’s core gets a change log and event stream in this phase” is done. Additive and migration-safe: migration 5 (change-log) adds two tables and 22 triggers; it changes no existing row, and a library at schema 4 upgrades through the normal backup-then-migrate path (tested). An older build refuses a v5 library (LibraryTooNewError), as for any migration.
| What | Where |
|---|---|
change_log(seq, at, object, kind, key, library_id, origin, silent) and change_ctx(name, value), triggers on items, collections, collection_items, item_tags, attachments, saved_searches, annotations, fulltext |
core/src/migrations/change-log.ts, registered in migrations/list.ts |
Reading it: GET /api/changes?since=N&limit=&library=&wait=ms (long poll, at most 30 s), GET /api/changes/head, GET /api/changes/stream (server-sent events) |
core/src/changes/api.ts, log.ts; the router calls handleChangesApi first (one line in api.ts) |
POST /api/batch: the whole WIT change variant in one SQLite transaction, all or nothing, failure names the index |
core/src/changes/batch.ts |
Every write path is covered because the triggers are on the tables, not on the code that writes them: the HTTP routes, import, sync (rows get origin sync), the connector host, the MCP server, the Word add-in. Unlike sync_outbox the log is not de-duplicated and not suppressed during sync. Details:
- Objects and kinds:
itemadd / modify / trash / restore / delete / index (full text),collectionadd / modify / delete,collection-itemadd / remove ("<collection>-<item>"),item-tagadd / modify / remove ("<item>-<tag>"),fileadd / modify / delete (the attachments row),searchadd / modify / delete,annotationadd / modify / delete (the notifier delivers these asitem+other, becauseobject-typehas no annotation). origin:syncwhile sync holds itssuppressflag; else theoriginrow ofchange_ctx, whichPOST /api/batchsets for the length of its transaction (plugin:<id>); elseunknown. A rollback leaves no context row and no log row.silentis set byskipNotify.sincesemantics: a cursor is the lastseqseen. The page carrieshead,cursor(pass it next time), andreset: truewhen the cursor is ahead of the log (a different library file) or older than what is kept. The log keeps about 100,000 rows (pruned when it passes 200,000).- Batch change shapes are the WIT variant in jco’s JSON form (
{tag, val}, camelCase records); the full list is at the top ofbatch.ts. Caller keys (new-keys) are honoured for items, collections and saved searches (small optionalkeyparameter added tocreateItem,createCollection,createSavedSearch).erase-itemsneeds the item to be in the trash already. An erase removes imported attachment files from disk before the transaction ends, so a later failing change cannot bring them back: puterase-itemslast (documented inbatch.ts). - Edits to existing core files (all small):
db.ts(optionalkey),searches/store.ts(optionalkey),maintenance/api.ts(ownStorageexported),api.ts(one import and one array entry),migrations/list.ts, andcore/test/libraries.test.ts(it usedSCHEMA_HEAD - 1to mean “before migration 4”; it now says 3).
Plugin host handlers (plugin-host/src/imports/)
All are defineHandler modules. File names follow discoverHandlers (*.host.ts or *.zinc.ts; the brief said library.ts and so on, which would not be found).
| File | where |
Interfaces |
|---|---|---|
library.host.ts |
host | library |
attachments.host.ts |
host | attachments (+ resource file-writer) |
fulltext.host.ts, schema.host.ts, cite.host.ts, translate.host.ts, sync-control.host.ts, data-utils.host.ts |
host | the interface of the same name |
notifier.host.ts |
host | notifier (+ the handle subscription-guard, see below) |
net.host.ts |
host | net (+ resource body-reader); also exports fetchForHandler for other handlers |
fs.host.ts |
host | fs except the pick dialogs |
fs.zinc.ts |
zinc | fs.pick-open, pick-save, pick-directory |
clipboard.zinc.ts, shell.zinc.ts |
zinc | clipboard, shell |
data/ |
helpers: core.ts (HTTP to core, status to error mapping, mcpText), library-map.ts (query and summary mapping), utils.ts (dates, CSL, extra, identifiers), nethosts.ts (allowlist), fs-roots.ts (roots, grants, quota), notifier-hub.ts, pick-hook.ts, os-hooks.ts, perm.ts |
Permissions: every handler calls requirePermission(ctx, id). data/perm.ts re-exports it from src/permissions/ (the pkg agent’s helper, which exists now).
Where did clipboard and shell go? Zinc process. The clipboard belongs to the window system (focus check for clipboard.read, first-read confirmation), and opening a URL or file is something the app should be able to replace (GPUI has its own). Both take a replaceable backend (data/os-hooks.ts: setClipboardBackend, setShellBackend). Defaults: text-only clipboard through the platform tool, opener by rundll32 / open / xdg-open with an argv array (never a shell), and every confirmation denied.
Opening URLs. shell.open-url accepts http, https, mailto (permission shell.open-url), plus schemes the manifest lists under permissions.shell.schemes (the user confirms the first use of each, default denied). file:, javascript:, data:, vbscript: and a dozen more are refused always. A URL with a space, a control character, more than 4096 characters, or credentials is refused. open-file refuses programs and scripts by extension (.exe .bat .cmd .ps1 .js .lnk .sh .app ...) and reaches only the plugin’s data, temp and granted files, never bundled.
Notifier. One NotifierHub per host process runs one long poll on /api/changes while any subscription exists. Per subscription: events merged per (object, kind, library, origin) with their ids, flushed after max-delay-ms (default 50, at most 1000); one batch in flight; queue of 256 batches; when it overflows the oldest are dropped and a refresh event per affected type is delivered first; skip-own, types, kinds, priority (subscriptions are fed in priority order); silent rows are never delivered; a log reset becomes refresh events. Origins: plugin:<this plugin> is self, another plugin plugin(id), sync, user and connector pass through, writes with no context are unknown. notifier.trigger accepts only other events with options.name and delivers them to the subscriptions of this host process. Subscriptions end when the plugin is unloaded: subscribe makes an (unreturned) handle of the resource subscription-guard whose drop runs from HandlerTable.dropPlugin.
net. Allowlist from permissions.net.hosts (scheme://host[:port], *. wildcards, https assumed, * means any public host); private and loopback hosts only by exact entry; every redirect hop is checked again; a name that resolves to a private address is refused unless listed. Caps: 30 s default timeout (never above 10 min), 64 MiB default response (at most 512 MiB), 16 requests per send-many, 20 redirects. The host owns Host, Content-Length, Cookie (unless a jar), Proxy-*, Sec-*; default User-Agent: Zinc/0.1. Per-plugin cookie jars. retry policy, success-codes, cancel-key, basic credentials. download streams to a fs destination (data, temp or a writable grant) through a .part file and renames it at the end. send-streamed returns a body-reader resource. Errors are net-error variants (the WitError payload is built directly).
fs. The four roots as lifecycle.md designs: data (<dataRoot>/<plugin id>/, 512 MiB quota), bundled (read only), temp (<tempRoot>/<plugin id>/), granted(id). Paths: relative, /, no .., no drive letter, no empty or reserved names (CON, NUL, COM1 …), no \ : * ? " < > | or control characters, no trailing dot or space; a symbolic link or junction that leaves the root is refused. 256 MiB whole-file cap, read-range for more. Conflict modes fail, overwrite, rename-unique (name 2.ext), atomic writes through a temp file. Grants live in <dataRoot>/.grants/<plugin>.json, because the dialogs run in the Zinc process and the reads in the host process; both read and write the same file. A session grant is valid while its Zinc process is alive. A granted file has the path ""; a granted folder is browsed with relative paths. A read grant refuses writes. A plugin never gets a path back (file-grant has id, name, is-directory, size).
The pick dialogs are where: "zinc" and call a hook the ui agent fills: setFilePicker(fn) from data/pick-hook.ts. fn({pluginId, kind: "open" | "open-multiple" | "save" | "directory", options, suggestedName}) returns the chosen absolute paths (or undefined on cancel) and must refuse without a focused Zinc window. Default: unavailable (denied). Drops use addGrant(pluginId, {path, isDirectory, mode: "read", size}) from data/fs-roots.ts and hand the returned record (id, name, isDirectory, size) to the plugin as types.file-grant.
Reaching core. Reads call HTTP routes. Where core has an MCP tool but no HTTP route (get_fulltext), data/core.ts mcpText calls POST /mcp (stateless JSON-RPC). library.apply calls POST /api/batch with origin = plugin:<id>.
Run it
cd plugin-hostbun check # typecheckbun test test/data # this area: about 7 s (the e2e test builds a QuickJS plugin first)bun test # everythingcd ../core && bun test # core (the change log is exercised in plugin-host/test/data/changes.test.ts)The tests start a throwaway core from core’s own test fixture (core/test/fixture.ts: a temp library with a few items, an attachment, full text) with startServer({port: 0}). They never open data/library.sqlite or ports 23119 / 23120. core/vendor is a junction to the main checkout’s core/vendor (git-ignored; the sparse worktree lacked it, so core’s server and 160-odd core tests could not load): created once, nothing written through it.
Tests and result
| File | Tests | What it proves |
|---|---|---|
test/data/changes.test.ts |
11 | every write path is logged with the right object, kind, key and library; paging, reset, long poll, event stream; batch: linked creates with caller keys, origin tagging and context cleanup, rollback (no item, no log row), index of the failing change, expectedModified conflict, erase needs the trash, publications, saved searches, collection erase with items, silent rows, the CSRF guard; migration of a v4 library |
test/data/library.test.ts |
22 | library (permissions, query scopes, tags, text, ad-hoc search, expand, get-items, summaries, children, collections, tags, facets, tag colours, apply atomic with permission split and {index, error}, trash / restore / erase / empty-trash), attachments (info, ranges, caps, import, file-writer resource, view state, import from URL through the allowlist), fulltext, schema, cite, sync-control, data-utils |
test/data/platform.test.ts |
21 | net (allowlist, redirect hops, header ownership, success codes, size cap, timeout, cancel, retry, cookie jars, body-reader, download), fs (roots, traversal and reserved names, links, quota, grants, picker hook, drops), shell (URL and file validation, confirmations), clipboard, the notifier hub (merge, filters, priority, one in flight, overflow, reset, trigger) and the handler against a core |
test/data/e2e.test.ts |
5 | a built QuickJS plugin, through the real plugin host process, subscribes, queries items, receives a notifier batch for a write made by someone else (origin unknown), writes with library.apply and receives its own batch (origin self), and gets permission-denied for library.trash |
Result: bun test test/data 59 pass, 0 fail (about 7 s); bun check clean. Core: bun test in core/ 1356 pass, 5 fail in one full run under load (three 30,000-item timing tests and two translator tests that exceed their time limits when several agents share the machine); all five pass when run alone (test/api-perf.test.ts, test/searches/perf.test.ts, test/translate, test/update: 90 and 2 pass). Full plugin-host run: 124 tests, 1 fails: see “Needs from other areas”.
WIT rows now implemented
Implemented (some with a stated limit); “unavailable” rows return the WIT’s unavailable error and are listed under “Stubbed or left”.
library:list-libraries,new-keys,query-items,query-keys,count-items,facets(nocollection,contentType),get-items,get-summaries,get-children,list-collections(versionis 0),get-collection-items,list-tags,get-tag-colors,set-tag-colors(the colour list is global in core,libraryis ignored),rename-tag,delete-tags,delete-automatic-tags,list-search-conditions,list-saved-searches,get-saved-search,apply(all 19changecases),find-duplicates,merge-items,mark-not-duplicates,empty-trash.attachments:get-info,best-attachment,best-attachments,read-file,read-file-all,import-bytes,file-writer(constructor,append,finish, drop),import-from-url,link-file,link-url,replace-file(without a new name),rename-by-template,reveal-file,open-file-external,storage-of(one file at most),get-view-state,set-view-state,find-available-files,get-find-job,cancel-find-job.fulltext:get-content,get-index-state(indexed,unindexed,unavailable),search.schema.get-schema;cite:list-styles,install-style,remove-style,format,quick-copy,get-quick-copy-settings;translate:list-translators(import and export),import-text,export-items,web,web-select;sync-control:get-state,sync-now;data-utils: all 11 functions;notifier:subscribe,unsubscribe,triggerand the export callnotifier-handler.on-events.net:send,send-many,download,send-streamed,body-reader.read,body-reader.close,cancel,list-cookies,clear-cookie-jar,is-online(always true).fs:read,read-range,read-text,write,write-text,append,stat,list-dir,make-dir,remove,move,copy,list-grants,release-grant,pick-open,pick-save,pick-directory(the last three through the hook).clipboard:write,read.shell:open-url,open-file,reveal.
Stubbed or left
Where core lacks the feature the handler answers the WIT error (unavailable, or limit-exceeded and so on where that is the case) after the permission check.
| Call | What is missing |
|---|---|
library.find-by-relation |
no relation index in core (relations are JSON in items.data) |
attachments.rename-file, delete-file, relink-file |
no core route (core renames only by template; deleting a file but keeping the item; relinking) |
attachments.replace-file with a filename |
core replaces the bytes in place only |
attachments.get-info |
path (needs a route; get_file_path exists only as an MCP tool and returns text), charset, last-read are none |
attachments.storage-of |
counts the one stored file; core has no folder listing |
fulltext.set-content, reindex, get-queue-status |
no core routes; fulltext.source has no plugin value |
fulltext.get-content |
source is always extracted, indexed-at is the epoch (core does not return them over MCP) |
fulltext.search |
matched-in and snippets come from /api/search; the library.read permission is required as well |
cite.get-style-xml |
no route returns the XML of an installed style |
cite.format |
note-citation is built here from the item keys with local URIs; core has no note-citation output |
translate.lookup |
core’s add-by-identifier saves the items; a resolve-only route is needed |
translate.list-translators for web, search |
no listing route; import-text into a group library |
translate.web |
the network permission is checked on the page URL only; core fetches |
net.download report-progress |
net-events.on-download-progress is never called (the host cannot call the export per chunk cheaply; no plugin needs it yet) |
sync-control.get-state |
route is off or zotero-local; last-synced is core’s lastSync when present |
sync-control.sync-now |
runs in the background; the sync notifier events are not produced (the log has no sync rows) |
| Notifier | extra (per-id data such as changed fields) is empty; events for tag, trash, relation, setting, group, feed, share are not produced; skip-notify hides a batch from every subscriber, including the plugin’s own origin |
apply |
required on a search condition is ignored; groupStart / groupEnd and operators outside core’s eight are invalid-argument |
| Query | any-join ad-hoc search cannot be combined with tags, text or a collection scope in one query; a saved-search scope cannot be combined with an ad-hoc search; text with an ad-hoc search or several tags becomes an anyField contains condition |
clipboard default backend |
text only (no HTML or PNG) |
fs |
the temp root is emptied by clearTempRoot(), which the app must call at startup; nothing watches the quota between calls beyond a 1.5 s cache |
WIT changes
None made. For review (unchanged):
net.download-options.report-progresshas no matching host-side delivery:net-events.on-download-progressis an export the host would call during the transfer. Either drop the option or define who calls it.types.errorhas nounsupportedcase (also in phase2-core.md): “core lacks this” isunavailablewith a text, and a plugin cannot tell it from “offline”.library.apply-options.skip-notifysays the batch still reaches other plugins’ observers marked with the origin; the log has onesilentflag, so it hides the batch from everyone.notifier.object-typehas noannotation, but the log records annotation changes; they are delivered asitem/otherwithoptions {"name": "annotation", "change": ...}.
Needs from other areas
- plugin-host core (supervisor, registry):
HandlerContexthas no folder of the plugin package, sofsrootbundledcannot find it.fs-roots.tsreadsctx.pluginDirif it exists, elseconfigureFs({bundledDirFor}), else<ZN_PLUGINS_DIR>/<id>. Please addpluginDirto the context (it isspec.dirinhost-main.tsand the supervisor). - plugin-host core, test:
test/supervisor.test.ts“WIT resources” now fails (the only failure in the full run). It registers stubzinc:plugin/attachments(file-writer) andzinc:plugin/net(send-streamed,body-reader) aswhere: "zinc"handlers; the realwhere: "host"handlers added here win the routing. Move the stubs into a host module (hostHandlerModules, whose entries override same-named functions) or point the test at other interfaces. - app wiring (gpui-ui / whoever owns
platform.ts):configureFs({dataRoot, tempRoot})andclearTempRoot()at startup;setFilePicker,setClipboardBackend(windowFocused,confirmRead),setShellBackend(confirmScheme, and the GPUI opener if wanted); passcoreUrl. - ui and extend agents: a dropped file becomes a grant with
addGrant(pluginId, {path, isDirectory, mode: "read", size}); return its record astypes.file-grant. - core features still missing (the same list as library.md, minus the change log and the batch route, which exist now): relation index, collection trash,
lastRead, charset andsource = pluginfor full text, a reindex route, a resolve-only lookup route, file routes (rename to a name, delete file, relink, path), a style XML route, note-citation output, group-library import, search compiler coverage (groups,required, number fields,lastRead, annotation conditions,retracted), sync events in the log. - mirror of the permission ids: nothing new; every id used is in
manifest.schema.json(library.*,clipboard.*,shell.*).permissions.shell.schemesis read from the manifest directly.