n8n content engine: RAG ingestion, scheduled generation, quality gates (2 instances)
What we need
One of two parallel tracks on a fully documented content-automation project. A formal requirements document with numbered requirements and an acceptance matrix is shared with shortlisted experts. This posting is the n8n engine track; a separate posting covers the Python / GitHub Actions delivery track. Bid on one or both. CONTEXT A client runs two small businesses in different sectors (one regulated). Each gets its own instance of the same engine: identical workflows, prompts, schema and gate code, differing only in configuration. Everything runs on accounts the client owns and pays for (n8n Cloud, Supabase, Anthropic API, GitHub). No vendor-hosted infrastructure; the vendor holds no production credential after handover. The engine delivers drafts only; a human publishes. SCOPE 1) Ingestion in n8n - Source registry as database rows, not nodes: adding or disabling a source requires no workflow edit. - Kinds: RSS/Atom, REST and GraphQL APIs (JSON, XML, CSV, text, HTML), cloud-folder PDF/DOCX, social sources. - Per-source config: extraction rules (JSONPath, XPath/CSS, CSV columns), pagination (cursor, page, offset), auth (API key, bearer, OAuth2 with refresh), fetch interval, trust tier, authority tier, failure counters. - Social adapters through official APIs only: X, Reddit, YouTube (uploads and captions), LinkedIn company page, Bluesky, Mastodon, Telegram public channels. No scraping, no logged-in surfaces, no writes to any platform. A new platform is a new adapter, not a pipeline change. - Polite fetching: per-host throttling, 429 back-off, content-type checks, boilerplate stripping, configurable truncation. - Sync of the client's own published posts and sent campaigns back into the store, re-chunked when modified. 2) Retrieval store: specification, not build You write the spec (tables, columns, indexes, embedding model and dimension, distance measure, query shapes). The internal team builds it in Supabase Postgres with pgvector; you test against their instance. Targets: HNSW-indexable dimension, sub-second queries at 100k chunks. 3) Generation pipeline: n8n + Anthropic API - Three content types (news, evergreen, newsletter issue) through one shared pipeline parameterised by type, with three thin wrapper workflows. - Schedules are configuration. News can instead fire on a significance trigger: hourly triage, a classification step, a minimum number of distinct citable sources, configurable caps. A check with no qualifying cluster ends silently but is logged. - Stages: title-embedding clustering and ranking, look-back against the client's own published work, grounding retrieval with a stand-down floor, outline call, section-by-section drafting with the voice guide and an auto-selected exemplar passage, assembly, deterministic gates, a model-editor pass, then targeted per-section rewrites. Sources list built from passages actually used. - Newsletter rules: one issue per send date, one section per post published in the period plus one exclusive item, sequential issue numbers assigned at ingestion. 4) Quality gates: deterministic code plus one model editor Length band, per-section length, grounding (every number, date, quotation and named claim must match a retrieved passage or linked post), authority-tier check for regulated claims, register bands (sentence length, variation, grade level), per-client banned-constructs list, structure and front-matter checks, link resolution, and a per-client rule pack for forbidden claim types, required disclosures and personal-data rules. Every threshold is configuration; every gate result with measured values goes on the audit row. 5) Operations Exactly one email per run (delivered / stood down with reason / failed with stage, node, raw error, audit id), an error-trigger workflow covering everything, a weekly summary, one audit row per run (documents, model per stage, gate results, token usage, outcome), configurable per-run token budget and monthly spend cap, dry-run mode, fixtures exercising every gate, daily reconciliation. BOUNDARY AND EXCLUSIONS Your pipeline ends by committing one Markdown file with YAML front matter on its own branch and opening a pull request in the client's repository. Validation, CMS ingestion, archiving and callbacks belong to the delivery track; both tracks share the front-matter schema and the signed callback contract. No WordPress, PHP or database work: the connector plugin and the Supabase store are built by other parties to your specifications. DELIVERABLES Normalised n8n workflow JSON in the repository as source of truth (deployed from Git, never imported by hand), prompt library as files, per-instance configuration file covering every configurable value, retrieval-store specification, runbook, fixtures, recorded handover plus a live session per operator. IP assigns on final payment. ACCEPTANCE PER INSTANCE Four consecutive weeks unattended with at most one manual intervention per week; at least 80% of scheduled runs deliver a draft passing every gate; zero unsupported figures or quotations in a ten-draft sample; no topic repeated inside the look-back window; every enabled source fetched within its interval or reported; spend inside the cap; the operator can add a source, change a threshold and re-run a failed run from the runbook alone. PROPOSAL Phases: discovery, shared build plus first instance, second instance, stabilisation (budgeted at no less than the build phase), handover. Give a duration and fixed price per phase, a support retainer option, your recommended n8n Cloud plan tier and model per stage with monthly running cost per instance, and two comparable deliveries including what broke during stabilisation and how you fixed it. The budget range shown is indicative; quote the real scope. Alternative architectures are welcome if you state which requirements change and how each is still met; client account ownership, drafts-only delivery and the client's repository as home for everything are fixed.