Data Sources

Overview — the engine posts only what a source verifies

Most "AI content" tools work by asking a language model to make something up that sounds plausible. This system works the opposite way. It reads real data that already describes the network — the list of sites and what each one does, a library of marketing principles, weekly quality-score snapshots — and it fills fixed sentence templates with those verified facts. The model isn't writing claims; the data is, and the templates just phrase them.

That inversion is the reason the honesty guard is even possible. Because every template declares the exact fields it needs, and every field maps to a real value in a source file, the engine can check before it publishes: is this fact present and trustworthy? If yes, the post goes out. If no, the post is held and flagged for a human — never filled with a guess.

So "data sources" isn't a back-office detail here. It is the product. Understanding what feeds the engine is understanding why the output can be trusted. The sections below walk every source in turn.

The master table of sources

Five things feed the engine. Four are inputs the content is built from; the fifth is the connected-account tokens that let it actually publish. Here they are at a glance — each gets its own deep-dive section below.

Source 1 — the network site list (wt-sites.json)

This is the roster of the network: 177 sites, each with a domain and a short description of what it is. It's the raw material for two of the four content families.

• Site features — the engine spotlights one network site, using its real domain and description to say what it does and why someone should visit.
• Spotlights — when a site is genuinely new to the network, it gets an introduction rather than a routine feature.

What the data looks like

Each entry is essentially a domain and a description. That's deliberately small. The description is the one field a feature post leans on, so it has to be a real, complete sentence about the site — not a slug, not a placeholder, not a fragment cut off mid-thought.

Under the hood

The feed builder (/root/gen-social-feed.py on MAIN) reads this file, and for each usable site emits a site_feature item. Rotation is least-recently-used first with a per-family cap, so the network doesn't spotlight the same handful of sites over and over — every site gets its turn, and variety stays high across a week of posts.

Source 2 — the marketing-skills library (skills-data.js)

The network maintains a library of 47 marketing skills — real, teachable marketing principles. This is what powers the AI tip family: short, genuinely useful pieces of marketing advice that make the feed worth following, not just self-promotion.

How a tip is chosen

For each skill, the feed builder prefers the skill's stated goal — the plain-English statement of what the skill is for. If a clean goal isn't available, it falls back to a real principle heading from within the skill. Either way, the tip is drawn word-from-source. The engine never invents a marketing tip.

This matters because a made-up "tip" is exactly the kind of confident-but-empty filler these tools are usually accused of. By pulling only from a curated library the operator stands behind, every tip is one the team would actually give a client.

Under the hood

Each usable skill becomes an ai_tip feed item. Like site features, tips rotate least-recently-used with a per-family cap, so the same principle doesn't repeat within a short window.

Source 3 — the quality-score snapshots (ama-snapshots)

Every week the network's sites are graded on an Agents-First AI-readiness score — a measure of how well a site works for AI agents and modern discovery. Those grades are saved as dated snapshots in /root/ama-snapshots/scores-*.json on MAIN. Two content families draw from them.

Milestones — real, earned improvement

The builder compares this week's snapshot to last week's. When a site's score has genuinely gone up, that's a milestone worth announcing. But it only counts a milestone when the improvement is real and meaningful:

• the gain must be at least 5 points week-over-week, and
• the current score must be at least 60.

A one- or two-point wobble isn't a story, and celebrating a still-low score would be hollow. Those thresholds mean a milestone post always reflects a site that actually moved and is actually doing well.

New-site spotlights

When a site appears in this week's snapshot that wasn't in last week's, it's newly under measurement — a genuine new addition. The builder emits a spotlight (new_site) item for it, so new work gets a proper introduction.

Under the hood

Because milestones are computed by diffing two dated snapshots, freshness is built in: the comparison is always current-week versus prior-week. If a site has no prior-week score to compare against, there's simply nothing to claim — and the engine claims nothing.

Source 4 — the built feed (wholetech.json on the hub)

The three raw sources above never reach the publishing brain directly. Instead, a single nightly job on MAIN reads all of them, cleans and validates every candidate, and writes one tidy file of finished content items. That file is then shipped to the HUB, where the drafter picks it up.

Each feed item carries a kind, and a template applies only to its own kind — and only when every field that template requires is present. This is the last checkpoint before drafting: the templates are deterministic, filling fixed sentence patterns with verified fields, so generating one post or a thousand costs exactly the same (nothing). See The Content Engine for how the four families and their templates work.

Source 5 — connected-account tokens (the hub database)

The fifth source isn't content — it's permission to publish. When an account is connected (Bluesky, X, and others), the platform issues OAuth tokens, and those live in the HUB's PostgreSQL database in the Postiz Integration table. That table holds one row per connected account, with fields like providerIdentifier, name, token, refreshToken, internalId, disabled, and organizationId.

Two design choices matter here:

• Tokens live in the database, not in code. Nothing sensitive is baked into a script or a config file that could leak. The platform app keys live separately in /opt/postiz/social.env (chmod 600, root-only), installed by the operator's own terminal — keys never pass through chat or an assistant.
• Refresh is the platform's job. Access tokens refresh via each network's own OAuth flow, so a connected account keeps working without re-entering credentials.

Verified in, or held out — the conservative principle

Every source above is treated the same way: a candidate gets in only if it's complete and trustworthy. When it isn't, the engine holds back rather than reaching. This is the honesty guard applied at the data layer, before a post even exists.

Data quality & cleaning

Between the raw sources and the built feed, the generator does real cleanup so that what reaches the drafter is already publish-grade. The work is quiet but it's the difference between "real data" and "usable data."

Decoding and normalizing text

Descriptions and headings are decoded and tidied so they read as clean, plain sentences — no stray encoding artifacts, no leftover markup slipping into a post.

Dropping truncated or empty text

Any candidate whose key field is cut off or blank is removed at build time. Better to skip a site tonight and feature it tomorrow (once fixed) than to publish a fragment.

Choosing only meaningful score gains

The milestone thresholds (≥5 points, score ≥60) are a quality filter, not just a numeric one. They ensure the engine only ever announces improvement that a reasonable person would agree is worth announcing.

Pulling a genuine principle from each skill

For tips, the builder deliberately prefers the skill's stated goal — the truest one-line summary of the principle — and only falls back to a real heading. The output is always a real idea lifted from the library, phrased for a post. Length is capped (roughly 278 characters) so it fits the channels cleanly.

Provenance — tracing a post back to its source

Because every post is assembled from verified fields rather than freely written, each one has a clear trail back to the exact item it came from.

• A site feature or spotlight traces to a specific domain + description in wt-sites.json.
• An AI tip traces to a specific skill's goal or heading in skills-data.js.
• A milestone traces to a specific week-over-week delta between two dated snapshots in ama-snapshots/.
• A new-site spotlight traces to a specific new entry in this week's snapshot.

The chain is end-to-end auditable: the raw source on MAIN → the tagged item in the built feed (wholetech.json) → the drafted post in the queue → the published post on the channel. If a post is ever questioned, you can walk it straight back to the real fact that produced it. That is what "never states a fact it can't verify" means in practice — not a promise, a paper trail.

What is NOT collected

Just as important as what feeds the engine is what doesn't. This system is built to publish and report — not to surveil an audience.

• No audience harvesting. The engine does not scrape, buy, or build profiles of the people who follow or see the posts. It reads the network's own facts and its own quality scores — that's it.
• No per-user tracking. There's no hidden data collection about individuals riding along with the content.
• Only what's needed to publish and report. The sensitive data the system does hold is the connected-account tokens — the minimum required to post on the operator's behalf — and those are kept in the hub database, refreshed by the platform, and rotatable.

Adding a new source

The four content families are not the ceiling — they're the pattern. A new source (a new kind of feed item drawn from a new file of verified facts) follows the same shape every existing one does:

1. Point at a real source of truth. A file the operator controls, with complete fields — the same standard as wt-sites.json or the skills library.
2. Give it a kind. Tag its feed items with a new kind so a template can target exactly those items.
3. Write a deterministic template. Fixed sentence patterns that fill only verified fields, and that declare every field they require so the honesty guard can hold a post when a fact is missing.
4. Let it clean and validate at build time, so incomplete candidates are dropped before they ever reach the drafter.