Extending the Markdown Pipeline

How to add a new Markdown / MDX feature to zfb — when a directive is enough, when to write a Rust visitor, and when the AST converter itself needs a new arm.

The pipeline that turns Markdown / MDX into HTML lives in crates/zfb-content. It is plugin-shaped: parse to mdast, run mdast visitors, convert to hast, run hast visitors, serialize. There is no runtime / userland plugin loader — every visitor compiles into the binary — but the surface for adding a new visitor in-tree is small and stable, and that is where almost every "I want a new Markdown feature" change lives.

If your feature is just a new directive name (:::callout, ::youtube, :badge), you do not need this page — the directive registry handles it without touching Rust beyond a one-line register call. See Custom Directives for that path.

Decision: directive, visitor, or AST arm

Most contributions land in the second row — a fresh visitor.

The two-phase pipeline

The pipeline runs two distinct passes over two different ASTs:

1. Parse the input into mdast (markdown::mdast::Node) using markdown-rs with MDX-aware options.

2. mdast visitors — MdastVisitor implementations rewrite the markdown AST in place. Run first because some transforms only make sense before HTML structure exists. The default mdast visitors are CjkAutolinkBoundaryPlugin (when GFM autolink literals are enabled) and CjkFriendlyPlugin; configured extras such as TranscludePlugin, CodeTabsPlugin, GithubAlertsPlugin, ReadingTimePlugin, RubyPlugin, and DirectiveRegistry are inserted in this phase when their feature flags are enabled. The directive registry folds runs of paragraphs delimited by :::name / ::: into a single MDX JSX element, which is much easier on mdast than after <p> tags have appeared.

3. Convert mdast → hast via mdast_to_hast. hast is zfb's minimal HTML AST (HastNode).

4. hast visitors — HastVisitor implementations rewrite the HTML AST in place. Most rewrites that target HTML element structure (heading anchors, <figure> wrappers, <img> → JSX markers, syntax highlighting) live here.

5. Serialize hast to an HTML string in zfb_content::serializer.

Pick the phase that matches what you are operating on. Rule of thumb: if you need to look at the original Markdown structure (a directive, a paragraph run, a particular link reference style), mdast. If you need to look at HTML element structure (a <pre>, a heading level, an <img> with a width attribute), hast.

Visitor trait shape

The visitor traits live in the zfb-md-ast crate and are re-exported from zfb_content::pipeline for local callers. Both traits are intentionally small: implement visit for context-free transforms, and override visit_with_context only when the plugin needs source paths, project roots, diagnostics, or heading registries from BuildContext.

pub trait MdastVisitor {
    fn visit(&mut self, node: &mut MdastNode);

    fn visit_with_context(&mut self, node: &mut MdastNode, _ctx: &mut BuildContext<'_>) {
        self.visit(node);
    }
}

pub trait HastVisitor {
    fn visit(&mut self, node: &mut HastNode);

    fn visit_with_context(&mut self, node: &mut HastNode, _ctx: &mut BuildContext<'_>) {
        self.visit(node);
    }
}

The pipeline calls visit exactly once, with the root node. Recursion is the visitor's responsibility — there is no auto-walk. A typical hast visitor looks like:

use zfb_md_ast::{HastNode, HastVisitor};

pub struct MyPlugin;

impl HastVisitor for MyPlugin {
    fn visit(&mut self, node: &mut HastNode) {
        match node {
            HastNode::Root { children }
            | HastNode::Element { children, .. } => {
                for child in children {
                    // mutate `child` here, then recurse
                    self.visit(child);
                }
            }
            _ => {}
        }
    }
}

Visitors can carry state (per-document slug counters, configuration options, references to a shared resource). HeadingLinksPlugin keeps a HashMap<String, usize> for github-slugger-equivalent dedup; SyntectPlugin holds an Arc<Highlighter> so the syntax theme is shared across all code blocks in the build.

When to add a Core vs. Opt-in feature

Both live as Rust visitors, but where you wire them depends on how many consumers need them.

Core (wire into Pipeline::with_defaults) when the behaviour is universal: every content-collection consumer would want it the same way, with no legitimate reason to opt out. Examples: CjkFriendlyPlugin, HeadingLinksPlugin, CodeTitlePlugin, SyntectPlugin.

Opt-in (wire into Pipeline::with_defaults_and_features) when the feature is valuable but not universally needed, or when it requires project-specific config (a source map, a feature flag, custom options). Examples: all 12 features in zfb-md-extras.

The promotion threshold follows the three-consumer rule: don't extract until the same pattern has been written by hand in three different zfb consumer projects. One project's convenience is a recipe.

Where files go

Core plugins live under crates/zfb-content/src/plugins/. Opt-in features live under crates/zfb-md-extras/src/. The convention is one file per feature:

crates/zfb-content/src/plugins/
├── cjk_autolink.rs
├── cjk_friendly.rs
├── code_title.rs
├── directives.rs
├── external_links.rs
├── hard_breaks.rs
├── heading_links.rs
├── mermaid.rs
├── resolve_links.rs
├── strip_md_ext.rs
├── syntect_plugin.rs
├── toc.rs                  # heading-marker TOC (wired via features)
└── util/

crates/zfb-md-extras/src/
├── code_enrichment.rs
├── code_tabs.rs
├── github_alerts.rs
├── heading_marker_toc.rs
├── image_dimensions.rs
├── link_validation.rs
├── mermaid.rs
├── reading_time.rs
├── ruby.rs
├── toc_export.rs
└── transclude.rs

For Core plugins, add your file and re-export from crates/zfb-content/src/plugins.rs:

// in plugins.rs
pub mod my_plugin;
pub use my_plugin::MyPlugin;

For Opt-in features, add your file and expose the feature from crates/zfb-md-extras/src/lib.rs, gated on the corresponding MarkdownConfig::features flag.

Tests typically live in a #[cfg(test)] mod tests {} block alongside the plugin, with cross-plugin integration cases in crates/zfb-content/tests/. The existing tests/integration_pipeline.rs is the reference shape.

Wiring into the default pipeline

Pipeline::with_defaults() is the project-wide default plugin chain. Adding your visitor there means every caller that uses the defaults picks it up automatically. Append it in the right phase:

// in crates/zfb-content/src/pipeline.rs, inside the defaults builder
let mut p = Self::with_resolved_gfm_constructs(resolved);

// mdast phase
if resolved.autolink_literal {
    p.push_config_derived_mdast_visitor(Box::new(CjkAutolinkBoundaryPlugin::new()));
}
p.push_config_derived_mdast_visitor(Box::new(CjkFriendlyPlugin::new()));
p.push_config_derived_mdast_visitor(Box::new(MyMdastPlugin::new())); // <-- new, if mdast

// hast phase
p.push_config_derived_hast_visitor(Box::new(HeadingLinksPlugin::new()));
p.push_config_derived_hast_visitor(Box::new(CodeTitlePlugin::new()));
p.push_config_derived_hast_visitor(Box::new(MyPlugin)); // <-- new
p.push_config_derived_hast_visitor(Box::new(MermaidPlugin::new()));
p.push_config_derived_hast_visitor(Box::new(SyntectPlugin::new(highlighter)));
p

If your plugin is opt-in, do not put it in with_defaults(). Wire it into Pipeline::with_defaults_and_features(), which accepts a MarkdownFeatures config struct and appends only the visitors whose flags are set. That is how all 12 features in zfb-md-extras are wired. ResolveLinksPlugin and StripMdExtensionPlugin are handled separately because they need a project-specific source map, not just a feature flag.

Ordering matters

Visitor order is load-bearing. The defaults document the full rationale in Pipeline::with_defaults_and_features's doc comment, but the rules that bite most often:

• HeadingLinksPlugin runs first in the hast phase. Anything that mutates headings later sees the slugified id attributes. TocPlugin and TocExportPlugin depend on these id values.

• CjkAutolinkBoundaryPlugin runs before CjkFriendlyPlugin when GFM autolink literals are enabled. It splits over-consumed bare URLs before CJK emphasis retokenisation changes the surrounding text nodes.

• CjkFriendlyPlugin runs early in the mdast phase. Visitors that depend on CJK-adjacent emphasis being tokenised should run after it.

• CodeTitlePlugin runs before SyntectPlugin. SyntectPlugin replaces the entire <pre> element with a HastNode::Raw HTML fragment; once that happens, the data-meta attribute that carries title="…" is no longer reachable as structured AST.

• MermaidPlugin runs before SyntectPlugin. MermaidPlugin flags mermaid code blocks with data-mermaid="true"; SyntectPlugin uses that flag to skip them rather than syntax-highlighting the diagram source.

• CodeEnrichmentPlugin runs after SyntectPlugin. It post-processes the per-line <span class="line"> structure that syntect emits; it cannot run before syntect produces those spans.

• ImageDimensionsPlugin runs in the hast phase before SyntectPlugin. It only touches <img> elements and is order-independent relative to heading / code-block visitors.

• GithubAlertsPlugin runs in the mdast phase, before the directive registry and before the mdast → hast conversion. It rewrites blockquote nodes into MDX JSX elements.

• TranscludePlugin runs first in the mdast phase. Included content must be merged into the AST before any other visitor sees it.

When inserting a new plugin, ask: do I need to see element shapes that a later plugin will erase? Run before that plugin. Do I need the results of an earlier plugin's rewrite (a generated id, a synthesised JSX element)? Run after it.

Adding genuinely new syntax

Some Markdown constructs that markdown-rs parses are currently dropped by the mdast → hast converter. Look at mdast_to_hast in crates/zfb-content/src/pipeline.rs:

// Unhandled: degrade to empty Raw so we never crash on
// unsupported input. Tables, footnotes, definitions, math,
// reference links/images, ESM, frontmatter, etc. fall here.
_ => HastNode::Raw(String::new()),

If you want zfb to surface tables, footnotes, definitions, math, reference-style links, or anything else that lives in the catch-all, two changes are needed:

1. Add a match arm to mdast_to_hast that turns the mdast variant into the right HastNode::Element (or Raw for passthrough JSX/HTML). Mirror the existing arms — handle children with convert_children, build attributes as Vec<(String, String)>.

2. Possibly toggle markdown::ParseOptions if the construct needs an extension flag. The current Pipeline::with_mdx() uses markdown::ParseOptions::mdx(); you may need a custom ParseOptions with additional constructs.* fields enabled. Check the markdown crate's docs for the exact flag.

Tests for converter changes belong in pipeline.rs's own #[cfg(test)] mod tests {} block (the file already covers headings, code blocks, links, images, lists, blockquotes, MDX JSX) plus a matching round-trip case in crates/zfb-content/tests/.

What about runtime / userland plugins?

There is no Markdown-pipeline plugin loader. Every visitor compiles into the binary. The plugins: [] field in zfb.config is a build-orchestration plugin system (five hooks: setup, preBuild, postBuild, devMiddleware, previewMiddleware) — it handles alias registration, virtual modules, extra file emission, and dev-server/preview-server routes. It does not reach into the Markdown parse-and-visitor chain, which runs entirely inside the Rust engine.

The practical Markdown-pipeline extension model is: add the visitor in-tree. The visitor traits are stable across the workspace, so a feature written as a fresh plugins/ file rarely needs follow-up changes when the rest of the codebase moves.

See also

• Markdown Features — full catalog of Core and Opt-in features, with per-feature ordering notes.

• Custom Directives — author-facing story for :::name / ::name / :name syntax, no Rust required.

• Customizing Markdown — what the Markdown rendering pipeline looks like from a content-collection consumer's perspective.

• crates/zfb-md-ast/src/lib.rs — MdastVisitor, HastVisitor, BuildContext, and the visit_with_context default methods.

• crates/zfb-content/src/pipeline.rs — Pipeline construction and the Pipeline::with_defaults() ordering rationale in its doc comment.

• crates/zfb-content/src/plugins/code_title.rs — small, stateless HastVisitor example.

• crates/zfb-content/src/plugins/heading_links.rs — stateful HastVisitor (per-document slug counter) example.

• crates/zfb-content/src/plugins/directives.rs — DirectiveRegistry and its MdastVisitor implementation.