Skip to content

Publishing a paper

How a finished manuscript becomes a typeset PDF and a website article. This is the production pipeline, not the editorial one — for what kind of paper to write, see Publication entry points.

Read this if you are about to publish something, or if you are a partner who wants the EMPN template without touching the website.

All six stages in one run

empn-core:run-publication-pipeline chains everything below in a background run and writes a single status file. Every phase degrades rather than aborting, so one pass surfaces every problem instead of stopping at the first. It never publishes — it stops at a saved draft.

feeds the typeset paper feeds the website feeds both Click a stage to pin it.

The two outputs are separate, and they need different figures

This is the thing that surprises everyone. One manuscript produces two deliverables, and they do not share figure files:

flowchart TB
    S["manuscript<br/>(.docx)"] --> P["produce-paper"]
    S --> W["website-article"]
    F["src/&lt;slug&gt;/config.json<br/>figure data"] --> C["create-chart"]
    C -->|"report.png + caption.json<br/>graphic only, caption as data"| P
    C -->|"embed.html<br/>interactive, self-contained"| W
    P --> PDF["typeset PDF<br/>via Word"]
    W --> WP["WordPress draft"]
Typeset paper Website
Figure file {slug}.report.png + {slug}.caption.json {slug}.embed.html
What it contains plot and legend only plot, caption, wordmark — everything
Caption becomes real document paragraphs stays inside the artefact
Sizing preset report_text_column (14 px ticks) iframe_embed (11 px ticks)
How it is placed injected into the Word template uploaded, then [empn_chart slug="…"]

Why the caption is split out for the paper. While the caption sits inside the bitmap, its rendered point size is a function of whatever scale the placement needs. Two figures placed at two widths get two different title sizes, and no arithmetic recovers it — the image scales as a unit. So the paper takes the graphic and sets the caption as document text.

Why the sizing preset matters. At the web preset's 11 px, axis ticks land near 6 pt on a printed page — under the legibility floor. One config can serve both: set chart.context: "report_text_column" and build with --embed-context iframe_embed, so the print outputs are report-sized while the web embed stays web-sized.

python3 .../create-chart/src/build_chart.py <slug> --embed-context iframe_embed

Every build emits all seven formats regardless, so nothing is lost by choosing wrong — you re-run and pick a different one.

Figures are rebuilt from data, never reused

The pipeline throws away the figures embedded in your manuscript. Both converters discard the source image and re-render it through create-chart.

That is deliberate. An embedded figure is usually in the wrong language, at screen resolution, in whatever colours the author's plotting library defaults to — and none of that is fixable in a bitmap.

You must supply the numbers

Because the figure is rebuilt, the data has to exist somewhere the pipeline can read: a CSV, an Excel sheet, or typed into src/<slug>/config.json. A picture of a chart is not data. If the numbers only exist inside the image, someone has to read them off it — and that is usually the single largest piece of work in publishing an existing paper.

Plan for this. If you are writing a paper you intend to publish through the pipeline, keep the figure data in a CSV next to the draft from the start.

Tables behave differently: they are carried through. Row data is extracted and re-rendered in the destination's own table style (core/table for WordPress, EMPN table style for the paper). Content survives; the author's formatting does not.

What you can start from

Source Works today What to do
.docx, real Heading styles ✅ Enter the pipeline directly
.docx, another template or another language's Word ⚠️ Usually works — see below
.md ❌ Convert to .docx first (Pandoc), then enter the pipeline
.pdf ❌ Recover the source, or retype. A PDF has already lost the structure
Google Docs ❌ Export as .docx

Both converters read .docx only — docx_to_publication.py walks word/document.xml, parse_manuscript.py uses python-docx. Everything else converts to .docx first.

Normalising any source

Everything downstream is deterministic, and the only reason a Markdown draft or a partner's Word file could not enter the pipeline was that nobody normalised it. normalise-manuscript closes that. The shape it follows:

1 · Convert the container (deterministic). Pandoc handles .md, .html, .odt, .tex → .docx losslessly enough for this purpose. This part is a one-liner and needs no judgement.

2 · Map the styles (deterministic where possible). Pandoc emits real Heading 1/Heading 2, so a converted Markdown file needs almost nothing. A foreign-template Word file needs its styles mapped — sections to a heading style, captions to a caption style, reference entries to a bibliography style. Mechanical once you know the mapping.

3 · Decide the ambiguous cases (model). This is the only part that needs a model, and it is a small part: is this bold paragraph a section heading or a lead-in sentence? Which caption belongs to which figure? Is this a reference list or an appendix? A rule cannot answer these; a model reading the document can.

4 · Verify by counting (deterministic, and this is the gate). Re-parse the normalised file and count sections, figures, tables and reference entries, then compare against what the source actually contains. This is what makes the model step safe — the model proposes a structure, the counter proves it. If the counts disagree, the run stops and reports the difference instead of quietly producing a half-parsed document.

That last point is the whole design. The failure mode of both parsers is a silent empty result: they complete, exit clean, and produce nothing usable. A verifier that counts converts that into a loud, specific error — the difference between a pipeline you can trust and one you have to babysit.

On the manuscript that exposed the problem — Swedish Word, sections styled Underrubrik — the raw source normalises with one flag and produce-paper then reads 10 chapters, 2 figures, 4 tables, 33 references and 1 footnote.

PDF is not in scope

"Any format" honestly means any text format. A PDF has already discarded the structure this stage needs to recover — headings are just larger text, tables are just aligned glyphs. Recover the source document instead; retyping is usually faster and always more accurate than guessing at a PDF's structure.

Word from another template

This is the common partner case, and the honest answer is it depends on styles, not on the template. Two mechanisms make a foreign-language Word file work:

  1. Built-in Word styles keep their w:name in English whatever the UI language. Swedish Rubrik2 carries w:name = "heading 2"; German Überschrift 1 carries heading 1. The converters key on that, not on the localised style id.
  2. A localised alias table catches the styles that are not built-ins — Swedish Underrubrik and Beskrivning, German Untertitel and Beschriftung, French Sous-titre and Légende, Italian Sottotitolo and Didascalia.

Reference-list headings are matched across English, Swedish, German, French and Italian — fifteen variants, from Bibliography and Works cited to Källor, Literaturverzeichnis and Riferimenti. Figure captions likewise match Figure, Figur, Abbildung, Abb. and Graphique.

What still breaks it:

  • Sections styled as Subtitle rather than a heading. Common when authors pick by look rather than by style — and note Swedish Underrubrik aliases to subtitle, not to a heading. The website converter handles subtitle-as-section and takes --sections-style "<styleId>" to force it. The paper parser does neither.
  • Sections that are just bold body text. No style, nothing to find. Apply real heading styles first — faster than any workaround.
  • A reference list headed something outside those fifteen variants is missed silently.

Check the counts, not the exit code

Both parsers can complete successfully having found nothing. After the parse step, verify the numbers — chapters, figures, tables, references — against the manuscript. A parse that reports "0 sections" and exits cleanly is the failure mode to watch for.

Who does which half

The pipeline splits cleanly, and a partner or researcher can own the first half without any EMPN website access.

flowchart LR
    subgraph A["Anyone — partner, researcher, author"]
      A1["figure data<br/>→ src/*/config.json"] --> A2["create-chart"]
      A2 --> A3["produce-paper"]
      A3 --> A4["Word: TOC + export PDF"]
    end
    subgraph B["Needs EMPN website access"]
      B1["website-article"] --> B2["upload embeds"]
      B2 --> B3["push draft"]
      B3 --> B4["publish"]
    end
    A4 -.->|"PDF attaches to the<br/>website publication"| B3

Partners and researchers can produce a fully branded EMPN paper — build the figures, run produce-paper, finish in Word, export the PDF. No WordPress account needed. The result is a publication-grade .docx and PDF in your own house template (or EMPN's, if your overlay declares no brand of its own).

The website half needs an EMPN login with edit_posts and upload_files, plus a logged-in wp-admin browser tab. If you do not have that, stop after the PDF and hand it over.

The steps

1 · Readiness gate

python3 shared/scripts/check_readiness.py paper.docx --report readiness.md

Sorts comments and tracked changes into blocking / needs-a-decision / settled, using the resolution state and reply threading Word itself records. Converts nothing. An unresolved editorial question is cheaper to fix now than after two outputs exist.

It also catches a specific trap: an image sitting inside an unaccepted deletion blocks, because it would silently vanish from every output. (docx_to_publication.py --readiness-only still exists as the converter's own lighter gate, but the three-bucket sort is the one to use.)

2 · Figures

One src/<slug>/config.json per figure, then build. Use --embed-context if the figure serves both outputs. Verify against figures/_manifest.json, never against the presence of a PNG — a failed build leaves output on disk that looks like a successful one.

3 · The typeset paper

produce-paper parses the manuscript, then injects it into the official template in place — cover panels, justified body, figures at their placement width, EMPN tables, footnotes converted to numbered margin sidenotes, chapters starting on new pages.

Three things it needs from you, and it will not invent any of them: the cover metadata (theme, byline with affiliations, a one-sentence excerpt), the policy-area icon for your theme, and key recommendations if the template's recommendation page is to be used.

Finish in Word: select all → Update Field to build the table of contents, confirm the brand fonts render, then export the PDF. Headless conversion substitutes fonts and cannot refresh the TOC field, so the PDF must come out of Word.

4 · The website

website-article converts the manuscript to WordPress block markup, then drives a logged-in wp-admin tab.

python3 shared/scripts/docx_to_publication.py paper.docx --out ./website
python3 shared/scripts/prepublish_check.py website/<slug>-body.txt --source paper.docx

Run the gate on the assembled markup, before the push — not after saving the draft. It checks link parity against the source, sidenote alignment, numbered headings, and that the reference list carries no bare URLs. Report-only: it blocks nothing and a human decides.

Then: upload each figure's .embed.html (Tools → EMPN Charts, slug must match the shortcode exactly), POST the body to the draft, fill the ACF fields through the editor UI, Save Draft.

Uploads need a person

Figure embeds are uploaded through an HTML file input. Browsers do not let script set a file input's value, and a page cannot fetch from localhost to get around it. Budget a few minutes of manual uploading, one per figure plus one for the body if it is large.

Folder layout

project-init scaffolds this; the canonical rules are in empn-folder-conventions.md.

<publication>/
├── _status.md                  ← entry point: index + pipeline state per step
├── _handoff.md                 ← decisions and their evidence
├── questions.md                ← open questions for co-authors
├── <manuscript>.docx           ← the source. Canonical by doctrine
├── src/<slug>/config.json      ← figure sources. A rebuild is create-chart on the slug
├── figures/<slug>/             ← built outputs, all seven formats
│   └── _manifest.json          ← one row per figure. Check THIS, not the PNGs
├── paper/                      ← typeset output + parse/meta JSON
└── website/                    ← block markup, manifest, prepublish report

Two conventions that pay off immediately:

  • Put the figure number in the slug (fig3-zone-prices-and-map). The slug is the build key, the output filename and the WordPress shortcode — carrying the number means it survives a rebuild, and the folder sorts in reading order.
  • _status.md is the entry point. On a publication that runs over weeks, the failure mode is a handoff note that accretes corrections until it contradicts itself. Keep the state in one file people read first, and let the detail file be the why.

Figure numbering follows the manuscript

Number figures the way the paper's own captions number them, not the order you happened to build them. If the manuscript calls something Table 3, it stays a table unless you also edit the prose that cites it.

Reference