# moca-convert

> @openmoca/moca-convert: turn Markdown folders, PDFs and YouTube videos into MOCA packages, with evidence that cites the page or the moment.

Source: https://openmoca.org/tools/convert/

Creates MOCA packages from existing sources, including PDFs and YouTube
videos and playlists. Every node it writes is an OKF
concept document with a `type`, and the package id must be an absolute URI.

```sh
moca-convert <input> -o <dir> --id <uri> [--title text] [--version semver] [--type type]
             [--language bcp47] [--license spdx] [--from directory|markdown|obsidian|openapi|pdf|youtube]
             [--exclude glob...] [--chunker operation|tag] [--min-description-ratio 0-1]
             [--sources copy|link]
             [--force] [--strict] [--format text|json]
```

| Adapter | Input | Default `type` |
| --- | --- | --- |
| `directory` | A folder of Markdown, structure preserved: the folders become `structure.ttl` (parts and folder order), each file bound to its concept; `--no-structure` to leave it out | `Document` |
| `markdown` | One file or a glob; names from titles | `Document` |
| `obsidian` | A vault (has `.obsidian/`); wikilinks become Markdown links | `Note` |
| `openapi` | An OpenAPI 3.x document with real descriptions | `API Operation` or `API Tag` |
| `pdf` | A PDF, a folder of PDFs, or a folder with a `pdfs.json` | `Document` |
| `youtube` | A capture folder written by `moca-youtube` | `Video` |

- Existing frontmatter is kept byte for byte. A file without a `type` gets
  `type` (and `title`, if missing) added at the top of its frontmatter.
- The adapter is detected from the input unless `--from` is given; ambiguous
  input is refused, not guessed.
- Output is read back with the reference Reader before anything is left on
  disk. If it would be invalid, nothing is written.

## PDF

PDFs with a text layer become one node per section:

- Headings come from the PDF's bookmarks when they cover the document, and
  otherwise from the type: sizes clearly larger than the body text are
  heading levels, and a whole line in bold at body size is one level below.
  A document is split at the first heading level that occurs more than once
  (the sections of a chapter, or the chapters of a book); deeper headings stay
  inside the node, where `moca-index --chunk headings` can split them.
- Running headers and footers, page numbers and small figure labels are
  dropped. Paragraphs run on across columns and pages, words hyphenated at a
  line end are rejoined, and bullets, numbered items and figure captions are
  kept as lists and captions.
- Each node cites every page it came from as an RFC 3778 `page=` fragment,
  with the printed page number (`p. 4-12`) as the note when the PDF has page
  labels or prints numbers in its footer. `structure.ttl` keeps the documents
  and their sections in order, and each node is bound to its section.
- `--sources copy` (the default) puts each PDF under `sources/`, so the
  package is `self-contained-evidence`. `--sources link` cites each PDF by
  its `url` instead, for PDFs too large to carry.
- A PDF without a text layer (a scan) is reported and left out; run OCR on
  it first.

A folder may list its documents, in order, in `pdfs.json`:

```json
{
  "title": "Workshop Manual",
  "description": "Brakes and torque settings.",
  "documents": [
    { "file": "ch7.pdf", "title": "Chapter 7: Brakes", "url": "https://example.com/ch7.pdf", "sha256": "…" }
  ]
}
```

Only `file` is required. `title` falls back to the PDF's own title, `url` is
the published copy that `--sources link` cites, and a file whose SHA-256 is
not `sha256` is refused, so a conversion never silently uses a changed
document. Without `pdfs.json`, every PDF in the folder is converted in path
order. moca-spec's [`scripts/fetch-pdfs.mjs`](https://github.com/openmoca/moca-spec/blob/main/scripts/fetch-pdfs.mjs) downloads the
listed files and pins their digests.

See [`examples/pdf`](/examples/pdf/).

## YouTube

`moca-youtube` captures a video or a playlist with
[yt-dlp](https://github.com/yt-dlp/yt-dlp), which must be installed, and then
runs `moca-convert --from youtube` on the capture:

```sh
moca-youtube <url> -o <dir> --id <uri> [--title text] [--version semver] [--type type]
             [--language bcp47] [--license spdx] [--capture-dir dir] [--no-convert]
             [--yt-dlp path] [--force] [--quiet]
```

- Each video becomes one node: its description, and a transcript from its
  captions grouped by chapter. The captions are written under `media/` as
  WebVTT and the description under `sources/`, and the node cites each
  chapter as a time range (`t=start,end`) and quotes the description, so the
  package is `self-contained-evidence`.
- A playlist also gets `content/index.md` and a `structure.ttl` that keeps the
  videos in playlist order.
- Captions the creator supplied in `--language` (default `en`) are preferred,
  then YouTube's automatic ones. A video without captions is kept, without a
  transcript, and reported.
- `--capture-dir` keeps the capture (`youtube-capture.json` and
  `videos/<id>/`); converting it again gives the same package. Without it the
  capture goes to a temporary directory.
- yt-dlp is run directly, without a shell, so this works on macOS, Linux and
  Windows. Point at it with `--yt-dlp` or `$YT_DLP` when it is not on `PATH`.

See [`examples/youtube`](/examples/youtube/).

## Develop

The tests read moca-spec's examples and fixtures. Check out moca-spec beside
this repository, or set `MOCA_SPEC_DIR`. CI uses the tags pinned at the top of `.github/workflows/test.yml`
(`MOCA_SPEC_REF` and the like); check out the same tags to match it.

```sh
npm install
npm test
```

## Release

1. Publish `@openmoca/moca-core` first if it changed.
2. Add a dated section to [CHANGELOG.md](https://github.com/openmoca/moca-convert/blob/v0.6.0/CHANGELOG.md) and update the version.
3. Point the `MOCA_*_REF` pins in `.github/workflows/test.yml` at the release tags
   ([moca-spec docs/releasing.md](/project/releasing/#moving-the-pins)).
4. `npm test` passes.
5. Tag `vX.Y.Z` and publish a GitHub release whose notes are the changelog
   section.
6. `npm publish --access public` from the tagged commit.

## Licence

Apache-2.0. See [LICENSE](https://github.com/openmoca/moca-convert/blob/v0.6.0/LICENSE) and [NOTICE](https://github.com/openmoca/moca-convert/blob/v0.6.0/NOTICE).

OpenMOCA™ is a trade mark of PI & Mash Solutions Ltd. The licence covers the
code, not the name: see the
[trade mark policy](https://github.com/openmoca/moca-spec/blob/main/TRADEMARKS.md).