Library

ocomment-core is the engine the CLI, the LSP server, and the plugin host are all built on, published as an ordinary crate. It does no I/O: it takes bytes and returns what it found and what it would write.

cargo add ocomment-core

The complete API reference lives on docs.rs and is generated from the source, so it is the authority on every type and every field:

• ocomment-core — the scanner, the policy, the transform, and the source map.
• ocomment-plugin-sdk — for writing a WebAssembly scanner plugin; see Plugins.
• ocomment — the CLI crate, if you need to depend on the binary’s own version metadata.

Every example on this page is a doctest in the crate, so CI compiles and runs it on every pull request. The longer ones are checked-in examples you can run:

cargo run -p ocomment-core --example strip
cargo run -p ocomment-core --example external_spans
cargo run -p ocomment-core --example incremental
cargo run -p ocomment-core --example profile

The three calls

scan reports. transform reports and also gives you the bytes. apply_edits is the last step of transform, exposed on its own for a caller that wants to filter or postpone the edits.

use ocomment_core::{CommentKind, Language, ScanOptions, scan};

let report = scan(b"let x = 1; // note\n", Language::Rust, ScanOptions::default());
assert_eq!(report.comments.len(), 1);
assert_eq!(report.comments[0].kind, CommentKind::Line);
assert!(report.comments[0].action().removes());

use ocomment_core::{Language, TransformOptions, transform};

// NOTE: A BOM, a CRLF ending, and a comment to take out.
let source = "\u{feff}fn main() {} // trailing\r\n".as_bytes();
let result = transform(source, Language::Rust, TransformOptions::default());
assert_eq!(result.output, "\u{feff}fn main() {} \r\n".as_bytes());

use ocomment_core::{Language, TransformOptions, apply_edits, transform};

let source = b"let x = 1; // note\nlet y = 2; // and\n";
let result = transform(source, Language::Rust, TransformOptions::default());

// NOTE: Sorted and non-overlapping, so one pass applies them.
assert!(
    result
        .edits
        .windows(2)
        .all(|pair| pair[0].span.end <= pair[1].span.start)
);
assert_eq!(apply_edits(source, &result.edits), result.output);

A TransformResult carries the output, the edits that produced it, the report those edits came from, and a source_map from the original byte offsets to the new ones — which is what lets an editor keep a cursor, a diagnostic, or a breakpoint pointing at the right place after a removal.

What the engine guarantees

• Byte spans are half-open: span.start..span.end.
• Edits are sorted and non-overlapping, so applying them in order is enough.
• The source is never required to be valid UTF-8. A BOM, CRLF line endings, a missing trailing newline, and non-UTF-8 bytes outside the edited spans all come back unchanged.
• The output of a scan is deterministic for the same bytes, language, and options — that is what the OCaml reference implementation is compared against.

The other axis

Policy decides whether a comment stays. StyleRules decides how it reads once it has, and the two are deliberately separate tables: a comment that fails a condition of survival is removed, and a comment that fails a style rule is rewritten.

A verdict is therefore three-valued. Action is Keep, Rewrite or Remove, and the two questions worth asking about one are removes() and changes_bytes() — not == Action::Keep, which is a category written as one variant’s name and answers wrongly the day the category gains a member. Disposition::Rewrite carries the bytes it would write, so the rule and the replacement cannot disagree.

Where the answer is about a paragraph rather than a comment it is not on any comment at all. ScanReport::runs holds them, in source order:

use ocomment_core::{Language, ProseOrigin, ScanOptions, StyleRule, StyleRules, Wrap, scan};

let options = ScanOptions {
    policy: ocomment_core::Policy::None,
    style: StyleRules { wrap: Wrap::Sentence, ..StyleRules::default() },
    ..ScanOptions::default()
};
let report = scan(b"fn a() {}\n// One sentence. Another one.\n", Language::Rust, options);
let run = &report.runs[0];
assert_eq!(run.origin, ProseOrigin::Comments);
assert_eq!(run.rule, StyleRule::Wrap);
assert_eq!(run.replacement, b"// One sentence.\n// Another one.");

A run is there and not on the comments because the bytes it replaces are not any one comment’s: joining two comment lines moves the newline and the indentation between them, and those belong to neither. ProseOrigin::Document is the same kind of answer about a Markdown page’s own prose, which is not a comment either. A caller that reads only comments finds every removal and no reflow.

What survives

Every comment is classified as a CommentKind first — from its delimiters, then from its own text and position — and the Policy then decides that kind:

The policy is the last word rather than the first: keep_kinds, keep_regex, remove_kinds and remove_regex on ScanOptions are all tested before it, in that order. Policies is the same table from the binary’s own mouth, and Why a comment was kept lists what makes a comment a directive.

explain_disposition answers why for one comment, naming the rule that applied rather than summarising it:

use ocomment_core::{
    Action, CommentKind, DispositionExplanation, Language, Policy, ScanOptions,
    explain_disposition,
};

let mut options = ScanOptions::default();
let why = explain_disposition(CommentKind::Line, b"// note", Language::Rust, &options);
assert_eq!(why.action(), Action::Remove);
assert!(matches!(
    why,
    DispositionExplanation::RemovedByDefault {
        policy: Policy::Standard,
        kind: CommentKind::Line,
    }
));

options.keep_regex.push(r"^//\s*NOTE\b".into());
let kept = explain_disposition(CommentKind::Line, b"// NOTE: why", Language::Rust, &options);
assert_eq!(kept.action(), Action::Keep);
assert!(matches!(kept, DispositionExplanation::KeptByRegex { index: 0, .. }));
assert_eq!(kept.to_string(), r"kept: matched keep_regex #0 `^//\s*NOTE\b`");

explain_disposition_with takes pattern sets compiled once, which is what you want when explaining a whole file rather than one comment.

One rule is missing from that account, and no reading of a comment’s bytes could supply it: a YAML block scalar decides where its body ends from the lines below it, so the comment a body ends at is kept for where it sits rather than for what it says. explain_comment takes the Comment a scan produced instead of a kind and some bytes, which is what lets it see that rule — it is the entry point that agrees with what the scan recorded, and for every other comment it answers exactly as explain_disposition does:

use ocomment_core::{
    Action, DispositionExplanation, Language, ScanOptions, explain_comment, scan,
};

let source = b"k: |\n  a\n# ends the block\n  # yamllint disable\nz: 1\n";
let options = ScanOptions::default();
let report = scan(source, Language::Yaml, options.clone());

// NOTE: The scan kept the first comment: removing its line would hand the directive under it back to the block scalar above.
let comment = &report.comments[0];
let why = explain_comment(
    comment,
    &source[comment.span.start..comment.span.end],
    Language::Yaml,
    &options,
);
assert_eq!(why.action(), Action::Keep);
assert!(matches!(
    why,
    DispositionExplanation::KeptStructural {
        language: Language::Yaml
    }
));

explain_comment_with is to explain_comment what explain_disposition_with is to explain_disposition: the same answer, against pattern sets the caller compiled once.

Choosing the language

detect_language resolves a path and its contents to a Language, and Language can also be named directly when you already know it — which is what you want for a buffer that has no path. Languages and dialects lists everything built in, and a profile describes a delimiter-based syntax that is not.

use std::path::Path;
use ocomment_core::{Dialect, Language, detect_language};

let found = detect_language(Some(Path::new("src/app.tsx")), b"").unwrap();
assert_eq!(found.language, Language::TypeScript);
assert_eq!(found.dialect, Dialect::Tsx);
assert_eq!(found.reason, "extension");

// NOTE: No name, so the shebang decides.
let piped = detect_language(None, b"#!/usr/bin/env python3\n").unwrap();
assert_eq!(piped.language, Language::Python);

A scanner of your own

transform_spans takes comment spans an external scanner already found and puts them through the same policy, layout, edit validation, and source map as a built-in scan, after checking that the spans are non-empty, sorted, non-overlapping, and inside the source. That is the hand-off point a WebAssembly plugin uses, and it is the one to use for a scanner you would rather keep in your own process.

use ocomment_core::{
    ByteSpan, CommentKind, ExternalSpanError, Language, TransformOptions, transform_spans,
};

let source = b"a/* ordinary */b/* directive */";
let result = transform_spans(
    source,
    Language::Unknown,
    &[
        (ByteSpan::new(1, 15), CommentKind::Block),
        (ByteSpan::new(16, source.len()), CommentKind::Directive),
    ],
    TransformOptions::default(),
)
.unwrap();
// NOTE: The same policy the built-in scanners get: the directive is kept.
assert_eq!(result.output, b"a b/* directive */");

let bad = transform_spans(
    source,
    Language::Unknown,
    &[(ByteSpan::new(2, source.len() + 1), CommentKind::Block)],
    TransformOptions::default(),
);
assert!(matches!(bad, Err(ExternalSpanError::OutOfBounds { .. })));

A DeclarativeProfile is the smaller answer: literal comment and string delimiters, read in a single byte-oriented pass. It needs no code, and what it cannot express it refuses rather than guesses — validate_profile rejects a delimiter that is a prefix of another, a nested block whose tokens overlap, and a delimiter containing a line terminator, because none of those has a single reading.

requires_boundary and requires_line_start are how a profile says where its token is allowed to open. The first keeps a token that also occurs inside an identifier from swallowing the rest of the line; the second is for the formats whose # is a comment as the first byte of a line and part of the data anywhere else, which is every pattern list — a .gitignore entry may contain one, and \#literal is how an entry that starts with one is written. Without the second, a removal in such a file writes a shorter pattern back and the file quietly stops matching what the line named.

use ocomment_core::{
    CommentKind, DeclarativeProfile, LineDelimiter, StringDelimiter, TransformOptions,
    transform_profile,
};

let profile = DeclarativeProfile {
    name: "lisp".into(),
    extensions: vec!["lisp".into()],
    line_comments: vec![LineDelimiter {
        start: ";;".into(),
        kind: CommentKind::Line,
        ..Default::default()
    }],
    strings: vec![StringDelimiter {
        start: "\"".into(),
        end: "\"".into(),
        escape: Some("\\".into()),
        ..Default::default()
    }],
    ..Default::default()
};

let source = b"(print \";; not a comment\") ;; a comment\n";
let result = transform_profile(source, &profile, TransformOptions::default()).unwrap();
assert_eq!(result.output, b"(print \";; not a comment\") \n");

The same profile can be written in .ocomment.toml instead, which is what most callers want; see Configuration.

Editing a live buffer

IncrementalDocument applies DocumentChanges and rescans only what moved, under a PositionEncoding of UTF-8, UTF-16, or UTF-32. That is the path the LSP server takes, and it is the one to use for anything that rescans on every keystroke rather than on every save.

apply_changes is transactional. A batch that fails validation — a stale version, an inverted span, a span reaching past the end — leaves the source, the report, the checkpoints, and the version exactly as they were, so a misbehaving client cannot corrupt the document:

use ocomment_core::{
    ByteSpan, DocumentChange, IncrementalDocument, IncrementalError, Language, ScanOptions,
};

let mut document = IncrementalDocument::new(
    b"let x = 1; // note\n".to_vec(),
    Language::Rust,
    ScanOptions::default(),
    1,
);

// NOTE: A span past the end of the document is refused, and nothing moves.
let outside = ByteSpan::new(0, document.source().len() + 1);
assert_eq!(
    document.apply_changes(
        &[DocumentChange {
            span: outside,
            replacement: Vec::new(),
        }],
        2,
    ),
    Err(IncrementalError::InvalidSpan),
);
assert_eq!(document.version(), 1);
assert_eq!(document.report().comments.len(), 1);

last_rescan_span reports how much of the document the last edit actually cost, so the saving is measurable rather than assumed.

Versioning

The workspace follows semantic versioning, and every crate in it is released at the same version by the same tag. The minimum supported Rust version is 1.88, and a dedicated CI job builds the workspace against exactly that toolchain on every pull request, so a dependency that quietly raises it fails the build rather than a user’s install.

Both library crates are documented item by item: missing_docs is denied in CI, the doctests on this page run there, and cargo doc runs with -D warnings, so a public item added without documentation, an example that stops compiling, or a broken intra-doc link fails the build. The ocomment binary crate is documented in the same run for its links alone — nothing publishes its rustdoc, but its modules describe each other, and a link naming an item somebody has since renamed is a wrong sentence wherever it is written.