atproto_sdl/lexer

Grapheme-by-grapheme scanner turning Lexicon SDL source into a token stream with 1-indexed line/col spans. Triple-slash /// doc comments become significant DocComment tokens; plain // comments are discarded. String and integer literals, identifiers, keywords, and single punctuation each map to one token. The stream always ends with Eof.

pub fn tokenize(
  source: String,
) -> Result(List(#(token.Token, token.Span)), error.ParseError)