Skip to content

Modes, Offsets, and Normalization ​

This page covers three settings that change what a token is and where it points: the segmentation mode, compound merging, and character normalization. It also explains how to map token offsets back to text in each binding.

Choosing a Mode ​

Every instance runs in one of three modes. The mode controls which compound candidates Suzume considers and whether it merges consecutive nouns after selecting the best path. It uses the same dictionaries in all three modes.

The mode decides which boundaries survive
API開発と高層ビル群 segmented in normal, search and split mode. Search merges 高層ビル群 into one token; split breaks API開発 into API and 開発.API開発と高層ビル群normaldefaultAPI開発と高層ビル群join rules appliedsearchAPI開発と高層ビル群noun runs mergedsplitAPI開発と高層ビル群join rules off= normal + mergeCompounds
Dashed lines mark the three boundaries that differ from normal: search removes the two inside 高層ビル群, and split adds one inside API開発. Every other boundary is the same in all three modes.
Loading WASM...
ModeWhat it doesTypical use
normal (default)Applies the join rulesDisplay, highlighting, general text processing
searchApplies the join rules and merges runs of consecutive nouns, except formal nounsSearch-index keys, tag extraction
splitSkips the join rules and the optional noun-merging stepPartial matching, recall-oriented indexing

The join rules offer longer tokens as candidates before Suzume selects the best path. For example, split returns Web / 開発 and 飛び / 込む. The chosen path can also affect the POS tags. In normal and search, 東京行き is one NOUN. In split, Suzume returns 東京 (NOUN) and 行き (VERB). All three modes return the honorific prefix and noun in お水 as お / 水.

Both the dictionary entry 取り扱い and the unmatched kanji run 高層建築群 can form a single token in every mode. 高層ビル群 splits into 高層 / ビル / 群 in normal and split because the script changes from kanji to katakana and back. In search, the noun-merging step combines those three into 高層ビル群.

split still runs merges that apply in every mode, such as joining 作業途中 and folding numeric expressions such as 3億5000万円 into one token. 東京都庁前 also stays one token in all three modes because the candidates leave the kanji run whole. To split it, register its parts in a user dictionary.

Try the modes on your own text:

Try it in the browser

Nothing is sent to a server. The Suzume WASM bundled with this page tokenizes, lemmatizes, and extracts tags locally.

Loading WASM...

Compound Merging ​

mergeCompounds enables the same noun-merging step as search mode, with formal nouns excluded. It is off by default.

ModemergeCompounds: falsemergeCompounds: true
normal取り扱い / 説明書 / を / 読む取り扱い説明書 / を / 読む
search取り扱い説明書 / を / 読む取り扱い説明書 / を / 読む
split取り扱い / 説明書 / を / 読む取り扱い / 説明書 / を / 読む (ignored)

normal with mergeCompounds and plain search give the same result. In split mode the option has no effect.

Setting the Mode ​

The mode is set at creation and can be changed later without reloading the dictionaries. mergeCompounds can only be set at creation.

BindingMode at creationChange laterCompound merging
JavaScriptSuzume.create({ mode })suzume.mode = ...mergeCompounds
PythonSuzume(mode=...)analyzer.mode = ...merge_compounds
GoOptions.ModeSetMode()Options.MergeCompounds
C++Options::modesetMode()Options::merge_compounds
CLI--mode—--merge-compounds
ts
import { Suzume } from '@libraz/suzume'

const suzume = await Suzume.create({ mode: 'search' })
suzume.mode = 'split'

const merged = await Suzume.create({ mergeCompounds: true })
python
from suzume import Mode, Suzume

with Suzume(mode=Mode.SEARCH) as analyzer:
    analyzer.mode = "split"

with Suzume(merge_compounds=True) as analyzer:
    print([m.surface for m in analyzer.analyze("取り扱い説明書を読む")])
go
opts := suzume.DefaultOptions()
opts.Mode = suzume.ModeSearch

s, err := suzume.NewWithOptions(opts)
if err != nil {
	log.Fatal(err)
}
defer s.Close()

if err := s.SetMode(suzume.ModeSplit); err != nil {
	log.Fatal(err)
}
cpp
suzume::Options options;
options.mode = suzume::Mode::Search;
auto tokenizer = suzume::Tokenizer::create(options);
if (!tokenizer) {
  std::fprintf(stderr, "%s\n", tokenizer.error().message.c_str());
  return 1;
}

if (auto split = tokenizer->setMode(suzume::Mode::Split); !split) {
  std::fprintf(stderr, "%s\n", split.error().message.c_str());
  return 1;
}
bash
# Python package CLI
suzume --mode search "API開発と高層ビル群"
suzume --merge-compounds "取り扱い説明書を読む"

# Native developer CLI
suzume-cli --mode split "API開発と高層ビル群"

In Go, start from DefaultOptions(): the zero value of Options turns off lemma correction and case/ヴ preservation.

Offsets ​

Every token carries start and end. In all bindings they count Unicode code points, and they index the normalized text, not the string you passed in. Each binding has a call that returns the normalized text together with the tokens.

BindingCall that returns the normalized textOffset unitHow to slice
JavaScriptanalyzeWithNormalizedText() → normalizedTextstart/end: code points; startUtf16/endUtf16: UTF-16 code unitsnormalizedText.slice(startUtf16, endUtf16)
Pythonanalyze_with_normalized_text() → normalized_textCode pointsnormalized_text[start:end]
GoAnalyzeWithNormalizedText() → NormalizedTextCode points[]rune(NormalizedText)[Start:End]
C++analyzeWithNormalizedText() → normalized_text (UTF-8)Code pointsConvert to byte offsets first
Csuzume_analyze_n() → normalized_text, normalized_text_sizeCode pointsConvert to byte offsets first
CLI--format json → normalized_textCode points—
Offsets index the normalized text
The input 𠮷野家でガイドを読む normalizes to 𠮷野家でガイドを読む, and token offsets count positions in that normalized text. 𠮷野家 ends at 3 in code points and at 4 in UTF-16 units.input𠮷野家でガイドを読む12 code points5 characters become 3normalized𠮷野家でガイドを読む10 code pointstokens𠮷野家でガイドを読む012345678910start · endcode points0234567891011startUtf16 · endUtf16UTF-16 code units
Because ガイド shrinks to ガイド, を sits at 7 in the normalized text but at 9 in the input. 𠮷 is one code point but two UTF-16 units, so JavaScript strings are sliced with startUtf16 and endUtf16.

Code points and UTF-16 units differ for characters outside the Basic Multilingual Plane, such as 𠮷. Passing start/end to String.prototype.slice() cuts such a token short.

ts
const text = '𠮷野家で食べた'
const { normalizedText, morphemes } = suzume.analyzeWithNormalizedText(text)

for (const m of morphemes) {
  console.log(normalizedText.slice(m.startUtf16, m.endUtf16))
}

// Offsets also index `text` only if normalization kept every character.
const aligned = [...normalizedText].length === [...text].length
python
text = "𠮷野家で食べた"
result = analyzer.analyze_with_normalized_text(text)

for m in result.morphemes:
    print(result.normalized_text[m.start:m.end])

# Offsets also index `text` only if normalization kept every character.
aligned = len(result.normalized_text) == len(text)
go
text := "𠮷野家で食べた"
result, err := s.AnalyzeWithNormalizedText(text)
if err != nil {
	log.Fatal(err)
}
runes := []rune(result.NormalizedText)

for _, m := range result.Morphemes {
	fmt.Println(string(runes[m.Start:m.End]))
}

// Offsets also index text only if normalization kept every character.
aligned := len(runes) == utf8.RuneCountInString(text)
cpp
// Byte position of code point `index` in UTF-8 text.
std::size_t byteOffset(std::string_view text, std::size_t index) {
  std::size_t pos = 0;
  while (pos < text.size() && index > 0) {
    ++pos;
    while (pos < text.size() &&
           (static_cast<unsigned char>(text[pos]) & 0xC0) == 0x80) {
      ++pos;
    }
    --index;
  }
  return pos;
}

const auto result = tokenizer->analyzeWithNormalizedText("𠮷野家で食べた");
if (!result) {
  std::fprintf(stderr, "%s\n", result.error().message.c_str());
  return 1;
}
for (const suzume::Morpheme& m : result->morphemes) {
  const std::size_t begin = byteOffset(result->normalized_text, m.start);
  const std::size_t end = byteOffset(result->normalized_text, m.end);
  std::printf("%s\n", result->normalized_text.substr(begin, end - begin).c_str());
}

Mapping Offsets Back to Your Input ​

Normalization either rewrites a character one-for-one or removes it; it never inserts characters. So if the normalized text has as many code points as your input, the offsets index your input as well. Width folding (API → API, カタカナ → カタカナ) keeps the count. The cases below shorten the text, and after them the offsets no longer line up with the input.

InputNormalizedWhy
ガイドを読むガイドを読むA half-width voicing mark is merged into the preceding kana
カ + U+3099ガA combining voicing mark is merged into the preceding kana
東京 + U+200B + タワー東京タワーZero-width spaces are removed
長いーー音長いー音Repeated ー before a kanji collapses to one
ヴァイオリン with preserveVu: falseバイオリンヴァ becomes one character

When the counts differ, slice the normalized text, or keep your own mapping from input positions to normalized positions.

Normalization ​

Before tokenizing, Suzume rewrites the input into one canonical character form, so dictionary lookups and pattern rules see a single spelling for text that can be typed several ways. Token surfaces and offsets come from this normalized text.

StepExampleWhen
Full-width ASCII (U+FF01–U+FF5E) to ASCIIAPI123 → API123Always
Half-width katakana to full-widthカタカナ → カタカナAlways
Half-width and combining voicing marks merged into the kanaガイド → ガイドAlways
Zero-width space (U+200B) removed東京 + U+200B + タワー → 東京タワーAlways
Repeated ー before a kanji collapsed長いーー音 → 長いー音Always
ASCII uppercase to lowercaseGitHub → githubOnly with preserveCase: false
ヴ-series to the バ rowヴァイオリン → バイオリンOnly with preserveVu: false
Loading WASM...

Hiragana and katakana are not folded into each other, and the ideographic space U+3000 is kept as is.

The two options mean the same thing in every binding:

BindingLowercase ASCIIFold ヴ
JavaScriptpreserveCase: falsepreserveVu: false
Pythonpreserve_case=Falsepreserve_vu=False
GoPreserveCase: falsePreserveVu: false
C++preserve_case = falsepreserve_vu = false
CLI--no-preserve-case--no-preserve-vu

preserveSymbols is not a normalization step. It decides whether punctuation-like SYMBOL tokens appear in the output; the normalized text keeps them either way.

Normalization Is Not Lemmatization ​

Both make text more uniform, but they act on different things.

NormalizationLemmatization
Works onCharacters, before tokenizingTokens, after tokenizing
Changes the surface and offsetsYesNo
ExampleGitHub → GitHub食べ → base form 食べる
Result appears insurface, normalized textbaseForm (base_form, BaseForm)
Controlled bypreserveCase, preserveVulemmatize (post-analysis base-form correction)

lemmatize (default true) replaces each token's source base form with the corrected one. With lemmatize off, tokens keep the base form from their dictionary entry or candidate, and POS and conjugation annotations are computed either way. Most base forms are identical either way: 食べている gives 食べる for 食べ in both cases. 蠢いている differs: the token 蠢い has the base form 蠢く with lemmatize on and 蠢い with it off.

See the glossary for a one-line definition and the JavaScript API reference for the full option list.