Glossary
Short definitions of the terms these docs use, written for readers who work with text but not with Japanese grammar. Each entry links to the page that covers the topic in depth. Examples show the bundled tokenizer's actual output.
POS
Part of speech: the grammatical category of a token. The pos field holds an uppercase English tag (NOUN, VERB, PARTICLE, …) and posJa holds the Japanese name.
東京に行く | 東京 | に | 行く |
|---|---|---|---|
pos | NOUN | PARTICLE | VERB |
posJa | 名詞 | 助詞 | 動詞 |
The tag set is listed in Part-of-Speech Labels. Where Suzume's tags differ from MeCab's, see POS Classification.
Extended POS
A finer label in extendedPos that adds the inflected form, the particle role, or the noun subtype to the POS: VERB_未然, AUX_否定, NOUN_形式. In 書かない, 書か is VERB_未然 and ない is AUX_否定. The values are listed in Part-of-Speech Labels.
Base Form
The dictionary form of a token, the lemma in linguistics. JavaScript exposes it as baseForm, Python as base_form, and Go as BaseForm; the CLI JSON key and the user-dictionary column are base_form. In 食べた, the token 食べ has the base form 食べる. Base forms let you match every inflected form of a word with one key. They are separate from normalization; see Normalization Is Not Lemmatization.
Surface
The token's text as it appears in the normalized input. In 飲んだ, the surface 飲ん has the base form 飲む. Characters left out of the output, such as whitespace and (by default) punctuation, show up as gaps between one token's end and the next token's start. See Offsets.
Conjugation Type
The inflection class of a verb or adjective, in conjType: which pattern its endings follow. 書い (from 書く) is 五段・カ行 and 食べ (from 食べる) is 一段. It is null for words that do not inflect. See Godan and Ichidan and the conjugation types a user dictionary accepts.
Conjugation Form
Which inflected form a token is, in conjForm: 未然形, 連用形, 終止形 and so on. Japanese verbs change their ending depending on what follows, so the form tells you the token's role in the phrase.
| Text | Token | conjForm |
|---|---|---|
書かない | 書か | 未然形 |
書いた | 書い | 連用形 |
東京に行く | 行く | 終止形 |
連用形 (Continuative Form)
The form a verb takes before ます, た, て or a second verb. In 飲み始めた, both 飲み and 始め are 連用形. The same spelling often doubles as a noun (飲み can mean "a drink"), so whether it is a verb or a noun depends on context. See Deverbal Compound Nouns.
未然形 (Irrealis Form)
The form a verb takes before negation ない and before passive or causative endings. In 書かない, 書か is 未然形. See Causative-Passive for how Suzume splits these chains.
Godan and Ichidan
The two main verb classes. A godan (五段) verb cycles its final kana through the five vowels of one kana row: 書か, 書き, 書く, 書け, 書こ. An ichidan (一段) verb keeps its stem and swaps only the ending: 食べ + ない, 食べ + た. conjType reports godan verbs together with their row, such as 五段・カ行. The irregular verbs する and 来る form a third, small group.
音便 (Euphonic Change)
A sound change in the 連用形 of godan verbs before た and て. The token gets extendedPos VERB_音便 and conjForm 連用形.
| Text | Token | Base form | Kind |
|---|---|---|---|
書いた | 書い | 書く | イ音便 (い) |
飲んだ | 飲ん | 飲む | 撥音便 (ん) |
行った | 行っ | 行く | 促音便 (っ) |
Suru-Verb
A verb formed by adding する to a noun, as in 点検する ("to inspect"). Suzume returns 点検 (NOUN) and する (VERB) as separate tokens. For 撮影しました, it returns 撮影 (NOUN), し (VERB, base form する), まし (AUX, base form ます) and た (AUX). A user-dictionary SURU row covers only some forms of a suru-verb. See What a SURU Row Changes.
I-Adjective and Na-Adjective
An i-adjective (イ形容詞) ends in い and conjugates itself: 美しい, 美しくない. A na-adjective (ナ形容詞) is a stem that takes な before a noun and does not conjugate: 静かな部屋. Suzume tags both ADJ. The extended POS is ADJ_終止 for the i-adjective 美しい and ADJ_NA for the stem 静か, with the な as a separate AUX token. See Na-Adjective Recognition.
Subsidiary Verb
A verb that loses its own meaning after another verb and only adds an aspect or attitude, such as いる in 食べている ("is eating") or みる in 食べてみる ("try eating"). Suzume splits it from the main verb and tags it AUX after the connective て: いる is AUX_継続 and みる is AUX_試行. See Subsidiary Auxiliary Splitting.
Copula
The "to be" element after a noun: だ, です, and contractions such as じゃ. Suzume tags it AUX with the base form だ. In 本じゃない, the tokens are 本 (NOUN), じゃ (AUX, base form だ) and ない (ADJ). See Copula Negation.
Formal Noun
A noun with little meaning of its own that turns a clause into a noun phrase, such as こと or もの. In 読むことが好き, こと has isFormalNoun set and extendedPos NOUN_形式.
Low-Information Token
A token with little value as a search keyword: formal nouns and pronouns. It has isLowInfo set, and tag generation drops it by default. In これはとても良いものです, これ and もの are low-information. See generateTags in the JavaScript API reference.
Unknown Word
A token that no dictionary supplied; Suzume generated it from character and grammar patterns. It has isUnknown set and isFromDictionary cleared. Because Suzume ships a compact dictionary, many ordinary words take this path: in 東京に行く, 東京 is generated from patterns while に and 行く come from the dictionary. Unknown does not mean wrong. See Pattern Recognition.
Normalization
Rewriting the input's characters into one canonical form before tokenizing, such as full-width API to API or half-width カタカナ to カタカナ. It changes surfaces and offsets. It is not lemmatization, which assigns each token its dictionary form. See Normalization.
Compound
Several words that work as one unit, such as 高層ビル群 ("high-rise cluster"). Suzume keeps some compounds whole by default (自然言語処理 is one token) and can merge consecutive nouns with search mode or mergeCompounds. See Compound Merging and Compounds, Quantities, and Names.
Join Rules
Candidate generators that offer a longer single token, such as Web開発 (mixed script), 飛び込む (compound verb) or 東京行き (noun + destination 行き), before the best path is chosen. normal and search mode use them; split does not. See Choosing a Mode.
User Dictionary
Your own word list, loaded at runtime as TSV or as a compiled binary, to add product names, jargon or boundaries the built-in dictionaries lack. Tokens from it have isUserDict set. See User Dictionary.
Dictionary Layers
The four dictionaries Suzume consults together: L1 built-in, L2 core.dic, L3 the bundled user.dic, and L4 the dictionaries you load. See Dictionary Layers.