Skip to content

Migrating to 0.10 ​

Suzume 0.10.0 changes the public interface of every binding at once, and none of the old names remain as aliases. Most changes are renames that fail at compile or import time. The ones in Changes on every surface keep their names and change behaviour, so read that section first.

Changes on every surface ​

BeforeAfterWhy
A POS filter naming PARTICLE (or AUX) returns nothing while particles (or auxiliaries) are excluded by defaultNaming a POS in the filter admits it; the particle and auxiliary exclusions apply only to POS the filter does not nameA filter for particles should return particles
POS filter names noun, adjective, adverb, auxiliaryExactly NOUN, VERB, ADJ, ADV, PARTICLE, AUX, case-sensitive; an old name fails the whole callThe filter takes the labels the output prints
Native error text starts with a C function name, as in suzume_analyze: invalid UTF-8 inputThe prefix is gone: invalid UTF-8 inputCallers see the failure, not the C entry point; match on the error code instead of the text
Use after release fails with a binding-specific message and codeSuzume instance has been released with code InvalidInput in npm and PythonOne message and one code for one condition
Option and flag names with lemma and a bare user dictionarybase form and bundled user dictionaryOne term per concept: the base form, and the dictionary that ships with the package
とか, だの and やら are PART_副; で and だっ are AUX_断定; 屋 and 代 after a katakana host or a kana-spelled word stay inside the nounPART_列挙; AUX_断定連用; 屋 and 代 become a separate token with SUFFIX_造語. The coarse POS is unchangedEach class needs its own connection rules. Accept the new labels, or test the coarse pos where the subtype does not matter
Scorer options accept unary.particle_prior, unary.aux_prior and unary.pronoun_priorThe keys are removed and fail like any unknown key: npm and Python raise SuzumeError (Parse, operation create) with invalid scorer options: Unknown scorer option: unary.particle_prior; a SUZUME_SCORER_CONFIG file naming one is ignored as a whole with a warning; SUZUME_SCORER_UNARY_PARTICLE_PRIOR and the like draw Unknown scorer environment variableThe scorer never read them, so setting them had no effect. Delete them; no result changes

npm ​

BeforeAfterWhy
import Suzume from '@libraz/suzume'import { Suzume } from '@libraz/suzume'; the default export is removedOne import form
suzume.version getterawait version()One way to read the version
loadUserDictionary() returns boolean; loadUserDictionaryCount(); loadUserDictionaryOrThrow()loadUserDictionary(tsv): number returns the installed entry count and throws SuzumeError on failureOne loader per format, and a failure is always an error
loadBinaryDictionary() returns boolean; loadBinaryDictionaryOrThrow()loadBinaryDictionary(bytes): number, same contractSame as the text loader
loadUserDictionary() takes a stringIt takes a string or a Uint8Array; non-UTF-8 bytes fail with Dictionary entry is not valid UTF-8 at line N (ErrorCode.Parse), and any other value throws invalid dictionary: <value> (expected a string or binary data)A file read as bytes reaches the library unchanged
lastError, lastErrorCodeRemoved; read error.message and error.code from the thrown SuzumeErrorErrors are thrown, so there is no last-error state
TagOptions.posRemoved; use posFilterOne name for the filter
posFilter: ['noun', 'adjective']posFilter: ['NOUN', 'ADJ'], or a single name: posFilter: 'NOUN'Same labels as Morpheme.pos
Type TagPosFilterName ('noun' | 'verb' | 'adjective' | 'adverb' | 'particle' | 'auxiliary')Removed; use TagPosLabel ('NOUN' | 'VERB' | 'ADJ' | 'ADV' | 'PARTICLE' | 'AUX')The type lists the names the filter accepts
useLemmauseBaseFormOne term for the base form
skipUserDictionaryskipBundledUserDictionaryThe option skips only the bundled user.dic; loaded dictionaries still apply
Native failures throw Suzume analyze failed: <native text>The message is the native text unchanged; SuzumeError.operation names the call (create, analyze, generate_tags, load_user_dictionary, …)Text stays identical across bindings; branch on code and operation
Invalid options throw a plain Error; an unpaired surrogate throws SuzumeError with InvalidUtf8SuzumeError with code === ErrorCode.InvalidInput, for example invalid mode: "fast" (expected one of: normal, search, split)One error type for bad input, with the same text as Python; instanceof SuzumeError now also catches these, so check code
Boolean options are coerced by truthinessA non-boolean value throws invalid <name>: <value> (expected a boolean)preserveVu: "false" no longer means true
A scorerOptions object holding NaN, ±Infinity or undefined is rewritten by JSON.stringifyThrows invalid scorer_options: not serializable as JSON, at any depth; a malformed JSON string fails with ErrorCode.Parse, operation createA value the library cannot read is an error, not a silent null
Use after destroy() throws a plain Error('Suzume instance has been destroyed')SuzumeError, InvalidInput, Suzume instance has been releasedCatchable like every other failure
A WASM load failure rejects with the loader's own errorRejects with SuzumeError (code Internal, operation create); the loader's error is causeSuzume.create() rejects with one error type
ErrorCode is a TypeScript enum; ErrorCode[1] returns a nameErrorCode is a const object; reverse lookup is gone; ErrorCode.Unknown = 255 is addedErrorCode.InvalidUtf8 keeps working; 255 marks a native code the binding does not know
Result types are mutablereadonly fields, readonly Morpheme[], readonly Tag[] (type level only)Results are values; code that mutates them no longer type-checks

New: RuntimeOptions { wasmPath?, freshWasmModule? } is a named type (create(options?: SuzumeOptions & RuntimeOptions), version(options?: RuntimeOptions)), AnalysisMode, PosLabel, TagPosLabel and SuzumeOperation are exported, and Morpheme.pos and Tag.pos are typed as PosLabel.

Python ​

BeforeAfterWhy
load_user_dict()load_user_dictionary(), returns the installed entry countFull noun, as in every binding
load_binary_dict() returns Noneload_binary_dictionary(), returns the installed entry countSame contract as the text loader
load_user_dict() takes a strload_user_dictionary() takes str, bytes, bytearray or memoryview; non-UTF-8 bytes raise SuzumeError (PARSE, Dictionary entry is not valid UTF-8 at line N), and any other value raises ValueError invalid dictionary: <value> (expected a string or binary data)Bytes reach the library unchanged
use_lemmause_base_formOne term for the base form
skip_user_dictionaryskip_bundled_user_dictionaryThe option skips only the bundled user.dic
pos_filter takes an int bitmask or names; default 0Names only: a str, or a list, tuple, set or frozenset of str; default NoneSame labels as Morpheme.pos; pos_filter=0 is now invalid
Invalid mode, POS name or size raises ValueError with a Python repr listValueError with the shared text, for example invalid min_length: 2.5 (expected an integer from 0 to 4294967295)Same text as npm
An unpaired surrogate raises SuzumeError(INVALID_UTF8)Raises ValueErrorBad input is a ValueError; except SuzumeError no longer catches it
Size options reject floats; boolean options accept any valueIntegral floats such as 2.0 are accepted; non-booleans are rejectedSame accepted values as npm
Options are checked in no fixed order, and a set passed as pos_filter is checked in hash orderThe released check comes first, then the text, then the options in C struct field order, as in npm; a set is checked in sorted orderThe reported option does not depend on argument or hash order
An installed wheel can read dictionaries from SUZUME_DATA_DIR or ~/.suzumeIt loads only its bundled dictionaries from the package directory; there is no data_directory argumentThe installed package behaves the same on every machine
Use after close() raises SuzumeError(INTERNAL, "...closed")SuzumeError(INVALID_INPUT, "Suzume instance has been released")Calling a closed instance is a caller error, not an internal one
analyze(), generate_tags(), AnalysisResult.morphemes and dictionary_warnings return listThey return tupleResults are immutable values
Result dataclasses accept positional arguments and carry __dict__Frozen, slotted and keyword-only: Tag("x", "NOUN"), __dict__ access and attributes added in subclasses failSmaller objects with a fixed shape

New: SuzumeError.operation, ErrorCode.UNKNOWN = 255 and suzume.__version__.

C++ ​

BeforeAfterWhy
suzume::Tokenizer tokenizer; / Tokenizer(options) plus valid()auto tokenizer = suzume::Tokenizer::create(options); returns Expected<Tokenizer>Construction can fail, and the failure is in the result
analyze, analyzeWithNormalizedText, generateTags return an empty result on failureThey return Expected<T>; error() holds Error { ErrorCode code; std::string message; std::string operation; }An empty result and a failure look different
Static Tokenizer::lastError() / lastErrorCode()Removed; enum class ErrorCode includes UnknownThe error travels with the result, not in shared state
loadUserDictionary() returns bool; loadUserDictionaryCount(); loadBinaryDictionary(ptr, size) returns bool; clearUserDictionaries() returns boolExpected<std::size_t> loadUserDictionary(std::string_view), Expected<std::size_t> loadBinaryDictionary(std::string_view bytes), Expected<void> clearUserDictionaries()One loader per format, returning the entry count
TagOptions::pos_filter is a uint8_t bitmaskstd::vector<TagPos>, with enum class TagPos { Noun, Verb, Adj, Adv, Particle, Aux }Named values instead of bits
Morpheme::lemmaRemoved; use base_formOne name for the base form
conj_type / conj_form are "" when not applicablestd::optional<std::string>"Not applicable" is explicit
setMode() returns boolReturns Expected<void>; mode() and hasCoreDictionary() are noexceptSame error model as the other members
Tokenizer::version()Free function suzume::version()The version belongs to the library, not to an instance
TagOptions::use_lemma, Options::skip_user_dictionaryuse_base_form, skip_bundled_user_dictionarySame names as the other bindings
generateTags(text) and generateTags(text, opts)One generateTags(text, const TagOptions& = {})One overload
Options::scorer_options_jsonOptions::scorer_options (JSON text)The type already says it is a string

Calling value(), * or -> on an Expected that holds an error calls std::abort(); check has_value() or operator bool first. The header builds with -fno-exceptions. Tokenizer::create() returns an Internal error when the header and the library disagree on the ABI revision.

C ABI ​

SUZUME_ABI_VERSION goes from 1 to 2. Rebuild against the 0.10 header; a program built against the 0.9 header does not run with the 0.10 library.

BeforeAfterWhy
suzume_extended_options_t, suzume_init_extended_options, suzume_create_with_extended_options, suzume_sizeof_extended_options, suzume_offsetof_extended_optionssuzume_options_t, suzume_init_options, suzume_create_with_options, suzume_sizeof_options, suzume_offsetof_optionsThere is only one options struct, so it needs no qualifier
int suzume_load_user_dict(), size_t suzume_load_user_dict_count(), int suzume_load_binary_dict()size_t suzume_load_user_dictionary(h, data, size), size_t suzume_load_binary_dictionary(h, data, size); the return value is the installed entry count, 0 means failureOne loader per format, both returning the count
suzume_tag_options_t.use_lemmause_base_form (layout unchanged)One term for the base form
suzume_options_t.skip_user_dictionaryskip_bundled_user_dictionary (layout unchanged)The flag skips only the bundled user.dic
Label functions return NULL for none or out-of-range codesThey never return NULL: suzume_pos_label returns "OTHER", suzume_extended_pos_label returns "UNKNOWN", the conjugation label functions return ""Labels are always printable; compare the code to tell an invalid value from a real OTHER
Error text starts with the function nameThe suzume_<name>: prefix is removedSee Changes on every surface

Unchanged but newly specified: the four suzume_generate_tags* functions keep their names, and NULL options in the _with_options pair mean the defaults. Bit n of pos_filter selects POS code n+1, a non-zero pos_filter is a whitelist, and an unknown bit selects nothing, so a mask of only unknown bits yields no tags. New: suzume_mode_t and suzume_tag_pos_t typedefs, SUZUME_*_FIELD_* index enums for the offsetof functions, and suzume_pos_label_ja().

Command-line tools ​

The Python suzume command and the native suzume-cli use the same flags. A flag is the option name in kebab case; an option that defaults to true is turned off with --no-<name>; tag options start with --tag-.

BeforeAfterWhy
--normalize-vu--no-preserve-vuFollows the option name preserve_vu
--lowercase--no-preserve-caseFollows preserve_case
--no-core-dict--skip-core-dictionaryFollows skip_core_dictionary
--no-user-dict--skip-bundled-user-dictionaryFollows skip_bundled_user_dictionary
--dict PATH--dictionary PATH (-d unchanged)One noun per concept
--include-particles, --include-auxiliaries, --include-formal-nouns, --include-low-info--tag-no-exclude-particles, --tag-no-exclude-auxiliaries, --tag-no-exclude-formal-nouns, --tag-no-exclude-low-infoFollows the exclude_* tag options
--tag-keep-duplicates--tag-no-remove-duplicatesFollows remove_duplicates
--tag-use-surface--tag-no-use-base-formFollows use_base_form
--tag-pos noun--tag-pos-filter NOUN (repeatable, exact labels)Follows pos_filter and its labels
A value-taking option consumes the next argument even when it is a flagA next argument that begins with - and is longer than one character is not a value: missing value for <option>, exit 2; pass such a value with the long option as --option=value (a bare - is still a value)-d --mode search should not load a file named --mode
--tag-min-length and --tag-max-tags take any integer0 to 4294967295; a larger value exits 2 with invalid --tag-min-length value: X (or --tag-max-tags)Same range as the binding options
-f morphemeRemoved; the default format is tsv, which prints the same columnsTwo names for one format
JSON key lemmabase_form; pos_ja, conj_type and conj_form are added (null when absent)JSON keys match the binding field names
A --tag-* flag with a format other than tags is ignoredanalyze exits 2 with <prog>: error: --tag-* options require -f tagsA flag that does nothing is a usage error
No input text exits 1Exits 2 (no input text provided); an empty argument or empty standard input prints nothing and exits 0Missing input is a usage error; empty input is valid
Native test -f FILE and test benchmark -f FILE--file FILE-f means the output format everywhere
A dictionary that fails to load prints a CLI-specific message<prog>: error: failed to load dictionary: <path>: <native text>, exit 1One message shape in both tools
Native --version prints a second description lineOne line: <prog> <version>Same output in both tools
Python suzume --version extra and suzume version extra exit 2They print the version and exit 0, as suzume-cli does; a usage error elsewhere on the line still exits 2 firstSame parsing in both tools
Python suzume reads a command word anywhere among the operandsOnly the first operand is a command word: suzume version prints the version, suzume 東京 version analyzes 東京 versionText that contains a command word stays text
Python suzume -d FILE decodes the file before loading itIt passes the bytes to the library, so a non-UTF-8 file reports failed to load dictionary: FILE: Dictionary entry is not valid UTF-8 at line N, exit 1Same message as suzume-cli
The two CLIs word their help differentlyBoth print the same option table, Output Formats list and notes for every shared flag; suzume-cli adds its developer flags and its Commands, Examples and Environment sectionsOne description per flag
Python suzume accepts attached short values such as -dPATH and -fjsonUnknown option, exit 2; write -d PATH and -f json, or --dictionary=PATH (long options keep --option=value)Same parsing in both tools
Invalid values report (expected normal, search, or split)invalid mode: fast (expected one of: normal, search, split); -f and --tag-pos-filter use the same shapeSame wording as the binding option errors
suzume-cli messages after error:, warning: or info: start with a capital letterThey start lowercase, for example info: loaded dictionary: PATHOne style for every message
The native CLI does not print warnings for a -d file; the Python CLI fills -f chasen conjugation columns whenever a token has themBoth print dictionary warnings after each -d load; -f chasen fills the conjugation columns only for VERB, ADJ and AUX and prints * otherwiseBoth tools produce the same output, and chasen readers take * as non-conjugating

suzume-cli subcommands also change:

  • dict interactive and dict edit are removed and exit 2 with unknown dict subcommand: X; use dict -i or its long form dict --interactive.
  • A usage error in test, dict or dict compile exits 2 instead of 1. I/O failures, parse and compile failures and a refusal to overwrite a file still exit 1.
  • Value errors name the flag: invalid --pos value: X, invalid --limit value: X, invalid --iterations value: X (also --samples and --warmup). The positional POS of interactive add and update keeps invalid POS: X.
  • dict list, the interactive list and test benchmark accept --pos NOUN, --limit 10 and --iterations 1000 as well as the = form.
  • The missing-value rule above also holds in test and dict: write --expect=-tag, since --expect -tag exits 2 with missing value for --expect.

Dictionary files ​

Source rows are surface<TAB>POS[<TAB>conj_type][<TAB>base_form]. The old layouts below were accepted and are now parse errors in every loader and in suzume-cli dict compile and dict validate. The Used legacy TSV layout warning no longer exists.

BeforeAfterWhy
Legacy TSV with reading and a numeric cost in column 4Removed legacy TSV layout at line N: column 4 is a numeric cost; rows are surface, POS[, conj_type][, base_form]The cost column never affected matching
A numeric third column, silently droppedRemoved cost column at line N: column 3 is numeric; rows are surface, POS[, conj_type][, base_form]; write a numeric base form in column 4 with column 3 emptyA dropped value hid a mistake
Comma-separated rowsRemoved legacy CSV layout at line N; use tab-separated rowsOne delimiter
A malformed quoted field fails with Invalid legacy CSV quoting at line N: <detail>Invalid quoting at line N: <detail>The CSV layout no longer exists

The POS column keeps its full set of spellings, including long names, Japanese names, SYM and PHRASE.

When migrating a source dictionary, run suzume-cli dict validate. Runtime loading now limits surface and base-form values to 255 UTF-8 bytes and rejects surfaces containing NUL. The validator reports these problems by source line before you load or compile the file. If runtime loading skips every entry, it reports Dictionary source contains no loadable entries.

Go ​

The current go-suzume module uses these API changes and checks the C ABI revision at program start. On a fresh checkout, make lib builds its pinned Suzume core and dictionaries. After updating the core, run make lib in the Go checkout and go build -a in the application module. If the header and linked library disagree, startup panics with suzume: C ABI mismatch: …; run 'make lib' to rebuild.

BeforeAfterWhy
Options with three fields and NewWithOptions(Options); ExtendedOptions, DefaultExtendedOptions(), NewWithExtendedOptions()One Options with every field, DefaultOptions() and NewWithOptions(opts Options) (*Suzume, error); the zero value of Options is not the defaultOne options type
Analyze, AnalyzeWithNormalizedText, GenerateTags, GenerateTagsWithOptions return only the resultThey also return an errorA failure and an empty result look different
LoadUserDictionary and LoadBinaryDictionary return error; LoadUserDictionaryCountBoth return (int, error) with the installed entry count; LoadUserDictionaryCount is removedOne loader per format, returning the count
LastError(), LastErrorCode()Removed; inspect the returned *ErrorThe error travels with the result
Error.Message joins the operation and the native text as operation: text*Error has Code, Op and Message; Message is the native text unchanged, or <op> failed when the library gives noneSame text as the other bindings; branch on Code and Op
Calls on a closed instance fail with a Go-specific errorErrorCodeInvalidInput, Suzume instance has been released; errors.Is(err, suzume.ErrClosed) is trueOne message and one code for one condition
Out-of-range MinLength and MaxTags are clampedinvalid min_length: -1 (expected an integer from 0 to 4294967295) (same for max_tags); an invalid mode or POS mask fails the same waySame validation and text as npm and Python
TagOptions.UseLemma, Options.SkipUserDictionary, Options.ScorerOptionsJSONUseBaseForm, SkipBundledUserDictionary, ScorerOptionsSame names as the other bindings
TagOptions.POSFilter is a uint8It is a TagPOS; the POS* constants keep their values, and POSNoun also admits pronounsA named type for the mask
ScorerOptions accepts unary.particle_prior, unary.aux_prior and unary.pronoun_priorNewWithOptions() returns ErrorCodeParse, Op create, and invalid scorer options: Unknown scorer option: unary.particle_prior (and the corresponding name for the other removed keys)The scorer did not read these options
Runtime dictionary loading accepts surfaces with NUL and base forms longer than 255 bytesSurface and base-form values are limited to 255 UTF-8 bytes, and surfaces cannot contain NUL. Partially accepted sources return the installed count and add warnings. A non-empty source with no loadable entries returns ErrorCodeParse, Op load_user_dictionary, and Dictionary source contains no loadable entriesMatch the source dictionary validation

New: ErrorCodeUnknown (255) and ErrClosed. AnalysisMode is a uint8 with the same constant values.