Migrating to 0.10
Suzume 0.10.0 changes the public interface of every binding at once, and none of the old names remain as aliases. Most changes are renames that fail at compile or import time. The ones in Changes on every surface keep their names and change behaviour, so read that section first.
Changes on every surface
| Before | After | Why |
|---|---|---|
A POS filter naming PARTICLE (or AUX) returns nothing while particles (or auxiliaries) are excluded by default | Naming a POS in the filter admits it; the particle and auxiliary exclusions apply only to POS the filter does not name | A filter for particles should return particles |
POS filter names noun, adjective, adverb, auxiliary | Exactly NOUN, VERB, ADJ, ADV, PARTICLE, AUX, case-sensitive; an old name fails the whole call | The filter takes the labels the output prints |
Native error text starts with a C function name, as in suzume_analyze: invalid UTF-8 input | The prefix is gone: invalid UTF-8 input | Callers see the failure, not the C entry point; match on the error code instead of the text |
| Use after release fails with a binding-specific message and code | Suzume instance has been released with code InvalidInput in npm and Python | One message and one code for one condition |
Option and flag names with lemma and a bare user dictionary | base form and bundled user dictionary | One term per concept: the base form, and the dictionary that ships with the package |
とか, だの and やら are PART_副; で and だっ are AUX_断定; 屋 and 代 after a katakana host or a kana-spelled word stay inside the noun | PART_列挙; AUX_断定連用; 屋 and 代 become a separate token with SUFFIX_造語. The coarse POS is unchanged | Each class needs its own connection rules. Accept the new labels, or test the coarse pos where the subtype does not matter |
Scorer options accept unary.particle_prior, unary.aux_prior and unary.pronoun_prior | The keys are removed and fail like any unknown key: npm and Python raise SuzumeError (Parse, operation create) with invalid scorer options: Unknown scorer option: unary.particle_prior; a SUZUME_SCORER_CONFIG file naming one is ignored as a whole with a warning; SUZUME_SCORER_UNARY_PARTICLE_PRIOR and the like draw Unknown scorer environment variable | The scorer never read them, so setting them had no effect. Delete them; no result changes |
npm
| Before | After | Why |
|---|---|---|
import Suzume from '@libraz/suzume' | import { Suzume } from '@libraz/suzume'; the default export is removed | One import form |
suzume.version getter | await version() | One way to read the version |
loadUserDictionary() returns boolean; loadUserDictionaryCount(); loadUserDictionaryOrThrow() | loadUserDictionary(tsv): number returns the installed entry count and throws SuzumeError on failure | One loader per format, and a failure is always an error |
loadBinaryDictionary() returns boolean; loadBinaryDictionaryOrThrow() | loadBinaryDictionary(bytes): number, same contract | Same as the text loader |
loadUserDictionary() takes a string | It takes a string or a Uint8Array; non-UTF-8 bytes fail with Dictionary entry is not valid UTF-8 at line N (ErrorCode.Parse), and any other value throws invalid dictionary: <value> (expected a string or binary data) | A file read as bytes reaches the library unchanged |
lastError, lastErrorCode | Removed; read error.message and error.code from the thrown SuzumeError | Errors are thrown, so there is no last-error state |
TagOptions.pos | Removed; use posFilter | One name for the filter |
posFilter: ['noun', 'adjective'] | posFilter: ['NOUN', 'ADJ'], or a single name: posFilter: 'NOUN' | Same labels as Morpheme.pos |
Type TagPosFilterName ('noun' | 'verb' | 'adjective' | 'adverb' | 'particle' | 'auxiliary') | Removed; use TagPosLabel ('NOUN' | 'VERB' | 'ADJ' | 'ADV' | 'PARTICLE' | 'AUX') | The type lists the names the filter accepts |
useLemma | useBaseForm | One term for the base form |
skipUserDictionary | skipBundledUserDictionary | The option skips only the bundled user.dic; loaded dictionaries still apply |
Native failures throw Suzume analyze failed: <native text> | The message is the native text unchanged; SuzumeError.operation names the call (create, analyze, generate_tags, load_user_dictionary, …) | Text stays identical across bindings; branch on code and operation |
Invalid options throw a plain Error; an unpaired surrogate throws SuzumeError with InvalidUtf8 | SuzumeError with code === ErrorCode.InvalidInput, for example invalid mode: "fast" (expected one of: normal, search, split) | One error type for bad input, with the same text as Python; instanceof SuzumeError now also catches these, so check code |
| Boolean options are coerced by truthiness | A non-boolean value throws invalid <name>: <value> (expected a boolean) | preserveVu: "false" no longer means true |
A scorerOptions object holding NaN, ±Infinity or undefined is rewritten by JSON.stringify | Throws invalid scorer_options: not serializable as JSON, at any depth; a malformed JSON string fails with ErrorCode.Parse, operation create | A value the library cannot read is an error, not a silent null |
Use after destroy() throws a plain Error('Suzume instance has been destroyed') | SuzumeError, InvalidInput, Suzume instance has been released | Catchable like every other failure |
| A WASM load failure rejects with the loader's own error | Rejects with SuzumeError (code Internal, operation create); the loader's error is cause | Suzume.create() rejects with one error type |
ErrorCode is a TypeScript enum; ErrorCode[1] returns a name | ErrorCode is a const object; reverse lookup is gone; ErrorCode.Unknown = 255 is added | ErrorCode.InvalidUtf8 keeps working; 255 marks a native code the binding does not know |
| Result types are mutable | readonly fields, readonly Morpheme[], readonly Tag[] (type level only) | Results are values; code that mutates them no longer type-checks |
New: RuntimeOptions { wasmPath?, freshWasmModule? } is a named type (create(options?: SuzumeOptions & RuntimeOptions), version(options?: RuntimeOptions)), AnalysisMode, PosLabel, TagPosLabel and SuzumeOperation are exported, and Morpheme.pos and Tag.pos are typed as PosLabel.
Python
| Before | After | Why |
|---|---|---|
load_user_dict() | load_user_dictionary(), returns the installed entry count | Full noun, as in every binding |
load_binary_dict() returns None | load_binary_dictionary(), returns the installed entry count | Same contract as the text loader |
load_user_dict() takes a str | load_user_dictionary() takes str, bytes, bytearray or memoryview; non-UTF-8 bytes raise SuzumeError (PARSE, Dictionary entry is not valid UTF-8 at line N), and any other value raises ValueError invalid dictionary: <value> (expected a string or binary data) | Bytes reach the library unchanged |
use_lemma | use_base_form | One term for the base form |
skip_user_dictionary | skip_bundled_user_dictionary | The option skips only the bundled user.dic |
pos_filter takes an int bitmask or names; default 0 | Names only: a str, or a list, tuple, set or frozenset of str; default None | Same labels as Morpheme.pos; pos_filter=0 is now invalid |
Invalid mode, POS name or size raises ValueError with a Python repr list | ValueError with the shared text, for example invalid min_length: 2.5 (expected an integer from 0 to 4294967295) | Same text as npm |
An unpaired surrogate raises SuzumeError(INVALID_UTF8) | Raises ValueError | Bad input is a ValueError; except SuzumeError no longer catches it |
| Size options reject floats; boolean options accept any value | Integral floats such as 2.0 are accepted; non-booleans are rejected | Same accepted values as npm |
Options are checked in no fixed order, and a set passed as pos_filter is checked in hash order | The released check comes first, then the text, then the options in C struct field order, as in npm; a set is checked in sorted order | The reported option does not depend on argument or hash order |
An installed wheel can read dictionaries from SUZUME_DATA_DIR or ~/.suzume | It loads only its bundled dictionaries from the package directory; there is no data_directory argument | The installed package behaves the same on every machine |
Use after close() raises SuzumeError(INTERNAL, "...closed") | SuzumeError(INVALID_INPUT, "Suzume instance has been released") | Calling a closed instance is a caller error, not an internal one |
analyze(), generate_tags(), AnalysisResult.morphemes and dictionary_warnings return list | They return tuple | Results are immutable values |
Result dataclasses accept positional arguments and carry __dict__ | Frozen, slotted and keyword-only: Tag("x", "NOUN"), __dict__ access and attributes added in subclasses fail | Smaller objects with a fixed shape |
New: SuzumeError.operation, ErrorCode.UNKNOWN = 255 and suzume.__version__.
C++
| Before | After | Why |
|---|---|---|
suzume::Tokenizer tokenizer; / Tokenizer(options) plus valid() | auto tokenizer = suzume::Tokenizer::create(options); returns Expected<Tokenizer> | Construction can fail, and the failure is in the result |
analyze, analyzeWithNormalizedText, generateTags return an empty result on failure | They return Expected<T>; error() holds Error { ErrorCode code; std::string message; std::string operation; } | An empty result and a failure look different |
Static Tokenizer::lastError() / lastErrorCode() | Removed; enum class ErrorCode includes Unknown | The error travels with the result, not in shared state |
loadUserDictionary() returns bool; loadUserDictionaryCount(); loadBinaryDictionary(ptr, size) returns bool; clearUserDictionaries() returns bool | Expected<std::size_t> loadUserDictionary(std::string_view), Expected<std::size_t> loadBinaryDictionary(std::string_view bytes), Expected<void> clearUserDictionaries() | One loader per format, returning the entry count |
TagOptions::pos_filter is a uint8_t bitmask | std::vector<TagPos>, with enum class TagPos { Noun, Verb, Adj, Adv, Particle, Aux } | Named values instead of bits |
Morpheme::lemma | Removed; use base_form | One name for the base form |
conj_type / conj_form are "" when not applicable | std::optional<std::string> | "Not applicable" is explicit |
setMode() returns bool | Returns Expected<void>; mode() and hasCoreDictionary() are noexcept | Same error model as the other members |
Tokenizer::version() | Free function suzume::version() | The version belongs to the library, not to an instance |
TagOptions::use_lemma, Options::skip_user_dictionary | use_base_form, skip_bundled_user_dictionary | Same names as the other bindings |
generateTags(text) and generateTags(text, opts) | One generateTags(text, const TagOptions& = {}) | One overload |
Options::scorer_options_json | Options::scorer_options (JSON text) | The type already says it is a string |
Calling value(), * or -> on an Expected that holds an error calls std::abort(); check has_value() or operator bool first. The header builds with -fno-exceptions. Tokenizer::create() returns an Internal error when the header and the library disagree on the ABI revision.
C ABI
SUZUME_ABI_VERSION goes from 1 to 2. Rebuild against the 0.10 header; a program built against the 0.9 header does not run with the 0.10 library.
| Before | After | Why |
|---|---|---|
suzume_extended_options_t, suzume_init_extended_options, suzume_create_with_extended_options, suzume_sizeof_extended_options, suzume_offsetof_extended_options | suzume_options_t, suzume_init_options, suzume_create_with_options, suzume_sizeof_options, suzume_offsetof_options | There is only one options struct, so it needs no qualifier |
int suzume_load_user_dict(), size_t suzume_load_user_dict_count(), int suzume_load_binary_dict() | size_t suzume_load_user_dictionary(h, data, size), size_t suzume_load_binary_dictionary(h, data, size); the return value is the installed entry count, 0 means failure | One loader per format, both returning the count |
suzume_tag_options_t.use_lemma | use_base_form (layout unchanged) | One term for the base form |
suzume_options_t.skip_user_dictionary | skip_bundled_user_dictionary (layout unchanged) | The flag skips only the bundled user.dic |
Label functions return NULL for none or out-of-range codes | They never return NULL: suzume_pos_label returns "OTHER", suzume_extended_pos_label returns "UNKNOWN", the conjugation label functions return "" | Labels are always printable; compare the code to tell an invalid value from a real OTHER |
| Error text starts with the function name | The suzume_<name>: prefix is removed | See Changes on every surface |
Unchanged but newly specified: the four suzume_generate_tags* functions keep their names, and NULL options in the _with_options pair mean the defaults. Bit n of pos_filter selects POS code n+1, a non-zero pos_filter is a whitelist, and an unknown bit selects nothing, so a mask of only unknown bits yields no tags. New: suzume_mode_t and suzume_tag_pos_t typedefs, SUZUME_*_FIELD_* index enums for the offsetof functions, and suzume_pos_label_ja().
Command-line tools
The Python suzume command and the native suzume-cli use the same flags. A flag is the option name in kebab case; an option that defaults to true is turned off with --no-<name>; tag options start with --tag-.
| Before | After | Why |
|---|---|---|
--normalize-vu | --no-preserve-vu | Follows the option name preserve_vu |
--lowercase | --no-preserve-case | Follows preserve_case |
--no-core-dict | --skip-core-dictionary | Follows skip_core_dictionary |
--no-user-dict | --skip-bundled-user-dictionary | Follows skip_bundled_user_dictionary |
--dict PATH | --dictionary PATH (-d unchanged) | One noun per concept |
--include-particles, --include-auxiliaries, --include-formal-nouns, --include-low-info | --tag-no-exclude-particles, --tag-no-exclude-auxiliaries, --tag-no-exclude-formal-nouns, --tag-no-exclude-low-info | Follows the exclude_* tag options |
--tag-keep-duplicates | --tag-no-remove-duplicates | Follows remove_duplicates |
--tag-use-surface | --tag-no-use-base-form | Follows use_base_form |
--tag-pos noun | --tag-pos-filter NOUN (repeatable, exact labels) | Follows pos_filter and its labels |
| A value-taking option consumes the next argument even when it is a flag | A next argument that begins with - and is longer than one character is not a value: missing value for <option>, exit 2; pass such a value with the long option as --option=value (a bare - is still a value) | -d --mode search should not load a file named --mode |
--tag-min-length and --tag-max-tags take any integer | 0 to 4294967295; a larger value exits 2 with invalid --tag-min-length value: X (or --tag-max-tags) | Same range as the binding options |
-f morpheme | Removed; the default format is tsv, which prints the same columns | Two names for one format |
JSON key lemma | base_form; pos_ja, conj_type and conj_form are added (null when absent) | JSON keys match the binding field names |
A --tag-* flag with a format other than tags is ignored | analyze exits 2 with <prog>: error: --tag-* options require -f tags | A flag that does nothing is a usage error |
| No input text exits 1 | Exits 2 (no input text provided); an empty argument or empty standard input prints nothing and exits 0 | Missing input is a usage error; empty input is valid |
Native test -f FILE and test benchmark -f FILE | --file FILE | -f means the output format everywhere |
| A dictionary that fails to load prints a CLI-specific message | <prog>: error: failed to load dictionary: <path>: <native text>, exit 1 | One message shape in both tools |
Native --version prints a second description line | One line: <prog> <version> | Same output in both tools |
Python suzume --version extra and suzume version extra exit 2 | They print the version and exit 0, as suzume-cli does; a usage error elsewhere on the line still exits 2 first | Same parsing in both tools |
Python suzume reads a command word anywhere among the operands | Only the first operand is a command word: suzume version prints the version, suzume 東京 version analyzes 東京 version | Text that contains a command word stays text |
Python suzume -d FILE decodes the file before loading it | It passes the bytes to the library, so a non-UTF-8 file reports failed to load dictionary: FILE: Dictionary entry is not valid UTF-8 at line N, exit 1 | Same message as suzume-cli |
| The two CLIs word their help differently | Both print the same option table, Output Formats list and notes for every shared flag; suzume-cli adds its developer flags and its Commands, Examples and Environment sections | One description per flag |
Python suzume accepts attached short values such as -dPATH and -fjson | Unknown option, exit 2; write -d PATH and -f json, or --dictionary=PATH (long options keep --option=value) | Same parsing in both tools |
Invalid values report (expected normal, search, or split) | invalid mode: fast (expected one of: normal, search, split); -f and --tag-pos-filter use the same shape | Same wording as the binding option errors |
suzume-cli messages after error:, warning: or info: start with a capital letter | They start lowercase, for example info: loaded dictionary: PATH | One style for every message |
The native CLI does not print warnings for a -d file; the Python CLI fills -f chasen conjugation columns whenever a token has them | Both print dictionary warnings after each -d load; -f chasen fills the conjugation columns only for VERB, ADJ and AUX and prints * otherwise | Both tools produce the same output, and chasen readers take * as non-conjugating |
suzume-cli subcommands also change:
dict interactiveanddict editare removed and exit 2 withunknown dict subcommand: X; usedict -ior its long formdict --interactive.- A usage error in
test,dictordict compileexits 2 instead of 1. I/O failures, parse and compile failures and a refusal to overwrite a file still exit 1. - Value errors name the flag:
invalid --pos value: X,invalid --limit value: X,invalid --iterations value: X(also--samplesand--warmup). The positional POS of interactiveaddandupdatekeepsinvalid POS: X. dict list, the interactivelistandtest benchmarkaccept--pos NOUN,--limit 10and--iterations 1000as well as the=form.- The missing-value rule above also holds in
testanddict: write--expect=-tag, since--expect -tagexits 2 withmissing value for --expect.
Dictionary files
Source rows are surface<TAB>POS[<TAB>conj_type][<TAB>base_form]. The old layouts below were accepted and are now parse errors in every loader and in suzume-cli dict compile and dict validate. The Used legacy TSV layout warning no longer exists.
| Before | After | Why |
|---|---|---|
| Legacy TSV with reading and a numeric cost in column 4 | Removed legacy TSV layout at line N: column 4 is a numeric cost; rows are surface, POS[, conj_type][, base_form] | The cost column never affected matching |
| A numeric third column, silently dropped | Removed cost column at line N: column 3 is numeric; rows are surface, POS[, conj_type][, base_form]; write a numeric base form in column 4 with column 3 empty | A dropped value hid a mistake |
| Comma-separated rows | Removed legacy CSV layout at line N; use tab-separated rows | One delimiter |
A malformed quoted field fails with Invalid legacy CSV quoting at line N: <detail> | Invalid quoting at line N: <detail> | The CSV layout no longer exists |
The POS column keeps its full set of spellings, including long names, Japanese names, SYM and PHRASE.
When migrating a source dictionary, run suzume-cli dict validate. Runtime loading now limits surface and base-form values to 255 UTF-8 bytes and rejects surfaces containing NUL. The validator reports these problems by source line before you load or compile the file. If runtime loading skips every entry, it reports Dictionary source contains no loadable entries.
Go
The current go-suzume module uses these API changes and checks the C ABI revision at program start. On a fresh checkout, make lib builds its pinned Suzume core and dictionaries. After updating the core, run make lib in the Go checkout and go build -a in the application module. If the header and linked library disagree, startup panics with suzume: C ABI mismatch: …; run 'make lib' to rebuild.
| Before | After | Why |
|---|---|---|
Options with three fields and NewWithOptions(Options); ExtendedOptions, DefaultExtendedOptions(), NewWithExtendedOptions() | One Options with every field, DefaultOptions() and NewWithOptions(opts Options) (*Suzume, error); the zero value of Options is not the default | One options type |
Analyze, AnalyzeWithNormalizedText, GenerateTags, GenerateTagsWithOptions return only the result | They also return an error | A failure and an empty result look different |
LoadUserDictionary and LoadBinaryDictionary return error; LoadUserDictionaryCount | Both return (int, error) with the installed entry count; LoadUserDictionaryCount is removed | One loader per format, returning the count |
LastError(), LastErrorCode() | Removed; inspect the returned *Error | The error travels with the result |
Error.Message joins the operation and the native text as operation: text | *Error has Code, Op and Message; Message is the native text unchanged, or <op> failed when the library gives none | Same text as the other bindings; branch on Code and Op |
| Calls on a closed instance fail with a Go-specific error | ErrorCodeInvalidInput, Suzume instance has been released; errors.Is(err, suzume.ErrClosed) is true | One message and one code for one condition |
Out-of-range MinLength and MaxTags are clamped | invalid min_length: -1 (expected an integer from 0 to 4294967295) (same for max_tags); an invalid mode or POS mask fails the same way | Same validation and text as npm and Python |
TagOptions.UseLemma, Options.SkipUserDictionary, Options.ScorerOptionsJSON | UseBaseForm, SkipBundledUserDictionary, ScorerOptions | Same names as the other bindings |
TagOptions.POSFilter is a uint8 | It is a TagPOS; the POS* constants keep their values, and POSNoun also admits pronouns | A named type for the mask |
ScorerOptions accepts unary.particle_prior, unary.aux_prior and unary.pronoun_prior | NewWithOptions() returns ErrorCodeParse, Op create, and invalid scorer options: Unknown scorer option: unary.particle_prior (and the corresponding name for the other removed keys) | The scorer did not read these options |
| Runtime dictionary loading accepts surfaces with NUL and base forms longer than 255 bytes | Surface and base-form values are limited to 255 UTF-8 bytes, and surfaces cannot contain NUL. Partially accepted sources return the installed count and add warnings. A non-empty source with no loadable entries returns ErrorCodeParse, Op load_user_dictionary, and Dictionary source contains no loadable entries | Match the source dictionary validation |
New: ErrorCodeUnknown (255) and ErrClosed. AnalysisMode is a uint8 with the same constant values.