Skip to content

Latest commit

 

History

History
44 lines (34 loc) · 1.79 KB

subsetting.rst

File metadata and controls

44 lines (34 loc) · 1.79 KB

WordInfo subsetting

It is possible to ask Suachi to return only a subset of fields in the WordInfo. To do that, you can use the fields parameter of the :py:meth:`sudachipy.Dictionary.tokenizer()` method. The parameter accepts a set of strings, each one representing a field to be returned. By default, all fields are returned.

Allowed values:

  • surface: in-dictionary surface word form.
    :py:meth:`sudachipy.Morpheme.surface()` method returns the slice of the input text and is not affected by that flag.
  • pos or pos_id: part-of-speech tag.
  • normalized_form
  • dictionary_form
  • reading_form
  • word_structure
  • synonym_group_ids
  • user_data
  • split_a
  • split_b

Note

If you want only tokenization (e.g. use only :py:meth:`sudachipy.Morpheme.surface()`, passing empty set is allowed.

You need to load splits if you want to use :py:meth:`sudachipy.Morpheme.split` method. If performing the tokenization with non-default mode, the required splits will be loaded automatically and can be omitted.:

dic.tokenizer(SplitMode.B, fields=set()) # implicitly becomes fields={'split_b'}

To access user-defined dictionary data when field subsetting is enabled, load user_data explicitly.:

tokenizer = dic.tokenizer(fields={"surface", "user_data"})
morpheme = tokenizer.tokenize("すだち")[0]
assert morpheme.user_data() == "徳島県産"

If fields is specified without user_data, :py:meth:`sudachipy.Morpheme.user_data()` follows the same partial-loading rule as other fields and may return an incorrect result.

Warning

Using a field not included in the passed subset will produce an incorrect result without any warning. Use tests to ensure that the needed fields are loaded when using this parameter.