Preprocessing
-------------


Language
^^^^^^^^

Module for automatic language detection

.. automodule:: sisu.preprocessing.language
    :members:

Tokenizer
^^^^^^^^^

Module for extracting text parts from documents and decomposing text into base units

.. automodule:: sisu.preprocessing.tokenizer
    :members:

IDF Embedding
^^^^^^^^^^^^^^

Embedding with ITF disabled.

.. automodule:: sisu.embedding_idf
    :members: