Skip to content

ISCC - Text Code#

A similarity preserving hash for plain-text content (soft hash).

The ISCC Text-Code is generated from plain-text that has been extracted from a media assets.

Tip

Plain-text extraction from documents in various formats (especially PDF) may yield very diffent results depending on the extraction tools being used. For reproducible Text-Code generation use Apache Tika v2.2.1 to extract text from your documents.

Algorithm overview

gen_text_code_v0(text, bits = 64) #

Create an ISCC Text-Code with algorithm v0.

Note

Any markup (like HTML tags or markdown) should be removed from the plain-text before passing it to this function.

:param str text: Text for Text-Code creation :param int bits: Bit-length of ISCC Code Hash (default 64) :return: ISCC schema instance with Text-Code and an aditional property characters :rtype: dict

Source code in iscc_core\code_content_text.py 
def gen_text_code_v0(text, bits=ic.core_opts.text_bits):
    # type: (str, int) -> dict
    """
    Create an ISCC Text-Code with algorithm v0.

    !!! note
        Any markup (like HTML tags or markdown) should be removed from the plain-text
        before passing it to this function.

    :param str text: Text for Text-Code creation
    :param int bits: Bit-length of ISCC Code Hash (default 64)
    :return: ISCC schema instance with Text-Code and an aditional property `characters`
    :rtype: dict
    """

    text = text_collapse(text)
    characters = len(text)
    digest = soft_hash_text_v0(text)

    text_code = ic.encode_component(
        mtype=ic.MT.CONTENT,
        stype=ic.ST_CC.TEXT,
        version=ic.VS.V0,
        bit_length=bits,
        digest=digest,
    )

    iscc = "ISCC:" + text_code

    return dict(iscc=iscc, characters=characters)

soft_hash_text_v0(text) #

Creates a 256-bit similarity preserving hash for text input with algorithm v0.

Note

Before passing text to this function it must be:

  • stripped of markup
  • normalized
  • stripped of whitespace
  • lowercased

:param str text: Plain text to be hashed. :return: 256-bit similarity preserving byte hash. :rtype: bytes

Source code in iscc_core\code_content_text.py 
def soft_hash_text_v0(text):
    # type: (str) -> bytes
    """
    Creates a 256-bit similarity preserving hash for text input with algorithm v0.

    - Slide over text with a
      [`text_ngram_size`][iscc_core.options.CoreOptions.text_ngram_size] wide window
      and create [`xxh32`](https://cyan4973.github.io/xxHash/) hashes
    - Create a [`minhash_256`][iscc_core.minhash.alg_minhash_256] from the hashes generated
      in the previous step.

    !!! note

        Before passing text to this function it must be:

        - stripped of markup
        - normalized
        - stripped of whitespace
        - lowercased

    :param str text: Plain text to be hashed.
    :return: 256-bit similarity preserving byte hash.
    :rtype: bytes
    """
    ngrams = ic.sliding_window(text, ic.core_opts.text_ngram_size)
    features = [xxhash.xxh32_intdigest(s.encode("utf-8")) for s in ngrams]
    hash_digest = ic.alg_minhash_256(features)
    return hash_digest

text_collapse(text) #

Normalize and simplify text for similarity hashing.

  • Decompose with NFD normalization.
  • Remove all whitespace characters and convert text to lower case
  • Filter control characters, marks (diacritics), and punctuation
  • Recombine with NFKC normalization.

Note

See: Unicode normalization.

:param str text: Plain text to be collapsed. :return: Collapsed plain text. :rtype: str

Source code in iscc_core\code_content_text.py 
def text_collapse(text):
    # type: (str) -> str
    """
    Normalize and simplify text for similarity hashing.

    - Decompose with NFD normalization.
    - Remove all whitespace characters and convert text to lower case
    - Filter control characters, marks (diacritics), and punctuation
    - Recombine with NFKC normalization.

    !!! note

        See: [Unicode normalization](https://unicode.org/reports/tr15/).

    :param str text: Plain text to be collapsed.
    :return: Collapsed plain text.
    :rtype: str
    """

    # Decompose with NFD
    text = unicodedata.normalize("NFD", text)

    # Remove all whitespace and convert text to lower case
    text = "".join(text.split()).lower()

    # Filter control characters, marks (diacritics), and punctuation
    text = "".join(
        ch for ch in text if unicodedata.category(ch)[0] not in ic.core_opts.text_unicode_filter
    )

    # Recombine
    text = unicodedata.normalize("NFKC", text)

    return text