xtrace_sdk.data_loaders.base
============================

.. py:module:: xtrace_sdk.data_loaders.base


Classes
-------

.. autoapisummary::

   xtrace_sdk.data_loaders.base.DataLoaderBase


Module Contents
---------------

.. py:class:: DataLoaderBase(execution_context, integration, **kwargs)

   This is the base class for a DataLoader.



   .. py:attribute:: execution_context


   .. py:attribute:: integration


   .. py:method:: _from_disk(path_to_execution_context, passphrase, integration)
      :classmethod:


      construct a DataLoader instance from saved execution context on disk.

      :param path_to_execution_context: the path to execution context saved on disk
      :type path_to_execution_context: str
      :param passphrase: The passphrase used for AES encryption.
      :type passphrase: str
      :return: a DataLoader instance.
      :rtype: DataLoader



   .. py:method:: _from_remote(passphrase, context_id, integration)
      :classmethod:

      :async:


      construct a DataLoader instance from saved execution context in remote storage.

      :param storage_integration: The storage integration instance that's going to be used for loading the index and DB.
      :type storage_integration: Type[StorageIntegrationBase]
      :return: a DataLoader instance.
      :rtype: DataLoader



   .. py:method:: init_default_execution_context(passphrase, homomorphic_client_type, embedding_length, key_len, path = None)
      :staticmethod:


      Initializes a default execution context with AES, embedding, and homomorphic encryption clients.

      :param passphrase: The AES key to be used for encryption and decryption of text chunks.
      :type passphrase: str
      :param homomorphic_client_type: The type of homomorphic client to be used, e.g., "PaillierClient"
      :type homomorphic_client_type: str
      :param path: The path to the execution context file, defaults to None
      :type path: str, optional
      :return: The initialized execution context
      :rtype: dict



   .. py:method:: dump_execution_context_to_disk(path)

      saves execution context to disk

      :param path: the path to which the execution context will be dumped
      :type path: str



   .. py:method:: dump_index(index, **kwargs)
      :async:


      Dumps index from memory to a persistent storage. Could be local disk or some remote storage service.

      :param index: index object in memory
      :type index: Index



   .. py:method:: dump_db(db, **kwargs)
      :async:


      dumps a db file to a persistent storage for future use

      :param index: a DB object that's going to be saved somethere
      :type index: EncryptedDB
      :param dest: the destination to which the db file will be stored. Could be local path or remote url. defaults to None
      :type dest: str, optional



   .. py:method:: upsert_one(chunk, vector, **kwargs)
      :async:


      This method is used for upserting a single chunk of data into the database. It encrypts the chunk and stores it in the database.

      :param chunk: the chunk of data to be upserted
      :type chunk: Chunk
      :raises ValueError: if the chunk is not a Chunk instance



   .. py:method:: delete_chunks(chunk_ids, **kwargs)
      :async:


      This method is used for deleting chunks from the database.

      :param chunk_ids: the list of chunk ids to be deleted
      :type chunk_ids: list[int]



   .. py:method:: update_chunks(chunk_updates, vectors, **kwargs)
      :async:


      This method is used for updating chunks in the database. It encrypts the chunks and updates them in the database.

      :param chunks: the list of chunks to be updated
      :type chunks: DocumentCollection
      :param vectors: the list of vectors corresponding to the chunks
      :type vectors: Iterable[list[int]]
      :raises ValueError: if any chunk is not a Chunk instance



   .. py:method:: load_data_from_memory(chunks, vectors, disable_progress = False)

      This method is used for loading document collection from memory and encrypting it. Modifies the document collection in place.

      :param chunks: the document collection to be loaded
      :type chunks: DocumentCollection
      :param vectors: the list of vectors corresponding to the chunks
      :type vectors: Iterable[list[int]]
      :param disable_progress: whether to disable the progress bar, defaults to False
      :type disable_progress: bool, optional
      :return: the index and the encrypted db as a tuple
      :rtype: tuple[Index, EncryptedDB]