Configuration¶

The XTrace SDK is designed to be highly configurable. You can configure it via environment variables or by passing parameters directly to classes and methods.

Cryptography Configuration¶

The SDK supports two production-ready homomorphic encryption schemes and one experimental scheme.

Both PaillierClient and PaillierLookupClient run on CPU by default.

Note

GPU backend (internal testing phase). XTrace maintains a GPU-accelerated implementation of the homomorphic encryption layer that is approximately 20× faster than the CPU path for large embedding collections. It is available as a compiled extension that slots into the same DEVICE=gpu switch — no application code changes required.

The GPU implementation is not open-sourced at this time as it is under internal testing. Contact us at liwen@xtrace.ai if you are interested in access.

Set DEVICE=gpu to activate the GPU backend once the compiled extension is in place.

Paillier — standard Paillier encryption:

from xtrace_sdk.x_vec.crypto.paillier_client import PaillierClient

paillier_client = PaillierClient(embed_len=512, key_len=1024)

Paillier-Lookup — optimised Paillier variant using precomputed tables for faster encryption. Recommended for large collections:

from xtrace_sdk.x_vec.crypto.paillier_lookup_client import PaillierLookupClient

paillier_client = PaillierLookupClient(embed_len=512, key_len=1024)

embed_len must be strictly less than key_len. Use at least key_len=1024 for security as it is bit-length for prime modulus.

Note

Goldwasser-Micali (xtrace_sdk.x_vec.crypto.goldwasser_micali_client) is included for research purposes and is experimental — it is not supported by DataLoader or Retriever and should not be used in production.

Execution Context Configuration¶

ExecutionContext bundles the homomorphic client and AES encryption under a single key-provider-protected object. A key provider supplies the AES key used to encrypt the homomorphic secret key at rest. The SDK ships two providers:

PassphraseKeyProvider — derives a 256-bit AES key from a passphrase using scrypt.
AWSKMSKeyProvider — envelope encryption via AWS KMS (the data encryption key is generated and wrapped by KMS and never persisted in plaintext).

Note

The key provider encrypts the secret homomorphic key at rest. There is no way to recover it through the SDK — manage your passphrase or KMS key securely.

Passphrase key provider¶

The simplest option — derive an AES key from a passphrase:

from xtrace_sdk.x_vec.utils.execution_context import ExecutionContext
from xtrace_sdk.x_vec.crypto.key_provider import PassphraseKeyProvider

provider = PassphraseKeyProvider("your-secret-passphrase")

ctx = ExecutionContext.create(
    key_provider=provider,
    homomorphic_client_type="paillier",
    embedding_length=512,
    key_len=1024,
    path="data/exec_context",   # optional: save immediately
)

You can also pass an explicit salt to PassphraseKeyProvider for additional key-derivation control.

AWS KMS key provider¶

For production workloads, use envelope encryption via AWS KMS. The data encryption key (DEK) is generated by KMS and never stored in plaintext:

import boto3
from xtrace_sdk.x_vec.utils.execution_context import ExecutionContext
from xtrace_sdk.x_vec.crypto.key_provider import AWSKMSKeyProvider

kms = boto3.client("kms")
provider = AWSKMSKeyProvider.create(kms, "alias/xtrace")

ctx = ExecutionContext.create(
    key_provider=provider,
    homomorphic_client_type="paillier_lookup",
    embedding_length=512,
    key_len=1024,
)

When loading a context that was saved with KMS, reconstruct the provider from the stored encrypted DEK (EDEK):

import base64, json, boto3
from xtrace_sdk.x_vec.crypto.key_provider import AWSKMSKeyProvider

# Read the EDEK from the saved context
with open("data/exec_context") as f:
    obj = json.load(f)
edek = base64.b64decode(obj["wrapped_key"])

kms = boto3.client("kms")
provider = AWSKMSKeyProvider.from_wrapped(edek, kms_client=kms, key_id="alias/xtrace")
ctx = ExecutionContext.load_from_disk(path="data/exec_context", key_provider=provider)

Persisting and reloading¶

You can persist and reload the execution context:

# Save to disk
execution_context.save_to_disk("data/exec_context")

# Load from disk (passphrase-based context)
execution_context = ExecutionContext.load_from_disk("your-secret-passphrase", "data/exec_context")

Or store/load it remotely via XTrace:

from xtrace_sdk.integrations.xtrace import XTraceIntegration
xtrace = XTraceIntegration(org_id="your_org_id", api_key="your_api_key")

# Save to remote
await execution_context.save_to_remote(xtrace)

# Load from remote
execution_context = await ExecutionContext.load_from_remote(
    "your-secret-passphrase", "ctx_id", xtrace
)

Each ExecutionContext has a unique id attribute you can use to reference it later.

The create convenience method accepts a key_provider along with homomorphic parameters and an optional path to save immediately:

ctx = ExecutionContext.create(
    key_provider=PassphraseKeyProvider("your-secret-passphrase"),
    homomorphic_client_type="paillier",
    embedding_length=512,
    key_len=1024,
    path="data/exec_context",   # optional: save immediately
)

DataLoader Configuration¶

DataLoader requires an execution context and an XTrace integration instance:

from xtrace_sdk.x_vec.data_loaders.loader import DataLoader
from xtrace_sdk.integrations.xtrace import XTraceIntegration

xtrace = XTraceIntegration(org_id="your_org_id", api_key="your_api_key")
data_loader = DataLoader(execution_context, xtrace)

To reconstruct a DataLoader from a saved execution context:

from xtrace_sdk.x_vec.utils.execution_context import ExecutionContext

ctx = ExecutionContext.load_from_disk("your-secret-passphrase", "data/exec_context")
data_loader = DataLoader(ctx, xtrace)

Retriever Configuration¶

Retriever mirrors the DataLoader setup. Pass parallel=True to decode Hamming distances using multiprocessing (useful for large KBs):

from xtrace_sdk.x_vec.retrievers.retriever import Retriever

retriever = Retriever(execution_context, xtrace)

# parallel decoding mode
retriever = Retriever(execution_context, xtrace, parallel=True)

Environment Variables¶

The following environment variables are read automatically:

Variable	Description
`XTRACE_API_KEY`	XTrace API key (used by `XTraceIntegration` when `api_key` is not passed explicitly).
`XTRACE_ORG_ID`	Organisation ID (used by `XTraceIntegration` when `org_id` is not passed explicitly).
`XTRACE_API_URL`	API base URL — defaults to `https://api.production.xtrace.ai`.
`XTRACE_EXECUTION_CONTEXT_PATH`	Default path to a saved execution context.
`INFERENCE_API_KEY`	API key for your inference provider (OpenAI, Redpill, etc.).