bittensor/bittensor/_dendrite/dendrite_impl.py

The following is a synposis and in-depth analysis of what this file does.

this is a lot of analysis to piece together please look through it before posting it to official documents

Import Statements

  • Import necessary modules and packages including torch, pandas, random, time, uuid, and others.
  • Import bittensor-related classes such as Endpoint, serializer, and synapse.

Logger Configuration

  • Set up the logger for logging purposes.

Global Prometheus

  • Set up Prometheus client with Summary, Counter, Histogram, and CollectorRegistry.

Dendrite Class

  • The Dendrite class is an implementation of bittensor.dendrite() and operates as a torch autograd-friendly operation.
  • It accepts a list of bittensor.endpoints and a list of torch tensors.
  • Implements __init__(), __str__(), __repr__(), and __del__() methods.
  • The forward() method is a static method that handles the autograd-friendly forward RPC call to a list of neuron endpoints.

Dendrite Class (continued)

Backward Method

  • Implements the backward() method as a static and once_differentiable method.
  • This internal autograd-friendly Backward RPC call sends gradients to a list of neuron endpoints.
  • Takes the following arguments: ctx, unused_code_grads, unused_time_grads, and *output_grads.
  • Returns a tuple with optional tensors.

_forward Method

  • Implements the _forward() method to forward tensor inputs to a list of neuron endpoints.
  • Takes the following arguments: endpoints, synapses, inputs, timeout, and requires_grad.
  • Returns a tuple containing outputs, codes, and times.

Prometheus Counters

  • Sets up various Prometheus counters to track performance metrics.
  • Tracks metrics for success and failure rates, response bytes, response params, and latency.
  • Offers additional debugging information with the DEBUG level enabled.

generate

Generates text using provided endpoints, prompt, and various other parameters that control the text generation process.

  • endpoints: The endpoints to send inputs to.
  • prompt: Tokenized sentences to send on the wire.
  • timeout: Request timeout (optional).
  • Various other parameters related to text generation.

Returns a tuple containing the prompt generations produced by endpoints with corresponding parsed codes and query times.

text

Forwards text inputs to a list of neuron endpoints and returns logit encodings or timeouts.

  • endpoints: The endpoints to send inputs to.
  • synapses: Bittensor synapse objects with arguments.
  • inputs: Tokenized sentences to send on the wire.
  • timeout: Request timeout (optional).
  • requires_grad: If true, the backward pass triggers passing gradients on the wire (optional).

Returns a tuple containing outputs from synapses, return codes per call per synapse, and times per call per synapse.

text_causal_lm

Forward text inputs to a list of neuron endpoints and returns logit encodings or timeout.

  • endpoints: Endpoints to send inputs to. Endpoint can be one of the following types:
    • a single endpoint tensor shape [250]
    • a set of endpoint tensors shape [n, 250]
    • a list of endpoints tensors each of shape [250]
    • a single endpoint object. Inputs will be sent to this endpoint alone.
    • a list of endpoint objects. All inputs will be sent to these endpoints.
  • inputs: Tokenized sentences to send on the wire. Inputs can be one of the following types:
    • a single string: the string will be tokenized using the bittensor tokenizer.
    • a list of strings: the strings will be tokenized using the bittensor tokenizer.
    • a tensor with shape [batch_size, sequence_len], assumed to be the output of bittensor tokenizer.
    • a tensor with shape [n, batch_size, sequence_len], the operation will unbind the tensor and pass inputs to endpoints.
    • a list of tensors of type long each representing a tokenized sentence to be sent to each endpoint.
  • synapse: Synapse axon function call which defaults to bittensor.synapse.TextCausalLM().
  • timeout: Request timeout. Queries that do not respond will be replaced by zeros.
  • requires_grad: If true, the backward pass triggers passing gradients on the wire.

Returns:

  • outputs: List of output logit encodings of inputs produced by each remote endpoints. Non-responses are zeroes of input shape plus output dimension. The first dimension will match the number of endpoints queried.
  • codes: dendrite call return ops.
  • times: times per call.

text_causal_lm_next

Forward text inputs to a list of neuron endpoints and returns logit encodings or timeout.

  • endpoints: Endpoints to send inputs to. Endpoint can be one of the following types:
    • a single endpoint tensor shape [250]
    • a set of endpoint tensors shape [n, 250]
    • a list of endpoints tensors each of shape [250]
    • a single endpoint object. Inputs will be sent to this endpoint alone.
    • a list of endpoint objects. All inputs will be sent to these endpoints.
  • inputs: Tokenized sentences to send on the wire. Inputs can be one of the following types:
    • a single string: the string will be tokenized using the bittensor tokenizer.
    • a list of strings: the strings will be tokenized using the bittensor tokenizer.
    • a tensor with shape [batch_size, sequence_len], assumed to be the output of bittensor tokenizer.
    • a tensor with shape [n, batch_size, sequence_len], the operation will unbind the tensor and pass inputs to endpoints.
    • a list of tensors of type long each representing a tokenized sentence to be sent to each endpoint.
  • synapse: Synapse axon function call which defaults to bittensor.synapse.TextCausalLMNext().
  • timeout: Request timeout. Queries that do not respond will be replaced by zeros.
  • requires_grad: If true, the backward pass triggers passing gradients on the wire.

Returns:

  • outputs: List of output topk phrases encodings of inputs produced by each remote endpoints. Non-responses are zeroes of input shape plus output dimension. The first dimension will match the number of endpoints queried.
  • codes: dendrite call return ops.
  • times: times per call.

text_last_hidden_state

The text_last_hidden_state function is designed to forward text inputs to a list of neuron endpoints and return the last hidden state responses or time out.

Arguments

  • endpoints: The endpoints to send the inputs to. This can be a single endpoint tensor, a set of endpoint tensors, a list of endpoint tensors, a single endpoint object, or a list of endpoint objects.
  • inputs: The tokenized sentences to send on the wire. This can be a single string, a list of strings, a tensor with a specific shape, or a list of tensors of type long.
  • mask: An optional mask used to select which hidden states of the sequence are to be returned by the query. This can be a single int, a list of ints, or None.
  • synapse: An optional synapse axon function call that defaults to bittensor.synapse.TextLastHiddenState.

Returns

  • outputs: List of output last hidden state encodings of inputs produced by remote endpoints. Non-responses are zeroes of input shape plus output dimension. The first dimension will match the number of endpoints queried.
  • codes: Dendrite call return ops as a LongTensor.
  • times: Times per call as a FloatTensor.

format_text_inputs

The format_text_inputs function formats endpoint and inputs args to a common format.

Arguments

  • endpoints: Endpoints to send inputs to. Endpoint can be one of the following types:

    • a single endpoint tensor shape [250]
    • a set of endpoint tensors shape [n, 250]
    • a list of endpoints tensors each of shape [250]
    • a single endpoint object. Inputs will be sent to this endpoint alone.
    • a list of endpoint objects. All inputs will be sent to these endpoints.
  • inputs: Tokenized sentences to send on the wire. Inputs can be one of the following types:

    • a single string: the string will be tokenized using the bittensor tokenizer.
    • a list of strings: the strings will be tokenized using the bittensor tokenizer.
    • a tensor with shape [batch_size, sequence_len], assumed to be the output of bittensor tokenizer.
    • a tensor with shape [n, batch_size, sequence_len], the operation will unbind the tensor and pass inputs to endpoints. If inputs are tensors they will be cast to int64 format before sending on the wire.

Returns

  • formatted_endpoints: A list of endpoint objects. All inputs will be sent to these endpoints.
  • formatted_inputs: A list of tensor of type long each representing a tokenized sentence to be sent to each endpoint.