Vostok.Tracing.Abstractions

Vostok tracing

Distributed tracing allows to reconstruct the history of the logical operation spanning many applications and machines in time as a tree of smaller actions or events called spans. Spans can represent HTTP requests, database queries or any other significant interactions or events in a distributed system. A single span always describes a local event in a single process: an HTTP request usually produces two spans (client-side and server-side). Each kind of span stores specific information about performed action.

• Span structure
• Common annotations
• Kind-specific annotations
  • HTTP requests
    • HTTP client (direct)
    • HTTP client (cluster)
    • HTTP server
  • Custom (not HTTP) requests
    • Custom (not HTTP) client (direct)
    • Custom (not HTTP) client (cluster)
  • Custom operations
  • Database requests
    • Cassandra
    • MongoDB
    • ElasticSearch
    • MS SQL
  • Distributed task queues
    • Queue (producer)
    • Queue (task-lifecycle)
    • Queue (task-lifecycle-event)
    • Queue (consumer)
    • Queue (manager)
    • Custom events
• Queue tracing conventions

Span structure

Every span consists of following fields:

• TraceId — unique identifier of the trace containing the span (Guid).

  • Gets assigned on first operation (usually on a front-end application instance).
  • Serves as a correlation identifier between spans

• SpanId — unique identifier of the span itself (Guid).

• ParentSpanId — identifier of the parent span in the tree (Guid).

  • May be absent for root span in the tree.

• BeginTimestamp — beginning timestamp of the event or interaction described by this span (UTC timestamp + timezone offset).

• EndTimestamp — ending timestamp of the event or interaction described by this span (UTC timestamp + timezone offset).

  • Always measured with the same clock as BeginTimestamp. This allows to derive span duration as a difference between EndTimestamp and BeginTimestamp.
  • May be absent for a special kind of 'endless' spans described further.

• Annotations — payload in key-value format (string --> object). Keys are case-sensitive.

Common annotations

These are the annotations relevant for any span:

Name	Description
kind	Span kind. There are a number of predefined span kinds for common use cases (e.g. http-request-server).
operation	Human-readable logical operation or event name (e.g. create-user).
status	Logical operation or event status (success, error, or warning). Might not have operation-specific values.
application	Name of the application the span originated from.
environment	Name of the environment the span originated from.
host	DNS name of the host the span originated from.
component	Name of a library or component in code responsible for producing the span.

Kind-specific annotations

HTTP requests

Common annotations for all spans related to HTTP requests:

Name	Description	Default value
operation	See common annotations.	{http.request.method} {normalized http.request.url}. Example: POST /page/{num}/process
http.request.method	Request method (e.g. GET, POST, PUT, etc).	N/A
http.request.url	Request URL without query parameters.	N/A
http.request.size	Request body size in bytes.	N/A
http.response.code	Response code (e.g. 200 or 404).	N/A
http.response.size	Response body size in bytes.	N/A

Normalized URL is a short URL without scheme, authority and query parameters. Unique path segments (entity ids, search queries, hex values) are replaced with placeholders. Example before and after normalization: http://vm-app1/users/a534bcbd/ --> users/{id}

HTTP client (direct)

Submitting an HTTP request directly to an external URL or a service replica.

Name	Description	Default value
kind	See common annotations.	http-request-client
http.request.targetService	Name of the service to which request is sent.	N/A
http.request.targetEnvironment	Name of the environment to which request is sent.	N/A

HTTP client (cluster)

Submitting an HTTP request to a clustered application with several replicas.

Name	Description	Default value
kind	See common annotations.	http-request-cluster
http.cluster.strategy	Name of the strategy used to send request (e.g. sequential, parallel, ...)	N/A
http.cluster.status	Status of interaction with a cluster (e.g. success, no-replicas, ...)	N/A
http.request.targetService	Name of the service to which request is sent.	N/A
http.request.targetEnvironment	Name of the environment to which request is sent.	N/A

HTTP server

Handling an HTTP request on server.

Name	Description	Default value
kind	See common annotations.	http-request-server
http.client.name	Name of the client application that sent the request.	N/A
http.client.address	Address of the client application instance (host name or IP address).	N/A

Custom (not HTTP) requests

Common annotations for all spans related to custom requests:

Name	Description	Default value
operation	See common annotations.	N/A
status	See common annotations.	N/A
custom.response.status	Custom request-specific status.	N/A
custom.request.size	Request size in bytes.	N/A
custom.response.size	Response size in bytes.	N/A
custom.request.targetService	Name of the service to which request is sent.	N/A
custom.request.targetEnvironment	Name of the environment to which request is sent.	N/A

Custom (not HTTP) client (direct)

Submitting custom request directly to a service replica.

Name	Description	Default value
kind	See common annotations.	custom-request-client
custom.request.replica	Name of the replica to which request is sent.	N/A

Custom (not HTTP) client (cluster)

Submitting custom request to a clustered application with several replicas.

Name	Description	Default value
kind	See common annotations.	custom-request-cluster

Custom operations

Performing custom server operation.

Name	Description	Default value
kind	See common annotations.	custom-operation
operation	See common annotations.	N/A
status	See common annotations.	N/A
custom.operation.status	Custom operation-specific status.	N/A
custom.operation.size	Processed data size in bytes.	N/A
custom.operation.targetService	Name of the service with which this operation is associated.	{application}
custom.operation.targetEnvironment	Name of the environment with which this operation is associated.	{environment}

Database requests

Submitting a request to database.

Name	Description	Default value
kind	See common annotations.	db-request
operation	See common annotations. Possible examples: read from table X, insert into table Z.	N/A
db.type	Database type (mssql, cassandra, mongodb, elastic, etc).	N/A
db.instance	Address of the database server instance (URL, hostname, connection string).	N/A
...	Any other database-specific annotations.	N/A

MS SQL

TODO

Cassandra

TODO

MongoDB

TODO

ElasticSearch

TODO

Distributed task queues

See queue tracing conventions for more context.

Common annotations for all spans related to distributed task queues:

Name	Description	Default value
queue.type	Queue type (rabbit, etc).	N/A
queue.topic	Name of the task type or the topic/queue it was inserted to.	N/A
queue.taskId	Task unique identifier.	N/A

Queue (producer)

A span that represents insertion of a task to queue (from the producer standpoint). Note that this span should not be in the same trace with subsequent spans related to inserted task: instead, it's connected with dedicated task trace with queue.taskTraceId annotation.

Name	Description	Default value
kind	See common annotations.	queue-producer
operation	See common annotations.	({queue.type}) Put to '{queue.topic}'. Example: (echelon) Put to 'reports'.
queue.actionResult	Result of action (success, error, timeout or something else).	N/A
queue.taskTraceId	Trace identifier assigned to the task.	N/A

Queue (task-lifecycle)

A span that represents whole lifecycle of the task in queue. It serves as a root span in the task's personal trace.

Spans of this kind are special in two ways:

1. They are non-local: task lifecycle spans do not directly relate with events happening inside any single process.
2. They do not have an ending timestamp: it must be inferred from the rightmost end timestamp of all the child spans.

Name	Description	Default value
kind	See common annotations.	queue-task-lifecycle
queue.producerTraceId	Trace identifier of the producer's (client app that created the task) request.	N/A

Queue (task-lifecycle-event)

A span that represents an event that somehow changes task state.

Such spans usually have zero duration and are produced by brokers or client libraries.

Name	Description	Default value
kind	See common annotations.	queue-task-lifecycle-event
operation	See common annotations. Examples: pass-to-consumer, prolong-execution, ...	N/A
queue.externalTraceId	Trace identifier of the client app request that triggered the event (if any).	N/A
...	Any other operation-specific annotations.	N/A

Queue (consumer)

A span that represents execution of a queued task on the consumer.

Name	Description	Default value
kind	See common annotations.	queue-consumer
queue.executionResult	Result of task execution (success, error, etc). Might have queue-specific values.	N/A

Queue (manager)

A span that represents a management action on the task, but from the client's standpoint, as opposed to queue-task-lifecycle-event spans. Note that this span should not be in the same trace with subsequent spans related to managed task.

Name	Description	Default value
kind	See common annotations.	queue-manager
operation	See common annotations. Examples: delete, prolong-execution, ...	N/A
queue.actionResult	Result of action (success, error, timeout or something else).	N/A

Queue tracing conventions

Following conventions apply to tracing of queue tasks:

• Each task should have a dedicated trace with a root span of queue-task-lifecycle kind. This span does not have an explicit end timestamp. It gets "stretched" by child spans instead.

• Each task should contain its personal traceId and root span's spanId embedded in content, available for consumers.

• Root lifecycle spans can have child spans of queue-task-lifecycle-event and queue-consumer kinds. External operations from producers and management clients happen in their own traces.

• A span of queue-task-lifecycle-event kind can have a link to traceId of external operation that triggered the event.

• A span of queue-producer kind should have a link to traceId of the produced task.

If a queue has no broker or there's no access to broker code, its tracing duties have to be performed by client libraries (creation of queue-task-lifecycle and queue-task-lifecycle-event spans).