Skip to main content

​
Prometheus Integration

Export Visent Telemetry GPU metrics to Prometheus for integration with existing monitoring infrastructure. Configure metric exporters, set up alerting rules, and visualize GPU performance alongside other system metrics. Supports both push and pull-based metric collection with configurable metric retention and aggregation.

​
Prerequisites

Coming soon - Prometheus server setup and configuration requirements.

​
Configuration

​
Prometheus Exporter

# visent-telemetry.yml
prometheus:
  enabled: true
  port: 9090
  path: /metrics
  interval: 30s
  labels:
    environment: production
    cluster: gpu-cluster

​
Metric Export

# Enable Prometheus export
visent telemetry config prometheus --enable

# Configure export endpoint
visent telemetry config prometheus --endpoint localhost:9090

# Test metric export
visent telemetry prometheus test

​
Exported Metrics

Coming soon - complete list of GPU metrics exported to Prometheus.

​
Core GPU Metrics

# GPU utilization percentage
visent_gpu_utilization{gpu="0", node="worker-1"} 85.2

# GPU memory usage in bytes
visent_gpu_memory_used{gpu="0", node="worker-1"} 16106127360

# GPU temperature in Celsius
visent_gpu_temperature{gpu="0", node="worker-1"} 72.5

​
Custom Metrics

Coming soon - configuring custom metric exports and labels.

​
Alerting Rules

​
Basic Alert Rules

groups:
  - name: visent-gpu-alerts
    rules:
      - alert: HighGPUTemperature
        expr: visent_gpu_temperature > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU temperature is high"
          description: "GPU {{ $labels.gpu }} temperature is {{ $value }}°C"

​
Advanced Alerting

Coming soon - complex alerting rules and notification configuration.

​
Service Discovery

Coming soon - automatic discovery of Visent Telemetry exporters.

​
Performance Optimization

Coming soon - optimizing metric export performance for large deployments.

​
Troubleshooting

Coming soon - diagnosing Prometheus integration issues.

​
Next Steps