odigos

open-source

gcs

googlecloud

opentelemetry

Sending OpenTelemetry data to Google Cloud Storage

Learn how to send OpenTelemetry data to Google Cloud Storage using Odigos, improving your data storage and analysis capabilities.

Sending OpenTelemetry data to Google Cloud Storage
Author
/eden.jpgEden Federman
Feb 16 2023

In this blog post, we will show you how to send OpenTelemetry data to Google Cloud Storage using Odigos.

Since version v0.1.4, Odigos supports sending OpenTelemetry data to Google Cloud Storage (GCS). For the first time, you can use storage services like GCS to store your OpenTelemetry data, and build custom solutions on top of it.

Creating a GCS bucket

First, you need to create a GCS bucket. You can do it using the Google Cloud Console or using the gsutil command line tool. For example, to create a bucket named otel-data in the us-central1 region, run the following command:

gsutil mb -p <project-id> -c regional -l us-central1 gs://otel-data

Creating a GKE cluster

Next, you need to create a GKE cluster with permissions to write to GCS. You can do it using the Google Cloud Console or using the gcloud command line tool. For example, to create a cluster named otel-cluster in the us-central1 region, run the following command (replace PROJECT_ID with your project ID):

gcloud container clusters create otel-cluster \
--region us-central1 \
--workload-pool=PROJECT_ID.svc.id.goog

Installing demo application

For this demo, we will use the emojivoto application. You can install it by running the following command:

kubectl apply -k kubectl apply -k github.com/keyval-dev/opentelemetry-go-instrumentation/docs/getting-started/emojivoto

Installing Odigos

After installing the demo application, you need to install Odigos to send data to GCS. The simplest way to install Odigos is via the helm chart by executing the following commands:

helm repo add odigos https://keyval-dev.github.io/odigos-charts/
helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

Granting permissions using workload identity

To grant permissions to the cluster, you need to create a service account and bind it to the cluster. You can do it using the Google Cloud Console or using the gcloud command line tool. For example, to create a service account named otel-sa and bind it to the cluster, run the following commands:

gcloud iam service-accounts create otel-sa
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/storage.objectCreator \
  --member "serviceAccount:otel-sa@<project-id>.iam.gserviceaccount.com" \
  otel-sa@<project-id>.iam.gserviceaccount.com

Finally, you need to bind the odigos-gateway service account to the GCP service account, run the following command:

kubectl annotate serviceaccount -n odigos-system odigos-gateway \
  iam.gke.io/gcp-service-account=otel-sa@<project-id>.iam.gserviceaccount.com

Sending data to GCS

The last step is to configure Odigos to send data to GCS. This can be done easily by accessing the Odigos UI and creating a new Google Cloud Storage destination. Make sure to specify the bucket name you created earlier.

Summary

In this blog post, we showed you how to send OpenTelemetry data to Google Cloud Storage using Odigos, an open-source observability control plane. Odigos supports more that 20 different destinations, including cloud providers like Azure and Google Cloud, as well as open-source solutions like Loki, Elasticsearch, and Prometheus.

If you want to learn more about how you can generate distributed traces instantly check out our GitHub repository. We'd really appreciate it if you could throw us a ⭐👇
https://github.com/keyval-dev/odigos

Related posts

Automatic Instrumentation As A Kubernetes Virtual Device

Automatic Instrumentation As A Kubernetes Virtual Device

Explore the various methods for extending containers, and learn why Odigos opts for Kubernetes Device Plugins to enhance container functionality.

Automatic Instrumentation As A Kubernetes Virtual Device

Eden Federman

Feb 28 2023

Faster and Better - Introducing the New Odigos Language Detector

Faster and Better - Introducing the New Odigos Language Detector

Discover the latest enhancements in language detection within Odigos, including improved speed and efficiency.

Faster and Better - Introducing the New Odigos Language Detector

Eden Federman

May 17 2023

Combining Odigos and Jaeger for instant distributed tracing

Combining Odigos and Jaeger for instant distributed tracing

This article explains how to simplify distributed tracing using Odigos and Jaeger for improved observability in microservices-based applications.

Combining Odigos and Jaeger for instant distributed tracing

Eden Federman

Apr 12 2023