feat(logging): add direct Kafka logging support via tracing-kafka crate

[gopikrishna000]

Description

This PR introduces a generic tracing-kafka crate that enables centralized logging to Kafka for any service in the workspace.

Key features:

• Generic, reusable Kafka logging layer with builder pattern API
• Fire-and-forget design ensures logging never blocks the application
• Feature-gated
• Supports static fields (service name, build version) for consistent log enrichment

Motivation and Context

Currently, we use Vector for log collection, which has limitations:

• Log rotation issues causing data loss
• Two-step process (file → Vector → Kafka) adds complexity and latency
• Limited control over log formatting and routing

This change provides direct Kafka integration, giving us:

• Single-step logging (app → Kafka)
• Full control over log format and delivery
• Simplified plug and use crate

Additional Changes

• This PR modifies the API contract

• This PR modifies application configuration/environment variables
  Configuration changes:

• Added Kafka configuration section to config/development.toml

  • New [log.kafka] section with brokers, topic, and enabled flag

• Added optional kafka feature flag in:

  • backend/grpc-server/Cargo.toml - Added tracing-kafka as optional dependency
  • Root Cargo.toml - Added tracing-kafka to workspace members

• Modified logger setup in backend/grpc-server/src/logger/setup.rs

  • Added Kafka layer initialization when feature is enabled

• Added new logger config types in backend/grpc-server/src/logger/config.rs

  • Added KafkaConfig struct for Kafka-specific settings

How did you test it?

Start Kafka locally

# Start Zookeeper
docker run -d --name zookeeper-test -p 2181:2181 \
  -e ZOOKEEPER_CLIENT_PORT=2181 \
  -e ZOOKEEPER_TICK_TIME=2000 \
  confluentinc/cp-zookeeper:7.5.0

# Start Kafka
docker run -d --name kafka-test -p 9092:9092 \
  --link zookeeper-test:zookeeper \
  -e KAFKA_BROKER_ID=1 \
  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  confluentinc/cp-kafka:7.5.0

Run with Kafka logging

cargo run --features kafka

Consume logs

docker exec kafka-test kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic connector-service-logs \
  --from-beginning

Log from server

Log from local kafka topic