Smart Data Injector

A Data Injection Pipeline tool that uses machine learning (LLM Model) for source to target association mapping and data injection. It utilizes Kestra for orchestrating workflows such as s3 triggers, db injections, ETL Services for data transformation and loading.

The Idea

One-shot

The workflow diagram

Requirements

Golang installed on your system.

• To run the backend, create .env from .env.example use command:

  go mod tidy
  go run ./cmd
  

• To start the Kestra platform locally, have docker setup in your system, and then use this script:

  source run_kestra.sh 
  

  • On your browser open localhost:8080/
  • Create New Flow
  • copy the flow configuration from here
  • add your AWS, MONGODB and GEMINI keys into variables.
  • Save the flow.

• To run the client in dev mode, create .env in /client directory and use pnpm package manager:

  cd client
  pnpm install
  pnpm dev
  

  • On your browser open localhost:3000/

Execution

1. Go for generate API key.

2. Use the API Key for data injection

3. Once completed, proceed to analytics dashboard:

Pipeline task execution log format