Data Ingestion tool for Modern Data

User interface web solution to automate data ingestion process/pattern for different source system to different target system.

Solution

• Designed and built 5 screens (Monitor, Ingest, Patterns, Connections, Settings) using React,NextJs, FastAPI and Python
• Built pre and custom connector to export/import data from different type of system (file, Database)
• Built code using python to generate dynamic Ingestion DAGS (such as SQL queries, connectors and pattern) in Airflow based on user inputs
• Implemented OIDC for multiple repositories
• Implemented dynamic rendering with configuration
• Scheduled Spark job to transfer data between different data sources using airflow
• Implemented Audit, Balance and Control to log each of the tasks in airflow to MongoDb