Required Skills
About the Job
Digital Impetus is seeking a Lead Data Engineer with 6-11 years of experience to join our team in Bengaluru/Bangalore, Karnataka. In this role, you will be instrumental in designing and developing robust, scalable data pipelines and ETL/ELT workflows using Databricks, Apache Spark, and PySpark. You will leverage Delta Lake for efficient data management and collaborate closely with data scientists to productionize machine learning models. Your responsibilities will include building and optimizing ML pipelines, ensuring data quality and governance, and integrating Databricks solutions with major cloud platforms like AWS, Azure, and GCP. You will also contribute to CI/CD automation for data and ML workflows and be responsible for monitoring and troubleshooting performance. Strong SQL and Python skills are essential, along with a solid understanding of data modeling and warehousing concepts. Experience with ML libraries and workflow orchestration tools is highly valued. This is an excellent opportunity to work in a fast-paced, agile environment and contribute to cutting-edge data solutions.
