Required Skills
About the Job
Infosys is seeking an experienced Azure Databricks Administrator and Data Engineer to join our team in Bengaluru, Karnataka. In this role, you will be responsible for designing, developing, and maintaining robust data pipelines using Azure Databricks. You will implement ETL/ELT workflows to process large-scale structured and unstructured data, and optimize Spark jobs for performance, scalability, and cost-efficiency.
Key Responsibilities: * Design, develop, and maintain data pipelines using Azure Databricks. * Implement ETL/ELT workflows for processing large-scale structured and unstructured data. * Optimize Spark jobs for performance, scalability, and cost. * Work with Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, and Azure SQL. * Develop and manage Databricks notebooks using PySpark, Scala, or SQL. * Integrate Databricks with other Azure services such as Azure Data Factory, Azure Synapse Analytics, and Azure Key Vault. * Implement data quality, validation, and monitoring frameworks.
Required Skills & Experience: * Strong hands-on experience with Azure Databricks. * Proficiency in Apache Spark (PySpark/Scala). * Experience with Azure Data Lake (ADLS Gen2) and cloud storage. * Strong SQL skills and experience with large datasets. * Experience with Azure Data Factory (ADF) for orchestration. * Knowledge of data modeling, partitioning, and performance tuning. * Experience working with big data frameworks.
