Career Details

About this Role

This role is accountable for running day to day operations of Data Platform in Azure / AWS Databricks NCS. Data Engineer is accountable for ongoing Development, Enhancement support and maintenance data availability and data quality, performance enhancement and stability of the system.

Job Description

About this role:

This role is accountable for running day to day operations of Data Platform in Azure / AWS Databricks NCS. Data Engineer is accountable for ongoing Development, Enhancement support and maintenance data availability and data quality, performance enhancement and stability of the system.

• Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks 

• Ensure data pipelines run smoothly and efficiently 

• Adherence to security, regulatory and audit control guidelines

• Drive optimization, continuous improvement and efficiency 

Qualification: 

• Bachelor’s degree in IT, Computer Science, Software Engineering, Business Analytics or equivalent.   

• Relevant accredited certifications for Azure, AWS, Cloud Engineering and/or Databricks.

Key Accountabilities & Responsibilities: 

• Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks

• Ensure data pipelines run smoothly and efficiently

• Developing scalable and re-usable frameworks for ingesting of data sets

• Integrating the end to end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times

• Working with event based / streaming technologies to ingest and process data

• Working with other members of the project team to support delivery of additional project components (API interfaces, Search)

• Working with other members of the project team to support delivery of additional project components (API interfaces, Search)

• Evaluating the performance and applicability of multiple tools against customer requirements

• Technically competent in Cloud and Databricks to provide technical advice to the team and to be involved in issue resolution

• Provide on-call support and after hours/weekend support as needed

• Fulfill Service Requests related to Data Analytics platform

• Lead and drive optimization and continuous improvement initiatives

• Play gate-keeping role and conduct technical review of the changes as part of release management

• Understand various data security standards and adhere to the required data security controls in the platform

• Minimum three years of experience in data analytics fields

• Experience with Azure/AWS Databricks

• Experience in building and optimizing data pipelines, architectures and data sets 

• Excellent experience in Scala or Python, PySpark and SQL

• Ability to troubleshoot and optimize complex queries on the Spark platform 

• Knowledgeable on structured and unstructured data design / modelling, data access and data storage techniques 

• Expertise in designing and deploying data applications on cloud solutions, such as Azure or AWS

• Hands on experience in performance tuning and optimizing code running in Databricks environment

• Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment

Technical / Professional Skills:

Please provide at least 3

• Azure/AWS Databricks

• Python / Scala / Spark / PySpark 

• HIVE / HBase / Impala / Parquet

• Sqoop, Kafka, Flume 

• SQL and RDBMS 

• Airflow 

• Jenkins / Bamboo 

• Github / Bitbucket 

• Nexus 

• Data Engineering expertise in Big Data Management using Cloudera, Python, Hive, Scala, Data Warehouse, Data Lake, AWS, Azure

• SQL, Python, ETL, pyspark, Data Engineer, Databricks . Excellent Comm Skills (No Compromise), Focus is SQL- 90% strong exposure

Education & Experience: 8+ years

Compensation:(depending upon the interview)

Location: Work from Office Hinjewadi, Pune.