Senior Data Engineer | 8+ years Experience | PySpark, SQL, Python, Hadoop, Azure databricks, ADF
Actively Looking for Work
SUMMARY:
Results-driven Data Engineer with 8+ years of experience, including 4 years in Big Data and cloud-based Data Engineering and 4 years as a Database Developer. Hands-on experience with modern data platforms including Azure Data bricks, Azure Data Factory, and Apache Spark, with a strong focus on scalable, secure, and high-performance data solutions.
Skills:
- Spark
- Azure Data Factory
- Azure DataLake
- AzureDatabricks
- Pyspark
- SQL
- Python
- PLSQL
- UNIX
- Hadoop
- Hive
- DataGovernance
- Migration
- Performance Tuning
- Data Ingestion
- Data Lake
- Data Modeling
WORK EXPERIENCE:
Senior Associate at Cognizant, Chennai
Apr 2021 - Present & Project : Cummins
-
Designed and maintained end-to-end ETL pipelines using ADF to orchestrate data ingestion, transformation, and loading across Azure SQL Database, Data Lake Storage.
-
Optimized PySpark ETL jobs processing large datasets by configuring Spark parameters, implementing effective data partitioning, and using caching strategies, resulting in a 50% improvement in job performance.
-
Established multi-layered data lakehouse architecture using Azure Databricks and Delta Lake, streamlining ETL workflows and improving data reusability across teams.
-
Implemented data partitioning and bucketing in PySpark for optimal query performance on large datasets, improving job execution speed and minimizing shuffle operations.
Associate Consultant at Capgemini, Chennai
Senior Associate at Cognizant, Chennai
Designed and maintained end-to-end ETL pipelines using ADF to orchestrate data ingestion, transformation, and loading across Azure SQL Database, Data Lake Storage.
Optimized PySpark ETL jobs processing large datasets by configuring Spark parameters, implementing effective data partitioning, and using caching strategies, resulting in a 50% improvement in job performance.
Established multi-layered data lakehouse architecture using Azure Databricks and Delta Lake, streamlining ETL workflows and improving data reusability across teams.
Implemented data partitioning and bucketing in PySpark for optimal query performance on large datasets, improving job execution speed and minimizing shuffle operations.
Jan 2019 – Mar 2021 & Project: SCB bank
-
Automated external data retrieval from web services between upstreams and downstream directly using SOAP calls and reducing manual data entry by 80%.
-
Upgraded over 100+ oracle forms from version 10G to 12C as part of legacy ERP modernization effort and achieving 0 post-deployment defects and 99.9% uptime during and post migration.
-
Led the migration of on-prem Oracle 12C database to OCI, reducing the infrastructure cost by 30%.
-
Analysed execution plan to identify slow running queries and implemented indexing strategies.
Senior Process Associate at TCS, Chennai
Aug 2016 – Jan 2019 & Project: FCIB Bank
-
Analyzed business requirements and prepared technical design and impact analysis documents.
-
Led POCs for migrating oracle database 12C to PostgreSQL, evaluating schema conversion, PLSQL compatibility using ora2pg with 50% save in licence cost post migration.
-
Tuned complex SQL queries, reducing execution time by 40% and improving dashboard performance.
ACHIEVEMENTS:
Analyzed business requirements and prepared technical design and impact analysis documents.
Led POCs for migrating oracle database 12C to PostgreSQL, evaluating schema conversion, PLSQL compatibility using ora2pg with 50% save in licence cost post migration.
Tuned complex SQL queries, reducing execution time by 40% and improving dashboard performance.
Cummins Ideathon Award (11/2024 - Present) Awarded 2nd prize among 500 participants for a process improvement that made a significant impact on a project.
Impact Award (04/2023-Present) Awarded as part of a high-performing team for optimizing ETL pipeline, significantly improving processing efficiency and reliability.
Star Award (03/2020 - Present) Awarded with Top impactor of the quarter award for creating an impact while achieving Customer success and growth for the company.
EDUCATION:
Sri Ramakrishna Institute of Technology, Coimbatore 2010 – 2014 B.E(ECE)