Systems Integration Advisor

Technical Lead with 10+ yrs of experience as well as expertise in data engineering and analytics in

• In-depth knowledge of Apache Spark, Pyspark and Databricks architecture.  
• Strong programming skills in languages such as Python or Scala.  
• Experience with data engineering concepts, ETL processes, and data modeling.  
• Proven experience as a Technical Lead with a focus on Databricks and big data technologies on Azure Cloud Platform.  
• Design and implement scalable and efficient data processing architectures using Databricks.
• Handson experience in writing python code, pyspark notebooks and Azure Cloud, Configure Databricks clusters and ensure adherence to best practices and coding standards
• Make strategic decisions regarding data organization, storage, and processing. 
• Lead and mentor a team of data engineers and data scientists.
• Provide technical guidance and support to ensure the team's success. 
• Oversee the implementation of data processing workflows, analytics, and machine learning models using Databricks.
• Optimize the performance of big data processing tasks and analytics jobs.
• Collaborate with cross-functional teams, including data science, business intelligence, and infrastructure teams.
• Ensure seamless integration of data pipelines with various systems. 
• Ensure consistency, quality, and maintainability of the codebase. 
• Optimize queries and resolve technical challenges during development and production. 
• Manage the technical aspects of projects, including timelines and resource allocation.
• Coordinate with stakeholders to ensure project goals are met. 
• Excellent leadership and communication skills.