DATABRICKS ENGINEER

Company Description iVedha Inc. is a leading global cloud managed services provider and managed hybrid cloud services provider for enterprises. Based in Canada, iVedha serves clients in the USA, Mexico, and worldwide, offering innovative solutions for digital transformation needs across industries. As a trusted professional services partner, iVedha empowers businesses through disruptive technologies to Migrate, Modernize, Secure, Manage, and Monetize their data. As a Databricks Data Engineer, you will play a key role in designing, implementing, and maintaining data pipelines on the Databricks platform that support business and technology objectives. Your primary focus will be on building scalable and efficient data ingestion, transformation, and processing solutions to support our data-driven initiatives. You enjoy working both autonomously and as part of a team and have the confidence to influence and communicate with stakeholders at all levels, and to work in a fast-paced complex environment. About the role: You will be working on client projects with teams from across Data Platform and Applied Intelligence practice alongside industry, functional and technology Subject Matter Experts on challenging projects.You will be involved in the design and implementation of complex data solutions ranging from batch to streaming and event-driven architectures, across cloud, on-premise and hybrid client technology landscapes. We are looking for 3+ years of experience in data engineering in a customer or business facing capacity and experience in the following: Preferred •

Ability to understand and articulate requirements to technical and non-technical audiences •

Stakeholder management and communication skills, including prioritizing, problem solving and interpersonal relationship building •

Experiences with design, develop, and implement end-to-end data engineering solutions using Databricks for large-scale data processing and data integration projects. •

Build and optimize data ingestion processes from various sources, ensuring data quality, reliability, and scalability. •

Perform data transformation tasks, including data cleansing, aggregation, enrichment, and normalization, using Databricks and related technologies. •

Monitor and troubleshoot data pipelines, identifying and resolving performance issues, data quality problems, and other technical challenges. •

Implement best practices for data governance, data security, and data privacy within the Databricks environment. •

Strong SQL, Python, PySpark knowledge •

Collaborate with DevOps and infrastructure teams to optimize the performance and scalability of Databricks clusters and resources. •

Provide guidance and mentorship to junior data engineers, fostering a culture of knowledge sharing and continuous learning within the team. Desirable •

Knowledge in DataOps and experience in delivering CI/CD and DevOps capabilities in a data environment •

Experience with advanced analytics and machine learning frameworks such as Apache Spark MLlib, TensorFlow, or PyTorch. •

Proficiency in data visualization tools such as Tableau, Power BI, or Looker •

Certification in Databricks Engineer Professional would be a plus.