Senior Data Engineer

  • Bengaluru
  • Acl Digital

Work Location:
Remote

Experience:
4+ Years

Please share your updated profile to sugantha.krishnan@acldigital.com

Notice Period:
Lessthan 15 days


Project Objective:

Build a robust and scalable data platform by integrating data from multiple sources, enabling seamless access and utilization by various teams.

Project Responsibilities:

  • Design, develop, and maintain scalable and efficient data platforms.
  • Develop and sustain data pipelines using Python and big data frameworks.
  • Employ Data Lake technologies for the effective management and orchestration of data workflows.
  • Work with diverse data formats to ensure optimal storage and retrieval.
  • Continuously monitor, optimize, and troubleshoot data pipelines to maintain peak performance.
  • Demonstrate strong problem-solving and analytical skills.
  • Collaborate effectively with cross-functional teams through clear communication.


Technical Requirements:

Core Skills:

  • Experience in software or app development using Python.
  • Proficiency in Big Data frameworks and handling large datasets.


Good to Have:

  • Programming experience:
    Spark/PySpark, Kafka
  • Data Lake and Workflow Management:
    AWS S3, AWS Glue, Delta Lake, AWS Lake Formation, Apache Airflow, Step Functions
  • Data Formats and Storage:
    Parquet, Avro, Iceberg, RDF
  • Advanced Data Technologies:
    Knowledge Graphs, AWS Neptune, Embedding Stores, Vector Databases