Data Engineer (Contract) - Google Cloud & Python

About the role

InTalent Asia, a recruitment partner in Sri Lanka, is recruiting on behalf of its client for a Data Engineer on a contract basis. The role covers designing and building scalable, enterprise-grade data-processing solutions, including ETL/ELT pipelines and cloud-native data applications.

Key responsibilities

• Design and develop scalable, enterprise-grade data-processing solutions and build distributed, highly available data applications.
• Build, test and maintain data pipelines connecting source systems to target platforms, and develop ETL and ELT workflows for batch and near-real-time data.
• Process, transform and validate large volumes of structured and unstructured data.
• Develop cloud-native solutions on Google Cloud Platform (BigQuery, Dataflow, Dataproc, Datastream, Pub/Sub, Cloud Functions, Cloud Run, Cloud Composer) and use AWS services (Glue, Lambda, EMR, Data Pipeline) when needed.
• Use Airflow and Cloud Composer for data orchestration; build apps in Python, shell scripting and SQL.
• Work with relational, NoSQL and distributed storage databases, and streaming tools such as Kafka, Spark or Pub/Sub.
• Contribute to data warehouses, data marts, data lakes and data-mesh solutions, and follow software-engineering best practices.
• Take part in code reviews, technical design and architecture discussions, and monitor pipeline health and reliability, investigating failures and incidents.

Required profile

• Bachelor's degree in Computer Science, Software Engineering, Information Technology or an equivalent field.
• 1 to 2 years of experience developing enterprise-grade data-processing applications.
• Hands-on experience with large volumes of data and with ETL/ELT pipelines.
• Experience with relational, NoSQL and distributed storage databases.
• Hands-on experience with Google Cloud Platform and its services; familiarity with Airflow.
• Advantageous: AWS data services, streaming technologies (Kafka, Spark Streaming), source control and CI/CD tools (GitHub, GitLab, Jenkins), and monitoring tools (Datadog).