Senior Data Engineer
NO Comp · Colombo
Job description
About the role
This Senior Data Engineer position leads the design, build and production of scalable ETL/ELT pipelines and big data environments, while also mentoring junior and mid-level data engineers. The role covers backend data-serving APIs, orchestration, infrastructure and MLOps.
Key responsibilities
- Architect and build highly scalable, fault-tolerant ETL/ELT pipelines from scratch and oversee them to production.
- Design and manage open-source big data environments using Hadoop, Apache Spark, Kafka and NiFi for high-throughput, real-time and batch processing.
- Lead backend development for data serving with Python FastAPI and Spring Boot, securing APIs with OAuth2.
- Use Apache Airflow for complex workflow orchestration and dbt for data transformation.
- Deploy and manage containerized applications with Docker and Kubernetes, using Linux command-line skills and network configuration.
- Design infrastructures supporting MLOps practices for model deployment and monitoring, and handle complex time-series data storage and analysis.
- Mentor junior and mid-level data engineers through technical guidance and code reviews.
Required profile
- Minimum 5+ years of hands-on data engineering experience.
- Ability to lead a small team and guide junior engineers.
- Follows best practices for production-ready setups and uses efficient code design patterns.
- Knowledge of multithreading and Git version control.
Required skills
- Strong Python; FastAPI and Spring Boot.
- Hadoop, Apache Spark, Kafka, Airflow and dbt (mandatory).
- Advanced SQL, including DB2 SQL and PostgreSQL; experience with IBM AS400 is advantageous.
- Docker, Kubernetes, Linux CLI, network configuration and OAuth2.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Sri Lanka.
Salary: Senior DevOps Engineer Based on 11 job offers in Sri LankaSalaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 6 days ago
Expires 1 month from now
12 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
NO Comp
Colombo
Related job offers
-
Senior QA Automation Engineer (Playwright)
Gallagher Colombo -
Senior QA Engineer - Playwright Automation
GSS HR Solutions Private Limted Colombo -
Senior QA Engineer (AI Testing)
Gallagher Colombo -
IT Procurement Executive (Night Shift)
TECEZE District de Jaffna -
Senior QA Lead - Banking Integration & Migration Testing
Schemax Export Techno Craft Pvt. Ltd Province de l'Ouest