Technical competence/skills

Python

SQL

Apache Airflow

PostgreSQL / MySQL

Git / GitHub / GitLab

Tasks and job roles

Designing and Architecting Data Pipelines:

Data Modeling:

ETL/ELT Development: Design, implement, and optimize ETL/ELT workflows to support both batch and real-time data ingestion from diverse sources.

Optimizing Data Storage and Performance: Analyze and improve data storage architecture to enhance performance, reduce latency, and support growing data volume. Leverage indexing, partitioning, caching, and other optimization strategies to ensure efficient querying and scalability.

Implementing Data Quality and Governance Practices:

Collaborating with Data Scientists and Analysts: Work closely with data scientists, analysts, and business stakeholders to ensure data availability, clarity, and usability. Translate analytical and reporting requirements into technical data solutions and pipelines that enable insight generation and AI/ML model development.

Automation and Monitoring: Implement automated workflows for data processing, testing, and deployment. Set up real-time monitoring tools and alerting systems to proactively identify and resolve issues in data pipelines, ensuring high availability and reliability.

Data Security and Privacy:

Documentation and Knowledge Sharing:

Infrastructure Management: Oversee and maintain the data engineering infrastructure, including scheduling tools (e.g., Airflow), data repositories, CI/CD pipelines, and development environments. Coordinate with IT and DevOps teams to ensure system performance, version control, and disaster recovery readiness.