Data Engineer

freeC's Client

Infrastructure Management:

• Design, build, and maintain robust and scalable data pipelines to manage large datasets on both on-premise and cloud platforms (e.g., AWS, GCP, Azure).
• Implement and oversee data storage solutions, including databases and data lakes, to ensure data integrity and optimal performance.

Data Integration:

• Integrate data from a variety of internal and external sources such as databases, APIs, flat files, and streaming data.
• Maintain data consistency, quality, and reliability through comprehensive validation and transformation procedures.

ETL Development:

• Design and develop ETL (Extract, Transform, Load) processes to automate data ingestion, transformation, and loading into data warehouses and lakes.
• Optimize ETL workflows to enhance processing efficiency and minimize data latency.

Data Quality & Governance:

• Implement data quality checks and validation protocols to guarantee data accuracy and completeness.
• Establish data governance frameworks and policies to manage data lifecycle, metadata, and lineage.

Collaboration and Support:

• Collaborate with data scientists, AI engineers, and developers to understand their data requirements and provide technical assistance.
• Promote effective communication and coordination between the AI and data teams and other technical departments.

Continuous Improvement:

• Identify and implement improvements in data infrastructure and pipeline operations.
• Stay informed about the latest industry trends and advancements in data engineering and big data technologies.

Job Requirements

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related discipline. A Master’s degree is an advantage.
• At least 3-5 years of experience in data engineering or a related position.
• Demonstrated experience with both on-premise and cloud platforms (e.g., AWS, GCP, Azure).
• Solid background in data integration, ETL development, and building data pipelines.
• Experience leading the design and creation of high-performance AI and data platforms, including IDEs, permission management systems, data pipelines, code management, and model deployment solutions.
• Proficiency in scripting and programming languages such as Python, SQL, and Bash.
• Strong knowledge of data storage systems and databases, including SQL, NoSQL, and data lakes.
• Experience with big data technologies like Apache Spark and Hadoop.
• Familiarity with CI/CD tools such as Jenkins, GitLab CI, and CircleCI.
• Understanding of data engineering principles and MLOps practices.
• Knowledge of security best practices in data environments.
• Excellent problem-solving abilities and strong attention to detail.

Preferred Qualifications:

• Experience managing on-premise Spark clusters for big data processing, with a focus on both deployment and operational usage.

• Competitive salary package aligned with your skills, experience, and contributions.
• Support with accommodation, meals, and working visa arrangements.
• Holiday and New Year bonuses.
• Provision of essential work equipment, including a MacBook and iPhone.
• A professional, dynamic, and inclusive workplace environment.
• Financial support for home visits twice a year.