26 việc làm data engineer

JOB DESCRIPTION

1.    Partner Collaboration and Template Management

●      Work directly with business partners to understand their data transformation requirements

●      Design and implement flexible NiFi templates that accommodate partner-specific data formats

●      Manage template versioning and changes based on partner requirements

●      Document template specifications and maintain template change history

●      Provide technical support and guidance to partners during template implementation

●      Conduct regular reviews with partners to optimize template performance

●      Handle template modifications and updates efficiently to minimize disruption

2.    Data Pipeline Development

●      Design, develop, and maintain data pipelines using Apache NiFi for data ingestion, transformation, and distribution

●      Create and manage NiFi templates for standardized data processing workflows

●      Implement robust data synchronization solutions between AWS S3 and Google Cloud Storage

●      Develop and optimize PostgreSQL-based data transformation processes

●      Ensure data quality, consistency, and reliability across all systems

3.    Template Development and Management

●      Design modular and reusable template components

●      Implement template validation and testing procedures

●      Manage template lifecycle and versioning

●      Create template documentation and usage guidelines

●      Develop template migration strategies

●      Monitor template performance and optimize as needed

4.    System Optimization

●      Monitor and optimize pipeline performance

●      Implement error handling and recovery mechanisms

●      Develop monitoring and alerting systems

●      Ensure cost-effective utilization of cloud resources

Requirements:

• Experience in building data models/ ETL pipeline/ data warehouse
• Experience with one in big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL/ NoSQL databases
• Experience with object-oriented/object function scripting languages: Python, Java, Scala, etc. is plus
• Having knowledge in the fintech market, GT distribution, online payment is plus
• Experience in Microsoft BI technologies and reporting solutions using Power BI/ PowerApps/ Microsoft Flow and other cloud-based solutions is plus

• Lunch allowance and parking free
• 13th month salary, project bonus, gifts (New Year, birthday, 8/3, 20/10..)
• Team building, company trip
• Insurance after 2 months of probation, PVI insurance for some levels
• Annual health check-up
• Training in work and fast-track your career path
• Working in a friendly and professional environment