Data Engineer

Job Purpose

The Data Engineer will be part of the team to develop operation & maintenance decision-support tools to enhance train reliability and maintenance efficiency. This position involves designing, developing, and maintaining data pipelines, APIs, and cloud infrastructure for various rail-oriented applications. The ideal candidate will have expertise in data analysis, transformation, ingestion, database design, API development, and preferably, cloud infrastructure setup. Collaborating closely with software engineers, data scientists, and frontend developers, the Data Engineer will contribute to building efficient, scalable, and reliable systems.

Responsibilities

The duties and responsibilities for Data Engineer, are as listed below. The list is not comprehensive and related duties and responsibilities may be assigned from time to time.

Data Engineering & Processing:

• Develop and maintain data pipelines for efficient data ingestion and transformation.
• Work with structured and unstructured data to ensure optimal storage and retrieval.
• Perform data analysis and report on results.

Database Design & Management:

• Design and implement relational and NoSQL database schemas for scalability.
• Optimize database performance through indexing, partitioning, and query tuning.
• Implement data security and compliance best practices.

API Development & Backend Engineering:

• Design and develop APIs for data access and application integration.
• Implement authentication, authorization, and API security best practices.

Cloud Infrastructure & Deployment (Supporting Role):

• Assist in design Azure cloud architectures
• Work with IT infrastructure team to set up cloud infrastructure for application hosting, data storage and processing. 

Collaboration & Best Practices:

• Collaborate with internal stakeholders to understand their business needs.
• Work with software engineers, data scientist, frontend developer to understand the data requirement and design architecture of the data platform.
• Implement CI/CD pipelines for automated testing, deployment and monitoring.
• Write testable and maintainable code and documentation to deploy to production.
• Engage continuously with end-user for feedback and improvements.

Qualifications & Work Experience

• Degree in Science, Technology, Engineering or Mathematics (STEM)
• Previous experience as a data engineer or in a similar role
• Data engineering certification is a plus
• Knowledge of security best practices in cloud and database management is a plus

Skills

Technical skills include:

• Programming and Data processing: MATLAB, Python, SQL, or similar languages.
• Databases: My SQL, SQL Server, MongoDB or similar.
• Cloud Platforms: Azure
• DevOps & CI/CD: Git Lab CI/CD, Docker

Generic skills include:

• Strong inclination and eager for continual learning and development
• Strong team player
• Critical thinking and problem-solving skills
• Ability to understand and explain complex data and effective interactions with the stakeholders
• Ability to think independently and actively propose solutions to the team.