Job Summary:
We are seeking a highly motivated and experienced Data Engineer Lead to oversee the design, development, and management of our data infrastructure. This role involves leading a team of data engineers to build scalable data pipelines, ensure data quality, and optimize data processes to support business intelligence, analytics, and AI/ML initiatives.
The ideal candidate will possess a blend of strong technical skills, leadership abilities, and a keen eye for detail. You will work closely with stakeholders across departments to ensure that data systems meet business requirements.
Key Responsibilities:
- Data Architecture & Strategy:
- Design and implement scalable, reliable, and secure data architectures.
- Define best practices for data management, governance, and security.
- Establish and maintain a cloud-based data platform (e.g., AWS, Azure, GCP).
- Data Pipeline Development:
- Build and maintain ETL/ELT pipelines for ingesting structured and unstructured data from various sources.
- Ensure data pipelines are optimized for performance and reliability.
- Develop batch and real-time data processing solutions.
- Data Quality & Governance:
- Implement data quality frameworks to ensure accuracy, consistency, and reliability.
- Establish data governance policies, including data security, privacy, and compliance.
- Team Leadership & Mentorship:
- Lead a team of data engineers, providing technical guidance and mentorship.
- Manage project timelines, delegate tasks, and ensure delivery of high-quality data solutions.
- Foster a culture of continuous learning and improvement within the team.
- Collaboration with Stakeholders:
- Work closely with data analysts, data scientists, and business stakeholders to understand data requirements.
- Collaborate with DevOps, Software Engineering, and Product teams to integrate data solutions.
- Technology & Tooling:
- Stay updated with the latest trends and best practices in data engineering.
- Recommend and implement appropriate data tools and technologies to improve data processes.
- Ensure that the data infrastructure aligns with business goals and scales with growth.
Required Qualifications:
- Bachelor's degree in Computer Science, Engineering, or a related field. A Master's degree is a plus.
- 6+ years of experience in data engineering, with at least 2 years in a leadership role.
- Proficiency in programming languages such as Python, Java, or Scala.
- Expertise in SQL and experience with data modeling.
- Hands-on experience with cloud platforms (AWS, Azure, or GCP).
- Experience with data orchestration tools (Apache Airflow, Luigi, etc.).
- Knowledge of distributed data processing frameworks (Spark, Hadoop).
- Familiarity with data warehousing solutions (Snowflake, Redshift, BigQuery).
- Experience with real-time data streaming (Kafka, Kinesis).
- Understanding of data security, GDPR, and data governance best practices.
Preferred Qualifications:
- Experience with containerization (Docker, Kubernetes) and CI/CD pipelines.
- Knowledge of AI/ML model deployment pipelines.
- Experience with NoSQL databases and graph databases.
- Familiarity with tools like dbt, Power BI.
Key Competencies:
- Strong problem-solving and critical-thinking skills.
- Excellent communication and stakeholder management skills.
- Ability to balance technical depth with business acumen.
- Leadership and team management capabilities.
- Strong organizational and time-management skills.
