Senior Data Engineer - Hybrid

Sobre o nosso cliente

Our client is committed to leveraging data and analytics to improve services and drive innovation in the financial services industry. This is a 100+ people department focused on digital, AI and innovation.

• Act as the primary technical point of contact for Product Owner and business stakeholders.
• Take ownership of data engineering responsibilities within projects.
• Lead technical architecture, design sessions, and solution breakdowns.
• Mentor and guide junior Data Engineers.
• Design and implement batch and real-time data ingestion pipelines using Databricks (PySpark) and DBT.
• Clean, transform, and optimize data for analytics and machine learning.
• Define and optimize database schemas, storage formats (Parquet, Delta Lake), and indexing strategies.
• Collaborate with data analysts, data scientists, and platform teams to ensure best practices and integration.
• Implement security, compliance, and role-based access controls in Azure environments.
• Configure and maintain Azure cloud infrastructure (Data Lake, Blob Storage, Data Factory, Databricks).
• Create documentation, support onboarding, and promote best practices.
• Monitor, troubleshoot, and continuously improve workflows and systems.
• Stay up to date with advancements in data engineering technologies and propose optimizations.
• Demonstrate initiative by proactively taking ownership of projects and tasks; deliver consistent, high-quality work; foster empathy and accountability within the team.
• Explain complex technical concepts clearly to non-technical stakeholders; actively listen and consider feedback; produce clear documentation; build strong relationships across teams; embrace constructive criticism for improvement.
• Apply analytical thinking to break down complex problems; use creativity to develop innovative solutions; show resilience when facing challenges and setbacks.

• Strong Python skills, including data manipulation libraries (e.g., Pandas) and database interfacing.
• Knowledge of Python ecosystem tools (e.g., poetry, venv) and best software engineering practices (design patterns, testing, clean code).
• Advanced SQL expertise for querying, transformation, and optimization.
• Ability to design scalable, robust data pipelines and efficient database schemas.
• Experience with dimensional modeling, 3NF, Data Vault, and ETL design.
• Proficiency with DBT for data modeling and transformation.
• Hands-on experience with Azure services (Data Lake, Data Factory, Blob Storage, SQL Database).
• Proficiency in Azure Databricks (Spark jobs, clusters, secret scopes, Unity Catalog).
• Strong PySpark skills and knowledge of Delta Lake for reliable data lakes.
• Familiarity with relational (SQL Server) and NoSQL databases (Cosmos DB).
• Understanding of data formats (Parquet, JSON, CSV, Delta).
• Experience designing and managing data integration solutions with Azure Data Factory.
• Expertise in query tuning and pipeline optimization.
• Knowledge of encryption techniques and role-based access controls integrated with Azure AD.
• Experience with CI/CD pipelines (Azure DevOps).
• Familiarity with Infrastructure as Code (ARM templates, Terraform) and containerization (Docker, Kubernetes).
• Knowledge of Azure Monitor, Log Analytics, and debugging data pipelines.

• Opportunity to work in a data-driven organization.
• Engaging and supportive company culture.