LiftmyCVLiftmyCV
IC

Senior Databricks Engineer

International Capability Centre Private LimitedOn-site2nd Floor, MFAR Manyata Tech Park, Greenheart Phase I, Kasaba Hobli, Nagavara Village, Bengaluru 560045

Job description

Key Responsibilities:

Databricks Platform Engineering • Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform. • Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows. • Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking. • Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance. • Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting. Data Pipeline & Integration • ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse. • Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services. • Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases. Software Engineering & DevOps • Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards. • CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs. • Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep. • Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments. Data Governance & Quality • Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations. • Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration. • Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions. Collaboration & Leadership • Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices. • Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders. • Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies. Required Qualifications • Databricks Certified Data Engineer Professional or Associate (mandatory). • Microsoft Certified: Azure Data Engineer Associate (mandatory). • 10+ years of hands-on experience in Data Engineering roles. • Deep expertise in Databricks — Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters. • Advanced PySpark and Python skills for large-scale distributed data processing. • Strong SQL skills: query authoring, performance tuning, and data modeling. • Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics. • Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep). • Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture). • Experience with Structured Streaming or other real-time data processing frameworks. Preferred Qualifications • Experience with Microsoft Purview for data cataloguing and governance. • Familiarity with dbt (data build tool) for SQL-based transformation workflows. • Knowledge of MLflow and Databricks Feature Store for ML engineering integration. • Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment. • Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures. • Background in data mesh or data product thinking. Key Skills at a Glance Core Platform Databricks, Delta Lake, Unity Catalog, DLT Languages PySpark, Python, SQL Cloud & Integration Azure Data Factory, ADLS Gen2, Synapse, Event Hubs DevOps & IaC Azure DevOps, Terraform, Bicep, DABs Data Quality & Governance Great Expectations, DLT Expectations, Purview Architecture Lakehouse, Medallion Architecture, Data Mesh Why Join Us? • Lead the build-out of a modern, Databricks-first data platform at scale. • Work with a collaborative, forward-thinking Data & Analytics team. • Access to cutting-edge tooling across Azure, Databricks, and the broader modern data stack. • Competitive salary, benefits, and a strong commitment to your professional development and certifications. • Opportunity to shape team standards, architecture decisions, and engineering culture. • Contribute to high-impact, global data initiatives. Additional Information

About Us, Our Culture & What We Can Offer You

Established over 200 years ago, Chubb Fire & Security are a global business driven by keeping people and the world safe. We provide essential systems, equipment and services, from digital CCTV surveillance and intruder alarms, to fire detection and suppression systems. Offering a full range of innovative products and services to a broad range of customers, from local independent business, to many of the FTSE 100 companies We believe we offer a unique working culture, where you are as important to us as our customers, and we want you to feel that everyday. We are proud to offer extensive training to all of our new Advisors, fully supporting, and enabling you to thrive in your new role and beyond, with ongoing career development opportunities throughout your career journey with us. #BR-CB

Job details

Published
August 24, 2026
Job source
myworkdayjobs
Workplace
On-site · 2nd Floor, MFAR Manyata Tech Park, Greenheart Phase I, Kasaba Hobli, Nagavara Village, Bengaluru 560045
First seen
August 24, 2026
AI match explained

See how well you match this job

Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.

Sign up to see your matchAlready have an account? Log in

More at International Capability Centre Private Limited

Similar jobs from this company

None listed.

Across the market

Similar jobs from other companies

Third-party content disclaimer

Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.

LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.

LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.

If you prefer, you may apply directly via the original listing for transparency.