Senior Data Quality Engineer
We are looking for a Senior Data Quality Engineer to lead the creation and automation of data quality frameworks within a Databricks environment. This role is crucial for establishing data quality controls and automated monitoring across a large-scale data platform. The ideal candidate will have strong experience in Databricks, PySpark, and data quality management practices.
Job description
About the Role
We are seeking an experienced Senior Databricks Data Quality Engineer to lead the design, implementation, and automation of enterprise-scale data quality frameworks within a Databricks environment. The successful candidate will play a key role in establishing data quality controls, profiling frameworks, remediation processes, and AI-assisted quality monitoring across a large-scale data platform consisting of 170+ datasets and over 1,300 Critical Data Elements (CDEs). This role requires strong expertise in Databricks, PySpark, Delta Lake, MLflow, and modern data quality management practices.
Key Responsibilities
Data Platform & Databricks Configuration
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations.
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDEs).
Data Profiling & Quality Assessment
- Develop AI-assisted profiling notebooks using PySpark to establish baseline data quality scores.
- Assess data quality across six key dimensions including:
- Completeness
- Uniqueness
- Validity
- Consistency
- Accuracy
- Timeliness
- Analyze null rates, duplicate records, invalid values, format violations, outliers, and schema drift.
Data Quality Rule Framework
- Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions.
- Enable automated deployment of over 6,700 data quality rules without manual rule-by-rule development.
- Create reusable rule templates across datasets and data quality dimensions.
Pipeline Quality Enforcement
- Integrate data quality controls within Bronze, Silver, and Gold Delta Lake layers.
- Implement quality gates that prevent data progression unless predefined thresholds are met.
- Develop reusable Databricks Jobs for automated validation and monitoring.
Data Cleansing & AI-Driven Remediation
- Build automated data cleansing pipelines for:
- Standardization
- Deduplication
- Schema harmonization
- Deploy MLflow-managed machine learning models for:
- Anomaly detection
- Fuzzy duplicate detection
- Exact duplicate identification
- Ensure explainability of model outputs and support human-in-the-loop validation processes.
Exception Management
- Design failed-record handling frameworks and quarantine Delta tables.
- Capture failure reasons, affected CDEs, rule references, and timestamps.
- Develop automated reprocessing mechanisms for corrected records.
Data Quality Monitoring & Reporting
- Build Delta Lake aggregation tables for data quality metrics.
- Deliver data quality KPIs to Power BI dashboards including:
- Dimension-level scores
- Rule pass/fail rates
- SLA adherence metrics
- Configure automated alerting using Databricks SQL Alerts and Azure Monitor.
Predictive Data Quality Analytics
- Develop predictive models to identify datasets at risk of quality degradation.
- Support AI-assisted Root Cause Analysis (RCA) using profiling outputs and machine learning techniques.
- Export and prepare remediation datasets for prioritization and governance reporting.
Job details
- Published
- June 14, 2026
- Job source
- jworkable
- Workplace
- On-site · Abu Dhabi, AE
- First seen
- June 14, 2026
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at robusta
Similar jobs from this company
None listed.
Across the market
Similar jobs from other companies
- Senior Data Quality EngineerrobustaAbu Dhabi, AEOn-site
- Senior Data Quality EngineerSallaMadinah, Al Madinah Province, Saudi ArabiaHybrid
- Senior Data Quality EngineerAmerican Century Services, LLCKansas City, MissouriOn-site
- Senior Data Quality EngineerLLOYDS BANK PLCBristolOn-site
- Senior Data Quality EngineerOntrac SolutionsCairo, EGRemote
- Senior Data Quality EngineerLivestock Improvement Corporation (LIC)Hamilton, NZHybrid
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.