
Senior Data Engineer - Financial Flows
Kpler is seeking a Senior Data Engineer for its Financial Flows team to manage end-to-end data pipelines and cloud infrastructure supporting commodity finance analytics. Key responsibilities include migrating services to AWS, optimizing legacy calculation engines in Python, and ensuring the reliability of data feeds. Applicants should have over 5 years of experience working with production-grade data systems, advanced Python skills, and experience with real-time data pipelines. The role emphasizes collaboration with quantitative researchers and offers opportunities for mentorship and team development.
Job description
At Kpler, we are dedicated to helping our clients navigate complex markets with ease. By simplifying global trade information and providing valuable insights, we empower organisations to make informed decisions in commodities, energy, and maritime sectors. Since our founding in 2014, we have focused on delivering top-tier intelligence through user-friendly platforms. Our team of over 850 experts from 69 countries works tirelessly to transform intricate data into actionable strategies, ensuring our clients stay ahead in a dynamic market landscape. Join us to leverage cutting-edge innovation for impactful results and experience unparalleled support on your journey to success.
About the Role
As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics. You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services. Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets.
About the Role
As a Senior Data Engineer on the Financial Flows team, you will take full technical ownership of the end-to-end data pipelines, predictive models, and cloud infrastructure powering Kpler’s commodity finance analytics. You will lead the modernization of our data foundation, spearheading the migration of production services to AWS and re-architecting legacy calculation engines into clean, performant Python services. Operating at the crossroads of quantitative research and cloud infrastructure, your work will directly ensure that market participants receive real-time, mission-critical insights across global commodities markets.
Key Responsibilities
- Design, build, and operate resilient batch and real-time streaming data ingestion pipelines connecting to external financial market data providers via APIs and WebSockets.
- Lead the architectural migration of legacy backend infrastructure to modern cloud-native environments on AWS, establishing high standards for system reliability and scalability.
- Drive the refactoring and rewrite of the CTA positioning indicator calculation blocks from PHP to Python, ensuring strict mathematical output parity, comprehensive test coverage, and optimized latency.
- Ingest and model complex financial datasets—including futures pricing, open interest, and trading volumes—across time-series stores and search engines like QuestDB and Elasticsearch.
- Implement automated data validation, proactive alerting, and system monitoring to ensure market data feeds remain accurate, continuous, and fault-tolerant.
- Deploy, maintain, and scale containerized services across Docker and Kubernetes environments using automated CI/CD pipelines and GitOps workflows.
- Partner closely with quantitative researchers, commodity analysts, and product managers to translate complex trading models into high-performance production pipelines.
- Provide technical mentorship to peer engineers, raising team-wide engineering rigor through thoughtful code reviews, system design guidance, and architectural documentation.
Experience & Background
What you'll need (Must-haves)
- 5+ years of professional experience building, operating, and scaling production-grade data pipelines and backend infrastructure.
- Advanced expertise in Python, with a strong background refactoring and rewriting production-critical code for performance and testability.
- Proven hands-on experience building and maintaining real-time or streaming API ingestion pipelines (REST, WebSockets) handling time-series datasets.
- Solid production experience deploying and managing containerized applications on AWS using Docker, Kubernetes, and modern CI/CD practices.
- Strong database fundamentals in SQL and experience working with distributed search or time-series databases (e.g., Elasticsearch, QuestDB, ClickHouse).
- Deep experience establishing automated data quality frameworks, observability (monitoring/alerting), and root-cause troubleshooting for production systems.
- Domain knowledge of financial market structures, futures contracts, open interest, and commodities trading symbology.
- Experience integrating data from major financial market vendors (e.g., Bloomberg, LSEG, Enverus, Trading Technologies).
- Familiarity with legacy PHP codebase migration or reading PHP algorithms to translate into Python.
- Practical familiarity with GitOps frameworks and infrastructure-as-code tooling.
We are a dynamic company dedicated to nurturing connections and innovating solutions to tackle market challenges head-on. If you thrive on customer satisfaction and turning ideas into reality, then you’ve found your ideal destination. Are you ready to embark on this exciting journey with us?
We make things happen
We act decisively and with purpose, going the extra mile.
We build together
We foster relationships and develop creative solutions to address market challenges.
We are here to help
We are accessible and supportive to colleagues and clients with a friendly approach. Our People Pledge Don’t meet every single requirement? Research shows that women and people of color are less likely than others to apply if they feel like they don’t match 100% of the job requirements. Don’t let the confidence gap stand in your way, we’d love to hear from you! We understand that experience comes in many different forms and are dedicated to adding new perspectives to the team. Kpler is committed to providing a fair, inclusive and diverse work-environment. We believe that different perspectives lead to better ideas, and better ideas allow us to better understand the needs and interests of our diverse, global community. We welcome people of different backgrounds, experiences, abilities and perspectives and are an equal opportunity employer. By applying, I confirm that I have read and accept the Staff Privacy Notice
Job details
- Published
- October 29, 2026
- Last checked
- November 18, 2026
- Experience
- Senior
- Job source
- lever
- Workplace
- Hybrid · London, GB
- First seen
- October 29, 2026
Requirements
5+ years of professional experience; Advanced expertise in Python; Proven experience with real-time or streaming API ingestion pipelines; Production experience with Docker and Kubernetes; Strong database fundamentals in SQL; Experience with financial market structures.
Skills mentioned in this job
LiftmyCV exclusive data
Kpler hiring intelligence
Kpler currently has 52 active jobs tracked by LiftmyCV. 3 new jobs in the past 7 days.
Verified November 18, 2026
Active openings
52
Currently tracked by LiftmyCV
New in 7 days
3
41 new jobs in the past 30 days
Published salaries
2%
Median disclosed salary: $150,000
Average opening age
22 days
Across 52 active openings
Workplace breakdown
Hiring countries
Top hiring cities
Most in-demand roles
- • Senior Account Executive (Government, Defense & Intelligence)
- • Senior Data Engineer
- • Fullstack Engineer
- • Account Development Representative
- • Account Executive - Automotive & Mobility Intelligence
- • Account Executive - Commodities - Middle East
Frequently requested skills
Median disclosed salary: $150,000 · 2% of jobs publish a salary
As of November 18, 2026
Hiring insights calculated from active jobs tracked by LiftmyCV.
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inVisa sponsorship insights
Sponsorship not specified for this role
This job posting does not state whether visa sponsorship is available.
Employer UK sponsor status
Kpler Limited
Licensed sponsor
Global Business Mobility: Senior or Specialist Worker, Skilled Worker
A-rated
sponsor rating
A sponsor licence does not guarantee that this role offers visa sponsorship.
More at Kpler
Similar jobs from this company
Across the market
Similar jobs from other companies
Related job searches
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.