LiftmyCVLiftmyCV
GL

Member of Technical Staff - ML Systems & Inference

Gimlet LabsRemoteSan Francisco, US
AI job summary

Gimlet Labs is pioneering a heterogeneous neocloud for AI workloads, overcoming infrastructure limitations in power, capacity, and cost. The ideal candidate will focus on designing and building scalable inference systems that efficiently manage ML workloads. Responsibilities include enhancing end-to-end inference pipelines, optimizing runtimes, and handling performance issues across models and systems. This role requires a strong engineering foundation with experience in ML inference or model serving systems, alongside a solid understanding of performance and memory management.

Job description

Gimlet Labs is building the first heterogeneous neocloud for AI workloads. As AI systems scale, the industry is hitting fundamental limits in power, capacity, and cost with today’s homogeneous, vertically integrated infrastructure. Gimlet addresses this by decoupling AI workloads from the underlying hardware. Our platform intelligently partitions workloads into components and orchestrates each component to hardware that best fits its performance and efficiency needs. This approach enables heterogeneous systems across multi-vendor and multi-generation hardware, including the latest emerging accelerators. These systems unlock step-function improvements in performance and cost efficiency at scale. On top of this foundation, Gimlet is building a production-grade neocloud for agentic workloads. Customers use Gimlet to deploy and manage their workloads through stable, production-ready APIs, without having to reason about hardware selection, placement, or low-level performance optimization. Gimlet works with foundation labs, hyperscalers, and AI native companies to power real production workloads built to scale to gigawatt-class AI datacenters. Gimlet Labs is seeking a Member of Technical Staff focused on ML systems and inference. In this role, you will design and build the inference systems that execute full models end-to-end under real production constraints. You will work at the intersection of model architecture, runtime behavior, and system performance to ensure inference is fast, predictable, and scalable. This role is ideal for engineers who deeply understand how modern models execute in practice and who care about latency, throughput, and memory behavior across the full inference lifecycle.

Responsibilities

  • Design and optimize end-to-end inference pipelines from request ingestion through execution and response
  • Build and evolve inference runtimes that balance latency, throughput, and concurrency under real-world load
  • Reason about batching, queuing, and scheduling tradeoffs, including their impact on tail latency and fairness
  • Manage KV cache allocation, placement, reuse, and eviction across models and requests
  • Optimize prefill and decode paths, including attention mechanisms and memory usage
  • Profile and debug inference performance issues across model, runtime, and system boundaries
  • Work closely with compilers, kernels, networking, and distributed systems to deliver end-to-end performance improvements

Qualifications

  • Strong software engineering fundamentals
  • Experience building or operating ML inference or model serving systems
  • Comfort reasoning about performance, memory usage, and system behavior under load

Preferred Qualifications

  • Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems
  • Deep understanding of modern model architectures and attention mechanisms
  • Experience with batching, scheduling, and concurrency control in inference systems
  • Familiarity with KV cache management and memory placement strategies
  • Experience profiling and tuning latency- and throughput-critical systems
  • Software development experience in Python and C++

Job details

Published
March 10, 2026
Last checked
November 17, 2026
Job source
ashby
Workplace
Remote · San Francisco, US
First seen
March 10, 2026
AI match explained

See how well you match this job

Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.

Sign up to see your matchAlready have an account? Log in

More at Gimlet Labs

Similar jobs from this company

None listed.

Across the market

Similar jobs from other companies

Third-party content disclaimer

Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.

LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.

LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.

If you prefer, you may apply directly via the original listing for transparency.