
Senior AI System Engineer - LLM Training System
Job description
Huawei Canada has an immediate permanent opening for a Senior Engineer.
About the team:
The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.
About the job:
As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).
- Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.
- Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).
- Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.
- Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.
- Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.
- Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.
About the ideal candidate:
- Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.
- Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.
- Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.
- Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.g., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).Preferred / Standout Qualifications:
- Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).
- Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.
- Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).
Additional Information:
Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs. All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.
Job details
- Published
- November 3, 2026
- Last checked
- November 17, 2026
- Experience
- Senior
- Job source
- recruitee
- Workplace
- On-site · Vancouver, CA
- First seen
- November 3, 2026
LiftmyCV exclusive data
Huawei Technologies Canada Co., Ltd. hiring intelligence
Huawei Technologies Canada Co., Ltd. currently has 23 active jobs tracked by LiftmyCV. 4 new jobs in the past 7 days.
Verified November 17, 2026
Active openings
23
Currently tracked by LiftmyCV
New in 7 days
4
15 new jobs in the past 30 days
Published salaries
9%
Median disclosed salary: $76,000
Average opening age
47 days
Across 23 active openings
Workplace breakdown
Hiring countries
Top hiring cities
Most in-demand roles
- • AI Research Engineer - Computer Vision & Neural Rendering
- • Associate Office Manager
- • Ecosystem Operations Specialist
- • Engineer - Agentic AI
- • Engineer - Computer Graphics
- • Hardware & Embedded Systems Engineer - AI-Edge Devices
Frequently requested skills
Median disclosed salary: $76,000 · 9% of jobs publish a salary
As of November 18, 2026
Hiring insights calculated from active jobs tracked by LiftmyCV.
See how well you match this job
Log in or sign up and Eva will compare your profile with this role, explain your strengths and show what to improve.
Sign up to see your matchAlready have an account? Log inMore at Huawei Technologies Canada Co., Ltd.
Similar jobs from this company
- Engineer - Agentic AIHuawei Technologies Canada Co., Ltd.Ottawa, CAOn-site
- Intern - ResearcherHuawei Technologies Canada Co., Ltd.Montreal, CAOn-site$58,000–$104,000 per year
- Engineer - Computer GraphicsHuawei Technologies Canada Co., Ltd.Vancouver, CAOn-site
- Associate Office ManagerHuawei Technologies Canada Co., Ltd.Markham, CAOn-site
- Intern Researcher- Computer VisionHuawei Technologies Canada Co., Ltd.Markham, CAOn-site
- KHRONOS Standard Expert LeadHuawei Technologies Canada Co., Ltd.Vancouver, CAOn-site
Across the market
Similar jobs from other companies
- Training Systems ManagerpaeOC-AUS-BrisbaneOn-site
- Training Systems AnalystTransCanada PipeLines LimitedCalgary, CAOn-site
- Training Systems ManagerMoog IncBlacksburg, USOn-site$115,000–$170,000 per year
- System Tester in TrainingUEC Ameren MissouriMaryland Heights, USOn-site
- Training Systems TechnicianpaeHouston, USOn-site$85,000 per year
- Systems Training SpecialistWillamette DentalHillsboro, USOn-site$69,000–$82,000 per year
Related job searches
Third-party content disclaimer
Full job descriptions are sourced directly from third-party applicant tracking systems (ATS), job boards, recruitment platforms, and employer career pages.
LiftmyCV does not claim ownership of this content or guarantee its accuracy or current availability. All original content is the intellectual property of the respective employers or other rights holders.
LiftmyCV hiring intelligence, availability checks, and related-job recommendations are generated from LiftmyCV data and public sources.
If you prefer, you may apply directly via the original listing for transparency.