Principal Software Engineer | MLI

Position Summary

As a Principal Software Engineer | MLI at ExtraHop, you’ll work on the design and implementation of scalable systems that support our machine learning projects. You’ll work closely with data scientists, ML engineers, and cloud teams to create efficient tools and infrastructure for managing machine learning workflows – from data collection and experimentation to deployment and monitoring.

You’ll lead engineering best practices through code reviews, mentorship, and high standards for security, performance, and reliability. Your experience will help improve our machine learning processes, focusing on clear documentation, easy version management, automated deployments (CI/CD), and secure operations. Staying informed on new developments in machine learning operations (MLOps), container management, and cloud technologies, you'll help guide decisions to keep our ML platform modern and effective.

Key Responsibilities

• Provide technical leadership in the architecture, design, and implementation of robust and scalable infrastructure to support the full lifecycle of machine learning systems—from data ingestion to model deployment and monitoring.

• Collaborate with data scientists, ML engineers, and cloud teams to build high-performance services, pipelines, and tooling that accelerate ML experimentation and production workflows.

• Drive the evolution of ML infrastructure capabilities, including support for reproducibility, versioning, CI/CD for ML models, observability, and secure deployment.

• Design and develop infrastructure-as-code (IaC) solutions using Terraform or similar tools to ensure infrastructure scalability, consistency, and automation.

• Champion engineering excellence through code reviews, mentorship, and by setting high standards for performance, security, and maintainability.

• Stay informed about industry trends in ML infrastructure, MLOps, container orchestration, and cloud-native tools, and help shape strategic technology decisions.

• Contribute to internal documentation, operational runbooks, and design specifications to enable knowledge sharing and operational resilience across the team.

Required Qualifications

• 8+ years of professional software engineering experience, with 3+ years focused on ML infrastructure, distributed systems, or platform engineering.

• Strong programming skills in Python and Go, with an emphasis on writing production-quality, well-tested, and modular code.

• Demonstrated experience building and scaling systems that support data-intensive or ML workloads in cloud or hybrid environments.

• Solid experience with IaC tools such as Terraform, and configuration management systems (e.g., Ansible, Packer).

• Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes), especially in support of ML workloads.

• Working knowledge of ML model lifecycle, MLOps best practices, and ML frameworks (e.g., PyTorch, TensorFlow, MLflow).

• Experience with CI/CD pipelines, automation frameworks, and observability tools (e.g., Prometheus, Grafana, Datadog).

• Excellent problem-solving skills with the ability to lead technically complex projects from concept to production.

• U.S. citizenship or lawful permanent resident status required for work in secure environments.

Preferred Qualifications

• Master’s or Bachelor's degree in Computer Science, Engineering, or a related field.

• Prior experience securing ML systems or data pipelines, with an understanding of role-based access control (RBAC), service accounts, and network segmentation.

• Familiarity with compliance and regulatory standards such as FedRAMP, NIST SP 800-53, or similar 

• Exposure to security in ML and cloud environments, including threat modeling, guardrails, and vulnerability scanning tools.

• AWS, GCP, or Azure certification (e.g., Solutions Architect or ML Specialist) is a plus.

• Experience supporting or building Network Detection and Response (NDR), intrusion detection systems, or other cybersecurity-focused ML applications is highly desirable.

The salary range for this role is 160,000 - 190,000 + bonus + benefits