All Openings

LLMOps/Evals Engineer

LLMOps/Evals Engineer

Bay Area, CA

|

Full-Time

|

Salary + Equity

About CodeRabbit

Current practices of ensuring quality in software workflows need an overhaul. They're often slow, inefficient, and costly, creating a major bottleneck. As developers, we have firsthand experience with these challenges and complexities. Our mission is to build an AI-first central code quality platform from the ground-up. That vision starts with augmenting manual code reviews using AI and eventually all other aspects of the quality workflows.

CodeRabbit is currently the most installed AI application on both GitHub and GitLab. It is used daily by thousands of developers and teams of all sizes, with over a thousand paying customers. We are backed by top-tier VCs and founders who have deep experience in developer workflows. If you are someone who cares about developer tooling and is interested in joining our team, we would love to hear from you.

Job description

We are looking for an experienced LLMOps/Evals Engineer for evaluating the application performance.

Challenges

  • Auto-evaluation of the output using LLMs.

  • Constructing the right datasets for evaluating the application output. 

  • Continuous tracking of the live application and spotting issues in real time.

What we’re looking for

Requirements
  • Experience constructing datasets from production data or existing sources to be used evals and few shot prompting. 

  • Experience in evaluation using LLMs to score application output. 

  • Setup regression testing to measure performance of the evaluation criteria by changes to the application.

  • Continuous tracking of the live application and spotting issues in real time.  

  • Experience with RAG embeddings and rerankers to improve the output quality.

  • Problem-solvers who can apply innovative ideas to solve the unknown challenges.

  • Ownership: You want to build something you are proud of. Monitor what you ship, learn, iterate, and make it better.

  • Velocity: Deliver high-quality software quickly through decisive action and effective execution.

Good-to-haves
  • 3+ years as an AI/ML Engineer

CodeRabbit is an innovative, AI-driven platform that transforms the way code reviews are done. Its automated reviews elevate the code quality while significantly reducing the time and effort tied to extensive manual code reviews.


The platform offers insightful, line-by-line feedback on code changes, suggesting improvements and corrections that can enhance the efficiency and robustness of the code.

Sign up for our newsletter!

CodeRabbit © 2024.

CodeRabbit is an innovative, AI-driven platform that transforms the way code reviews are done. Its automated reviews elevate the code quality while significantly reducing the time and effort tied to extensive manual code reviews.


The platform offers insightful, line-by-line feedback on code changes, suggesting improvements and corrections that can enhance the efficiency and robustness of the code.

Sign up for our newsletter!

CodeRabbit © 2024.

CodeRabbit is an innovative, AI-driven platform that transforms the way code reviews are done. Its automated reviews elevate the code quality while significantly reducing the time and effort tied to extensive manual code reviews.


The platform offers insightful, line-by-line feedback on code changes, suggesting improvements and corrections that can enhance the efficiency and robustness of the code.

Sign up for our newsletter!

CodeRabbit © 2024.