All Openings
Bay Area, CA
|
Full-Time
|
Salary + Equity
About CodeRabbit
Current practices of ensuring quality in software workflows need an overhaul. They're often slow, inefficient, and costly, creating a major bottleneck. As developers, we have firsthand experience with these challenges and complexities. Our mission is to build an AI-first central code quality platform from the ground-up. That vision starts with augmenting manual code reviews using AI and eventually all other aspects of the quality workflows.
CodeRabbit is currently the most installed AI application on both GitHub and GitLab. It is used daily by thousands of developers and teams of all sizes, with over a thousand paying customers. We are backed by top-tier VCs and founders who have deep experience in developer workflows. If you are someone who cares about developer tooling and is interested in joining our team, we would love to hear from you.
Job description
We are looking for an experienced LLMOps/Evals Engineer for evaluating the application performance.
Challenges
Auto-evaluation of the output using LLMs.
Constructing the right datasets for evaluating the application output.
Continuous tracking of the live application and spotting issues in real time.
What we’re looking for
Requirements
Experience constructing datasets from production data or existing sources to be used evals and few shot prompting.
Experience in evaluation using LLMs to score application output.
Setup regression testing to measure performance of the evaluation criteria by changes to the application.
Continuous tracking of the live application and spotting issues in real time.
Experience with RAG embeddings and rerankers to improve the output quality.
Problem-solvers who can apply innovative ideas to solve the unknown challenges.
Ownership: You want to build something you are proud of. Monitor what you ship, learn, iterate, and make it better.
Velocity: Deliver high-quality software quickly through decisive action and effective execution.
Good-to-haves
3+ years as an AI/ML Engineer