AI Engineer

Job Description

Join Ruby Labs as a Senior AI Engineer, where you'll be instrumental in shaping the future of AI infrastructure and crafting production-ready LLM experiences. Collaborate on pioneering advanced prompt engineering, creating dynamic prompt templates that utilize conditional logic for optimal AI output and reasoning. Implement structured output schemas like JSON mode and function calling to guarantee predictable AI outputs seamlessly integrated into application logic. Develop robust evaluation pipelines using Langfuse to gather feedback and score response quality in real-time. Perform in-depth debugging of complex LLM chains, identifying bottlenecks and optimizing for cost, latency, and context window efficiency. Conduct systematic A/B tests across various models via OpenRouter, basing deployment decisions strictly on quantitative benchmarks and trace data, rather than intuition. Develop scoring systems for analyzing the Problem → Solution chain, pinpointing root causes of hallucinations or logic errors with Langfuse analytics. Regularly reassess model performance as new architectures emerge, fine-tuning as needed to meet specific domain requirements.

Qualifications

1. Deep understanding of Node.js & Next.js to build reliable services and manage complex LLM-generated data. 2. Proven experience in dynamic prompting, where content is highly dependent on input variables and context injection. 3. Experience with OpenRouter unified APIs, managing rate limits, and selecting cost-effective models for specific tasks. 4. Understanding of LLM observability principles with Langfuse (or similar): setting up tracing, creating test datasets, and integrating scoring systems. 5. Experience with evaluation frameworks like RAGAS or building custom LLM-as-a-judge systems. 6. Analytical mindset: ability to transform raw generation logs into actionable business metrics and technical insights. 7. Iterative mindset: focus on continuous product improvement through constant feedback loops.

Benefits

- Remote Work Environment: Enjoy the flexibility to work from anywhere, promoting a healthy work-life balance. - Unlimited PTO: Take advantage of unlimited paid time off to recharge and prioritize your well-being. - Paid National Holidays: Unwind and celebrate on national holidays with paid time off. - Company-provided MacBook: Enhance your productivity with a top-notch Apple MacBook, if needed. - Flexible Independent Contractor Agreement: Experience flexibility, autonomy, and entrepreneurial opportunities, including tax advantages and reduced employment obligations.