AgentOps Engineer – AI Agent Observability

✅ What we need from you:

Essential to have at least 2 years of experience in Cloud environments and practical experience in:

• Azure Observability: Azure Monitor, Log Analytics, and Application Insights, with KQL queries.
• Grafana: creation and maintenance of dashboards, panels, variables, and alerts on sources such as Azure Monitor, Log Analytics, or Prometheus/PromQL.
• Agent Observability: monitoring AI agents through traces, metrics, and logs (OpenTelemetry or similar): latency, tokens, cost, errors, response quality, and tool usage.
• AI Agents: understanding how they work (LLMs, prompting, RAG, tool calling, orchestration) and having worked with them in a real environment, not just at a theoretical level.
• Production service operations: incident management, RCA, and working with SLAs and ticketing tools.
• Fluent English, oral and written (the service is provided to an international client).

Additionally, although not essential, it is important that you have notions of:

• Copilot Studio: how an agent is built and configured (topics, actions, connectors, and knowledge sources) and where to consult its analytics.
• Azure AI Foundry / Azure OpenAI: how models and agents are deployed and which services are involved.
• Python: enough to read code, understand an agent, or automate simple tasks.

🔍 It's a plus if you also have:

• You have developed pro-code agents with frameworks such as Microsoft Agent Framework, Semantic Kernel, AutoGen, or LangChain.
• You know LLM evaluation (quality metrics, groundedness, golden datasets, non-regression testing).
• You have used Azure DevOps, Git, and CI/CD pipelines to deploy and version agents.
• You have experience with AI security: Azure AI Content Safety, guardrails, prompt injection, or PII management.
• You have worked with infrastructure as code (Terraform or Bicep) or services such as Azure Functions, APIM, or Key Vault.
• You have technical training (Computer Science, Telecommunications, Mathematics, etc.).
• You have official certifications such as AI-102, AZ-104, PL-200, AI-900, or similar.

🧠 What do we value?

• Drive to contribute, work as a team, and grow
• Consultative attitude and client focus
• Analytical and problem-solving mindset regarding production incidents
• Ability to move between operations and development, understanding both the platform and the agent
• Responsibility for your work and commitment to the team
• Curiosity about generative AI and the ability to stay up to date

🚀 Want to join the challenge?

At BIP - xTech we are looking for people with a desire to grow, learn, and take on real technology projects: cloud, data, AI, security, modern workplace, automation, DevSecOps, IoT, Industry 4.0... You won't get bored here!

We are a diverse and enthusiastic team, where we combine technology with strategy to deliver top solutions to our clients. If you are interested in being at the center of digital transformation, this is the place for you.

💼 What will you do?

We are looking for people who move fluently between the world of cloud observability and that of AI agents to join our AgentOps managed service for a leading international consumer goods client. You will be part of the team that monitors, operates, and evolves a growing fleet of enterprise AI agents (Azure AI Foundry and Copilot Studio) in production, ensuring they function well, securely, within agreed service levels, and improve over time.

Your day-to-day will include:

• Agent observability: design, deploy, and evolve dashboards in Grafana and Azure Monitor to track agent health, latency, availability, errors, token consumption, and cost.
• Proactive monitoring and alerting: define and monitor KPIs, SLAs, and alert thresholds, and early detect degradations, anomalies, and quality loss (drift).
• Incident management: triage, log and trace analysis, root cause analysis (RCA), and resolution or escalation within SLAs, stepping into the configuration of the agent itself when necessary.
• Agent improvement and evolution: adjust prompts, knowledge sources, connectors, and actions, validate changes with non-regression testing, and manage model updates.
• Lifecycle and deployments: support the versioning, deployment, and rollback of agents, and ensure that new agents reach production with the necessary observability and runbooks.
• Reporting and documentation: periodic service health reports, runbooks, and incident playbooks.