References & Further Reading

On this page

References

Claude Code course from Anthropic: here.
Source for some info above: https://www.youtube.com/watch?v=-v7UQ1GPln8
Helpful: https://www.youtube.com/watch?v=uW-VUlDuk8w (good description of CoT variants)
PROCESS THIS: post (additional details about the kinds of agents).
React implemented in LangChain: https://www.promptingguide.ai/techniques/react (other design patterns explained too).
AI Agents with Tools + React, Chat, Web search, and other Custom Agents: article + notebook (downloaded).
Building AI Agents From Scratch - GitHub Repo
OpenAI AgentKit: https://openai.com/index/introducing-agentkit/?utm_source=substack&utm_medium=email Amazon Bedrock Guardrails - help avoid general harmful content by using a custom, user-defined fine-grained mechanism that provides additional customizable safeguards on top of the built-in protections of FMs.
19 courses on Agents, RAG, and real-world LLMs: link.
8 AI Agent building guides by AWS: link.
10 AI Agent Github repos: link.
10 Agent & GenAI GitHub repos: link
AI Evals free course: link

AGENTS.md prompt for persistent memory for long-running agentic tasks: here.

Video: https://www.youtube.com/watch?v=bZzyPscbtI8&t=1247s. Resources:
- GitHub: https://github.com/daveebbelaar/ai-cookbook/tree/main/patterns/workflows
- OpenAI API: https://platform.openai.com/docs/api-reference/introduction
- Anthropic. Building Effective Agents. https://www.anthropic.com/engineering/building-effective-agents

Agents on GCP:

Google's AI can now surf the web for you, click on buttons, and fill out forms with Gemini 2.5 Computer Use: https://venturebeat.com/ai/googles-ai-can-now-surf-the-web-for-you-click-on-buttons-and-fill-out-forms

Google AI Agents Course: copy from emails

From white paper: How We Evaluate AI Search for the Agentic Era by You.com (downloaded in Books)

Golden Set: of curated queries and expected answers (organization's consensus on quality - must define what "excellent results" mean) to evaluate / benchmark the performance of search engines, AI models & other systems. 
Evaluation Execution: Run the full golden set across different agents, models, systems and capture structured top-K results. For agentic or LLM-based search applications, run it across different search agents using the same synthesis LLM w/identical prompt so that only search quality varied. Submit the synthesized answers to an LLM judge using a consistent evaluation prompt or rubric.
Metrics & Reporting: clear, comparable metrics exposing both retrieval quality and downstream answer quality. Compute core scores such as accuracy, relevance, ranking quality, failure rates, and statistical confidence where appropriate. Aim - fast, unbiased interpretation of performance & repeatable, trustworthy decision-making.
If custom golden sets aren't feasible, use established benchmarks - SimpleQA, FRAMES, FreshQA, BrowseComp, FinSearchComp, etc.

To build a golden set that accurately predicts real-world performance, follow these six strategic steps:

Implement Strict Versioning: Freeze datasets into distinct versions (v1, v2) with full documentation; never edit existing sets quietly to ensure all evaluation results remain reproducible.
Define Success Metrics: Create a single, clear sentence defining "success" for your specific use case (e.g., specific numerical accuracy or multi-step reasoning depth).
Stratify and Scale Queries: Curate a pool of 100–500 queries that balance complexity, language, and edge cases to ensure statistical stability and fast iteration.
Source from Reality: Use anonymized production data, stakeholder interviews, or competitor analysis to ensure your test queries mirror your actual user distribution.
Expert Multi-Labeling: Have 2–3 domain experts independently judge results and reconcile differences, documenting the specific rationale behind their relevance decisions.
Develop Specific Rubrics: Create tailored grading criteria for different query types (e.g., requiring exact matches for facts vs. diversity for exploration) to remove ambiguity.

Evaluation:

Judge LLM with human validation - ask LLM to score across a range (0-4) or using bullet points (enumerating them).
Cohen’s Kappa (for binary or categorical relevance) - κ > 0.80: Excellent agreement, κ > 0.60: Substantial agreement κ < 0.40: Weak agreement.
Pearson Correlation (for graded or continuous scores) Pearson’s r measures how consistently the LLM’s scores track human scores across the full range. r > 0.85: Strong alignment r > 0.70: Moderate, usable with caution r < 0.60: Weak alignment.
Threshold-Based Agreement - easily interpretable measure for stakeholders: the percentage of cases where the model’s score falls within a small tolerance of the human score: |LLM_score – Human_score| ≤ 1 on a 0–4 scale. Target: ≥ 75% agreement within ±1 point This metric reflects practical usability—whether the model is “close enough” for downstream decisions.

Issues with Search Evaluation

The Scale Problem: Manually labeling search results is mathematically impractical; even a modest dataset would require billions of individual relevance judgments to be comprehensive.
Lack of Golden Sets: Most teams lack high-quality "ground truth" data, and defining what is "correct" is difficult because relevance is often subjective and changes over time.
The False Negative Problem: Since it is impossible to label every document, search engines are often unfairly penalized for returning relevant results that simply weren't included in the original label set.
Distribution Mismatch: Standard industry benchmarks are often outdated or generic, failing to accurately predict how a search engine will perform for specific niches like medical or developer queries.