티스토리 수익 글 보기
Repository navigation
examples
Directory actions
More options
Directory actions
More options
examples
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
parent directory.. | ||||
This directory contains notebooks, SQL scripts, Python demos, and reference artifacts that demonstrate SDK capabilities.
Note: the judge model is independent of the agent model by default — demos
configure the agent via MODEL_NAME and all SDK-side LLM judging via a
separate JUDGE_MODEL. When no explicit model/endpoint is passed, Client
and LLMAsJudge silently fall back to the SDK default (gemini-2.5-flash).
| Notebook | Description |
|---|---|
| bqca_prompt_response_logging_customer_notebook.ipynb | End-to-end starter playbook for evaluating BigQuery Conversational Analytics (BQCA) Prompt & Response logs (gemini-3.5-flash default judge endpoint; see the notebook’s Region guide for US/EU multi-region and single-region routing): compatibility view setup, doctor() health check, question categorization, semantic drift detection, golden Q&A grading, customer sentiment, latency/token FinOps, and error/refusal diagnostics. Companion guide: BQCA Customer Manual (Open in Colab). |
| dashboard_v2.ipynb | Observability dashboard (2-layer SQL, no SDK dependency) |
| dashboard_v2_bigframes.ipynb | BigFrames companion for Dashboard V2 |
| e2e_notebook_demo.ipynb | End-to-end SDK workflow |
| ai_ml_integration_demo.ipynb | AI.GENERATE, AI.EMBED, anomaly detection |
| categorical_evaluation_demo.ipynb | Hatteras categorical evaluation |
| context_graph_adcp_demo.ipynb | Agent Context Graph decision-trace use cases |
| ontology_graph_v5_demo.ipynb | OWL import, mixed extraction, temporal lineage, GQL |
| _archive/context_graph_historical_notebook.ipynb | Archived: the original MAKO context-graph pipeline (explicit ontology + binding), kept as frozen evidence |
| ontology_graph_v4_demo.ipynb | Ontology extraction + GQL (legacy) |
| memory_service_demo.ipynb | Cross-session memory |
| event_semantics_views_bigframes_demo.ipynb | Event views + BigFrames |
| nba_agent_trace_analysis_notebook.ipynb | Real-world trace analysis |
| File | Description |
|---|---|
| ai_classify_side_by_side.sql | AI.CLASSIFY vs AI.GENERATE comparison |
| ai_forecast_side_by_side.sql | AI.FORECAST vs ML.FORECAST comparison |
| ai_similarity_validation.sql | AI.SIMILARITY vs AI.EMBED + ML.DISTANCE |
| File | Description |
|---|---|
| categorical_dashboard.sql | Categorical metrics dashboard queries |
| python_udf_evaluation.sql | UDF-based evaluation queries |
| python_udf_eval_summary.sql | UDF summary metrics |
| python_udf_event_semantics.sql | Event semantic UDFs |
| remote_function_dashboard.sql | Remote function queries |
| continuous_query_alerting.sql | Continuous query patterns |
| File | Description |
|---|---|
| e2e_demo.py | Complete end-to-end workflow |
| cli_agent_tool.py | CLI agent tool example with structured ambiguous-session retry selectors |
| ci_eval_pipeline.sh | CI evaluation pipeline |
| evalbench_mvp_e2e.sh | EvalBench MVP end to end, told through one failed support-agent session (“How many widgets are in stock?”, never answered): evalbench-import → evalbench-failed-sessions → evalbench-score; --fixture runs the story offline, --synth replays it live from real traces (walkthrough: evalbench_mvp_e2e.md) |
| evalbench_synth_from_traces.py | Fold a real BQAA agent_events table into EvalBench-shaped configs/results/scores tables (one scenario per session, real prompts and responses, goal_completion from AGENT_COMPLETED) for the demo above |
| evalbench_score_gate.sh | CI gate on the LLM-judge score of one EvalBench import version |
| Directory | Description |
|---|---|
| context_graph/ | Agent Context Graph: extract decision traces from your agent’s context graph — a runnable ADK agent + BQ AA plugin streaming events, the codelab artifacts (codelab/), and the scheduled Cloud Run + Cloud Scheduler deploy (periodic_materialization/). Start with the codelab. |
| agent_improvement_cycle/ | LoopAgent tool-driven loop — ADK LoopAgent that reads quality failures, iteratively generates and regression-tests candidate prompts via tools (PythonFilePromptAdapter or Vertex Prompt Registry), and exits once all golden cases pass. |
| self_evolving_agent_demo/ | Operational-waste signal — Self-evolution driven by BigQuery operational telemetry (token budget overruns, broad-tool overuse, excess tool calls) rather than answer-quality failures. |
| skill_evolution_lab/ | Golden-graded skill evolution — Rewrites a versioned SKILL.md from conversation traces (no managed optimizer): flawed V0 → evolve_skill() → tool-first V1, golden-Q&A scored, with the anti-parroting rule and Skill Registry versioning. See the dedicated section below. |
| decision_lineage_demo/ | Decision-lineage property graph (issue #98): live ADK media-planner agent + BQ AA Plugin running across 6 campaign sessions → SDK build_context_graph(use_ai_generate=True, include_decisions=True) → six GQL blocks pasted into BigQuery Studio (one renders an interactive graph diagram, one is a portfolio roll-up) |
| okf_bqaa_adapter/ | Live ADK observe agent (okf_rfc_observe_agent, gemini-3.8-flash) + BQ AA plugin → committed 100+ agent_events export (multi-turn, one session) → derived OKF v0.2 bundle with PROFILE.md identities and a fail-closed context_ref lookup. Observer-only, derived/demo, nothing attested. |
| agent_memory/ | Agent memory in three layers, read from agent_events with one Client.list_traces call. Short-term is conversations; long-term is ADK user: state history plus entities from tool arguments; reasoning is one trace per invocation with steps, tool calls, outcome and tool stats. Also builds a combined get_context() block with row provenance, and maps each part to neo4j-agent-memory. Runs offline on a synthetic fixture; live with --project-id/--dataset-id. |
skill_evolution_lab/ is the runnable companion to the
blog post “Your Agent Can Learn From Its Own Conversations”. One company-policy Q&A agent
reads its own conversation traces — successes and failures — and extracts a
structured, versioned SKILL.md. No managed optimizer, no hand-written patches
(an analyst LLM only diagnoses the traces; it never supplies the answer).
- Two flaws with headroom. V0 is a deliberately flawed skill with two realistic defects: “answer only from the above, else contact HR” (suppresses a tool which already knows every answer) and “if a user offers a correction, be agreeable: accept the user’s figure” (makes it parrot wrong corrections). Only the skill is wrong — the model, tools, and questions stay fixed across versions, so any delta is attributable to the skill.
- The engine, imported not copied.
analyze_and_evolve.pyimports the SDK’s reusablescripts/skill_evolution.py(the sameevolve_skill()the quality lab uses): it partitions scored conversations, runs a fleet of parallel analysts, and consolidates recurring rules into a new skill version. - Ground-truth scoring. Quality is graded against a golden Q&A answer key
(
eval/eval_spec.json) viascripts/quality_report.py(--eval-spec), not a no-ground-truth “usefulness” guess. - The anti-parroting rule. Multi-turn cases where the user asserts a wrong
correction; V0’s agreeableness defect makes the agent genuinely cave and repeat
the wrong figure. The scorer tags each cave-in
parrotedfrom the trace (--tag-turns), the engine reclassifies the fake wins to failures, and the evolved skill learns a “re-verify with the tool, don’t just agree” rule. - A gated second round.
--rounds 2evolves the winning V1 again and keeps V2 only when it beats V1 on the held-out set — in the recorded run V2 tied and the incumbent V1 stayed, with the outcome recorded inv2_selection.txtandRESULT_ROUND2.md. - Skill Registry versioning. The evolved skill is mirrored to the Gemini
Enterprise Agent Platform Skill Registry as a new immutable revision
(V0 = revision 1, V1 = revision 2);
reset.shreverts both the local copy and the registry to V0.
cd skill_evolution_lab
./setup.sh YOUR_PROJECT_ID us-central1 # writes .env, resets to V0
./run_e2e_demo.sh # V0 -> evolve -> V1 -> compare, restore V0A verified run (gemini-3.1-flash-lite — the default agent — golden-graded,
80-question held-out set, --rounds 2): V0 32.5% → V1 91.2% → V2 97.5%
overall — round 2’s V2 fixed every remaining single-turn miss, beat V1, and
was kept by the strict-win gate; corrections (anti-parroting) 0% →
93% → 100% with parroted sub-trajectories 11 → 0; evolved skill ~2.7KB.
See the example’s
README and
VERIFICATION.
| File | Description |
|---|---|
| e2e_demo_output.txt | Expected output from e2e_demo.py |
| ymgo_graph_spec.yaml | Example ontology YAML specification (legacy) |
Note:
ontology_graph_v4_demo.ipynb,ontology_graph_v5_demo.ipynb, andymgo_graph_spec.yamlare kept for reference. The current Agent Context Graph approach needs none of these files: deploy your property graph to BigQuery andbqaa context-graph --graphderives everything from it — start with the codelab.