티스토리 수익 글 보기

티스토리 수익 글 보기

BigQuery-Agent-Analytics-SDK/examples at main · GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK · GitHub
Skip to content

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Examples

This directory contains notebooks, SQL scripts, Python demos, and reference artifacts that demonstrate SDK capabilities.

Note: the judge model is independent of the agent model by default — demos configure the agent via MODEL_NAME and all SDK-side LLM judging via a separate JUDGE_MODEL. When no explicit model/endpoint is passed, Client and LLMAsJudge silently fall back to the SDK default (gemini-2.5-flash).

Notebooks

Notebook Description
bqca_prompt_response_logging_customer_notebook.ipynb End-to-end starter playbook for evaluating BigQuery Conversational Analytics (BQCA) Prompt & Response logs (gemini-3.5-flash default judge endpoint; see the notebook’s Region guide for US/EU multi-region and single-region routing): compatibility view setup, doctor() health check, question categorization, semantic drift detection, golden Q&A grading, customer sentiment, latency/token FinOps, and error/refusal diagnostics. Companion guide: BQCA Customer Manual (Open in Colab).
dashboard_v2.ipynb Observability dashboard (2-layer SQL, no SDK dependency)
dashboard_v2_bigframes.ipynb BigFrames companion for Dashboard V2
e2e_notebook_demo.ipynb End-to-end SDK workflow
ai_ml_integration_demo.ipynb AI.GENERATE, AI.EMBED, anomaly detection
categorical_evaluation_demo.ipynb Hatteras categorical evaluation
context_graph_adcp_demo.ipynb Agent Context Graph decision-trace use cases
ontology_graph_v5_demo.ipynb OWL import, mixed extraction, temporal lineage, GQL
_archive/context_graph_historical_notebook.ipynb Archived: the original MAKO context-graph pipeline (explicit ontology + binding), kept as frozen evidence
ontology_graph_v4_demo.ipynb Ontology extraction + GQL (legacy)
memory_service_demo.ipynb Cross-session memory
event_semantics_views_bigframes_demo.ipynb Event views + BigFrames
nba_agent_trace_analysis_notebook.ipynb Real-world trace analysis

SQL — BigQuery AI Operators

File Description
ai_classify_side_by_side.sql AI.CLASSIFY vs AI.GENERATE comparison
ai_forecast_side_by_side.sql AI.FORECAST vs ML.FORECAST comparison
ai_similarity_validation.sql AI.SIMILARITY vs AI.EMBED + ML.DISTANCE

SQL — Deployment Surfaces

File Description
categorical_dashboard.sql Categorical metrics dashboard queries
python_udf_evaluation.sql UDF-based evaluation queries
python_udf_eval_summary.sql UDF summary metrics
python_udf_event_semantics.sql Event semantic UDFs
remote_function_dashboard.sql Remote function queries
continuous_query_alerting.sql Continuous query patterns

Python Scripts

File Description
e2e_demo.py Complete end-to-end workflow
cli_agent_tool.py CLI agent tool example with structured ambiguous-session retry selectors
ci_eval_pipeline.sh CI evaluation pipeline
evalbench_mvp_e2e.sh EvalBench MVP end to end, told through one failed support-agent session (“How many widgets are in stock?”, never answered): evalbench-import → evalbench-failed-sessions → evalbench-score; --fixture runs the story offline, --synth replays it live from real traces (walkthrough: evalbench_mvp_e2e.md)
evalbench_synth_from_traces.py Fold a real BQAA agent_events table into EvalBench-shaped configs/results/scores tables (one scenario per session, real prompts and responses, goal_completion from AGENT_COMPLETED) for the demo above
evalbench_score_gate.sh CI gate on the LLM-judge score of one EvalBench import version

Demo Bundles

Directory Description
context_graph/ Agent Context Graph: extract decision traces from your agent’s context graph — a runnable ADK agent + BQ AA plugin streaming events, the codelab artifacts (codelab/), and the scheduled Cloud Run + Cloud Scheduler deploy (periodic_materialization/). Start with the codelab.
agent_improvement_cycle/ LoopAgent tool-driven loop — ADK LoopAgent that reads quality failures, iteratively generates and regression-tests candidate prompts via tools (PythonFilePromptAdapter or Vertex Prompt Registry), and exits once all golden cases pass.
self_evolving_agent_demo/ Operational-waste signal — Self-evolution driven by BigQuery operational telemetry (token budget overruns, broad-tool overuse, excess tool calls) rather than answer-quality failures.
skill_evolution_lab/ Golden-graded skill evolution — Rewrites a versioned SKILL.md from conversation traces (no managed optimizer): flawed V0 → evolve_skill() → tool-first V1, golden-Q&A scored, with the anti-parroting rule and Skill Registry versioning. See the dedicated section below.
decision_lineage_demo/ Decision-lineage property graph (issue #98): live ADK media-planner agent + BQ AA Plugin running across 6 campaign sessions → SDK build_context_graph(use_ai_generate=True, include_decisions=True) → six GQL blocks pasted into BigQuery Studio (one renders an interactive graph diagram, one is a portfolio roll-up)
okf_bqaa_adapter/ Live ADK observe agent (okf_rfc_observe_agent, gemini-3.8-flash) + BQ AA plugin → committed 100+ agent_events export (multi-turn, one session) → derived OKF v0.2 bundle with PROFILE.md identities and a fail-closed context_ref lookup. Observer-only, derived/demo, nothing attested.
agent_memory/ Agent memory in three layers, read from agent_events with one Client.list_traces call. Short-term is conversations; long-term is ADK user: state history plus entities from tool arguments; reasoning is one trace per invocation with steps, tool calls, outcome and tool stats. Also builds a combined get_context() block with row provenance, and maps each part to neo4j-agent-memory. Runs offline on a synthetic fixture; live with --project-id/--dataset-id.

Skill Evolution Lab — a self-improving agent

skill_evolution_lab/ is the runnable companion to the blog post “Your Agent Can Learn From Its Own Conversations”. One company-policy Q&A agent reads its own conversation traces — successes and failures — and extracts a structured, versioned SKILL.md. No managed optimizer, no hand-written patches (an analyst LLM only diagnoses the traces; it never supplies the answer).

  • Two flaws with headroom. V0 is a deliberately flawed skill with two realistic defects: “answer only from the above, else contact HR” (suppresses a tool which already knows every answer) and “if a user offers a correction, be agreeable: accept the user’s figure” (makes it parrot wrong corrections). Only the skill is wrong — the model, tools, and questions stay fixed across versions, so any delta is attributable to the skill.
  • The engine, imported not copied. analyze_and_evolve.py imports the SDK’s reusable scripts/skill_evolution.py (the same evolve_skill() the quality lab uses): it partitions scored conversations, runs a fleet of parallel analysts, and consolidates recurring rules into a new skill version.
  • Ground-truth scoring. Quality is graded against a golden Q&A answer key (eval/eval_spec.json) via scripts/quality_report.py (--eval-spec), not a no-ground-truth “usefulness” guess.
  • The anti-parroting rule. Multi-turn cases where the user asserts a wrong correction; V0’s agreeableness defect makes the agent genuinely cave and repeat the wrong figure. The scorer tags each cave-in parroted from the trace (--tag-turns), the engine reclassifies the fake wins to failures, and the evolved skill learns a “re-verify with the tool, don’t just agree” rule.
  • A gated second round. --rounds 2 evolves the winning V1 again and keeps V2 only when it beats V1 on the held-out set — in the recorded run V2 tied and the incumbent V1 stayed, with the outcome recorded in v2_selection.txt and RESULT_ROUND2.md.
  • Skill Registry versioning. The evolved skill is mirrored to the Gemini Enterprise Agent Platform Skill Registry as a new immutable revision (V0 = revision 1, V1 = revision 2); reset.sh reverts both the local copy and the registry to V0.
cd skill_evolution_lab
./setup.sh YOUR_PROJECT_ID us-central1   # writes .env, resets to V0
./run_e2e_demo.sh                        # V0 -> evolve -> V1 -> compare, restore V0

A verified run (gemini-3.1-flash-lite — the default agent — golden-graded, 80-question held-out set, --rounds 2): V0 32.5% → V1 91.2% → V2 97.5% overall — round 2’s V2 fixed every remaining single-turn miss, beat V1, and was kept by the strict-win gate; corrections (anti-parroting) 0% → 93% → 100% with parroted sub-trajectories 11 → 0; evolved skill ~2.7KB. See the example’s README and VERIFICATION.

Reference Artifacts

File Description
e2e_demo_output.txt Expected output from e2e_demo.py
ymgo_graph_spec.yaml Example ontology YAML specification (legacy)

Note: ontology_graph_v4_demo.ipynb, ontology_graph_v5_demo.ipynb, and ymgo_graph_spec.yaml are kept for reference. The current Agent Context Graph approach needs none of these files: deploy your property graph to BigQuery and bqaa context-graph --graph derives everything from it — start with the codelab.