Reinforcement Learning

An important ML training paradigm is Reinforcement Learning (RL). RL models rely on a reward value generated at the end of each training run/ epoch to update the parameters (weights) of the model. This is different from the other ML methods such as Supervised Learning where labelled data/ examples are given from which the models learns. It's is also different from the Unsupervised Learning approach where inherent features of the unlabeled data are explored used by the model through the learning phase to identify clusters, etc.    

The keras-io examples has some RL implementations such as actor_critic, ppo, etc. All of them work solely with the TensorFlow (tf) backend. In keras_io_examples_rl these have been ported to the Torch/ PyTorch backend. The typical changes include:   

• Torch Imports 
• Use torch specific Optimizer - torch.optim.Adam
• deep_q_network_breakout_pytorch () requires grad_clipping, in torch done before optimizer.step() 
• Gradient computations in torch 
• Replace tf GradientTape with torch autograd 
• Disable gradient globally torch.set_grad_enabled(False)
• Enable autograd within specific flows/ methods where needed
• Call loss.backward(), optimizer.step() for backpropagation
• Few torch specific tensor & function changes/ wrappers  

The ported pytorch compatible files are:

• actor_critic_cartpole_pytorch.py
• ddpg_pendulum_pytorch.py
• deep_q_network_breakout_pytorch.py
• ppo_cartpole_pytorch.py

References

• http://www.derongliu.org/adp/adp-cdrom/Barto1983.pdf
• https://hal.inria.fr/hal-00840470/document
• https://link.springer.com/content/pdf/10.1007/BF00992698.pdf
• https://www.semanticscholar.org/paper/Human-level-control-through-deep-reinforcement-Mnih-Kavukcuoglu/340f48901f72278f6bf78a04ee5b01df208cc508
• Continuous control with deep reinforcement learning: https://arxiv.org/abs/1509.02971)
• Deep Deterministic Policy Gradient (DDPG) 
• https://gymnasium.farama.org/
• Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto

Drift Detection across Distinct Reviews Datasets

 Model Drift leads to invalid results from AI/ ML inference models in production. There could be various causes of Drift such as conceptual drift, structural changes and ingestion pipeline issues with upstream data sources, domain change, prompt injections and other model exploits, etc. These lead to the AI/ ML model that was trained on a certain kind(s) of data having to run inferences on completely different drifted data which causes to wrong/ incorrect results. So Drift detection (periodical, near real-time, etc) is crucial for any productionized model. 

As mentioned previously Evidently is a handy library to do drift detection. Evidently has features like Metrics, Descriptors, Eval, etc that can be plugged in to detect drift in the current data vis-a-vis a reference baseline data (~training data).

In the DriftTextReviews.py Drift detection is done for an existing Text Classification model in PyTorch originally trained on an Imdb Movie's review dataset.  For Reference data a sample of the same Imdb Movie data is used. For Current, data from a completely different domain of Code Reviews is used. As expected, there was significant drift detected for these two datasets that belong to two completely different domains. Evidently reports below make the drift evidently clear!

• The characteristic words have changed across the two domains. While the movie domain includes words like frame, character, minutes, etc, the coding domain has words like readable, test, method, etc. 
• In terms of Length of review text, Imdb reviews are much much longer and include many more words than the Code reviews. These word length and count features hooked in as Descriptors are duly detected and shown in the reports.
• Interestingly, the Label either Positive (1) or Negative (0) shows no Drift. Across both datasets equal no of the two classes Positive & Negative is seen.

 

Fig 1: Drift Review Length & Word Count	Fig 2: No Drift in Label
Fig 3: Characteristic Words - Current	Fig 4: Characteristic Words - Reference

LangWatch Scenario with Ollama

LangWatch Scenario is a framework for a Agent testing based on pytest. Scenario runs with Openai compatible api's. Here we show how to get LangWatch running using local Llm's with Ollama.

The code test_ollama_client.py is along the same lines as the test_azure_api_gateway.py from the scenario python examples folder. 

Changes specific to Ollama being:

1. Set-up

    pip3 install langwatch-scenario 

Environment variables

    export OPENAI_API_BASE_URL=http://localhost:11434/api/
    export OPENAI_API_KEY=NOTHING

2. Create Ollama client

    ollama_client() -> OpenAI(base_url=<OLLAMA_BASE_URL>)

3. Configuring the Ollama model (gemma, etc) & custom_llm_provider ("ollama") in the Agents (UserSimulatorAgent & JudgeAgent)           

    scenario.UserSimulatorAgent(model=OLLAMA_MODEL, client=custom_client, custom_llm_provider=CUSTOM_LLM_PROVIDER)...

For better clarity see test_ollama_client.py.

4. Offline LangWatch Scenario Reporter

For every run LangWatch uploads run results to app.langwatch.ai endpoint. For a truly offline run set the LANGWATCH_ENDPOINT location: 

    export LANGWATCH_ENDPOINT= <https://YOUR_REPORTING_ENDPOINT>

There's no option to disable scenario reporting for now. Only work around is to set  to LANGWATCH_ENDPOINT to an invalid value (eg "http://localhost2333/invalid").

 

Agent2Agent (A2A) with a2a-sdk and Http2

Continuing with A2A evaluation next up is a2a-sdk (unrelated to previously evaluated a2a-server). This evaluation is largely based on getting the hello world from the a2a-samples project working as per the instruction of a2a-protocol. With additional, integration with other Http2 based non Python clients.

(I) Installation

pip install a2a-sdk 

# uvicorn python-dotenv (packages existing) 

# For Http2 support 

pip install hypercorn 

pip install h2==4.2.0 (See Issue 1 at the end & the bug details) 

git clone https://github.com/a2aproject/a2a-samples.git -b main --depth 1

(II) Replace uvicorn server with hypercorn (support for Http2) 

The a2a-samples make use of the uvicorn python server. However, uvicorn is a Http1.x compliant server and doesn't support Http2. Keep seeing the following messages if client requests from Http2: 

"WARNING:  Unsupported upgrade request. "

In order to support a wider & more updated category of clients, uvicorn is replaced with a hypercorn which is Http2 compliant.

In order to switch to hypercorn, the following changes are done to _main_.py of helloworld python project: 

#import uvicorn
 

# Use Hypercorn for Http2
import asyncio
from hypercorn.config import Config
from hypercorn.asyncio import serve

 ....

    config = Config()
    config.bind="127.0.0.1:8080"  # Binds to all interfaces on port 8080

    asyncio.run(serve(server.build(), config))
   # uvicorn.run(server.build(), host='127.0.0.1', port=8080, log-level='debug') 

(III) Run helloworld

python a2a-samples/samples/python/agents/helloworld/__main__.py 

(IV) View AgentCard

Open in the browser or via curl:

curl http:///127.0.0.1:8080/.well-known/agent-card.json

Response: 

{"capabilities":{"streaming":true},"defaultInputModes":["text"],"defaultOutputModes":["text"],"description":"Just a hello world agent","name":"Hello World Agent","preferredTransport":"JSONRPC","protocolVersion":"0.3.0","skills":[{"description":"just returns hello world","examples":["hi","hello world"],"id":"hello_world","name":"Returns hello world","tags":["hello world"]}],"supportsAuthenticatedExtendedCard":true,"url":"http://127.0.0.1:8080/","version":"1.0.0"} 

For the Authorized Extended Agent Card:

curl -H "Authorization: Bearer dummy-token-for-extended-card" --http2 http://127.0.0.1:8080/agent/authenticatedExtendedCard 

Response: 

{"capabilities":{"streaming":true},"defaultInputModes":["text"],"defaultOutputModes":["text"],"description":"The full-featured hello world agent for authenticated users.","name":"Hello World Agent - Extended Edition","preferredTransport":"JSONRPC","protocolVersion":"0.3.0","skills":[{"description":"just returns hello world","examples":["hi","hello world"],"id":"hello_world","name":"Returns hello world","tags":["hello world"]},{"description":"A more enthusiastic greeting, only for authenticated users.","examples":["super hi","give me a super hello"],"id":"super_hello_world","name":"Returns a SUPER Hello World","tags":["hello world","super","extended"]}],"supportsAuthenticatedExtendedCard":true,"url":"http://127.0.0.1:8080/","version":"1.0.1"} 

(V) Send/ Receive message to Agent

curl -H "Content-Type: application/json"  http:///127.0.0.1:8080 -d '{"jsonrpc":"2.0","id":"ee22f765-0253-40a0-a29f-c786b090889d","method":"message/send","params":{"message":{"role":"user","parts":[{"text":"hello there!","kind":"text"}],"messageId":"ccaf4715-712e-40c6-82bc-634a7a7136f2","kind":"message"},"configuration":{"blocking":false}}}' 

Response: 

 {"id":"ee22f765-0253-40a0-a29f-c786b090889d","jsonrpc":"2.0","result":{"kind":"message","messageId":"d813fed8-58cd-4337-8295-6282930d4d4e","parts":[{"kind":"text","text":"Hello World"}],"role":"agent"}}

(VI) Send/ Receive via Http2

curl -iv --http2 http://127.0.0.1:8080/.well-known/agent-card.json

& 

curl -iv --http2  -H "Content-Type: application/json"  http://127.0.0.1:8080 -d '{"jsonrpc":"2.0","id":"ee22f765-0253-40a0-a29f-c786b090889d","method":"message/send","params":{"message":{"role":"user","parts":[{"text":"dragons and wizards","kind":"text"}],"messageId":"ccaf4715-712e-40c6-82bc-634a7a7136f2","kind":"message"},"configuration":{"blocking":false}}}'

(The responses are the same as shown above)

(VII) Send/ Receive from Java client

TBD

(VIII) Issues 

Issue 1: Compatibility issue with hypercorn (ver=0.17.3) & latest h2 (ver=4.3.0)

Ran in to the issue in the mentioned here:

    |   File "/home/algo/Tools/venv/langvang/lib/python3.13/site-packages/hypercorn/protocol/h2.py", line 138, in initiate
    |     event = h2.events.RequestReceived()
    | TypeError: RequestReceived.__init__() missing 1 required keyword-only argument: 'stream_id' 

Issue was resolved by downgrading to h2 (ver=4.2.0).

 

AI Agentic Frameworks

With prolification of AI Agents, it's only logical that there will be attempts at standardization and building protocols & frameworks:

• MCP covered previously
• Any-Agent from Mozilla.ai to switch between agents, vendors, clouds, etc
• Agent2Agent interoperability protocol