Text Classification from Scratch using PyTorch

The AI/ ML development framework Keras 3x supports in recent times has got support for Torch & Jax backends, in addition to Tensorflow. However, given Keras's Tensorflow legacy large sections of the code are deeply integerated with Tensorflow. 

One such piece of code is text_classification_from_scratch.py from the keras-io/ examples project. Without tensorflow this piece of code simply won't run!

Here's text_classification_torch.py a pure Torch/ PyTorch port of the same code. The bits that needed modification:

• Removing all tensorflow related imports
• Loading the Imdb text files in "grain" format in place of "tf" format, by passing the appropriate param: 

    keras.utils->text_dataset_from_directory(format="grain") 

Also grain needs to be installed:

    pip3 install grain 

• For building Vocab, Tokenizer, Vectorizing use torchtext:

    pip3 install torchtext

• Few other changes such as ensure max_features constraint's honoured, text is standardized, padded, and so on   

Guardrails & Guard-Llm's

With wide scale adoption of Llm's & Agentic models in production, there's also a pressing need to verify both the inputs & output for GenAI use cases. This should ideally be done in real-time just before serving the response to the end user. This would ensure that no invalid, harmful, hateful, confidential, etc content goes through in either direction. Guardrails are the answer to that very problem.

The simple idea with Guardrails is to apply intelligent input/ output filters that can sanitize and filter out both bad requests/ responses from getting through. There are many ways of implementing Guardrails as pattern based, rule engines, etc. Though these have worked so far, in an ever changing Agentic world it's now up to the self learning guard Llm's to judge & flag! 

Guard llm's are specifically trained to flag out harmful content. One such implementation is llama-guard which flags out violations of any of the ML Commons AI Safety Taxonomies/ Categories.

An implementation of the guard-llm can be found in the ApiCaller project. More specifically the ApiCaller->invokeWithGuardrails():

•  First calls a local Ollama model with sanitized input to get a response
•  Then calls the isSafe() method with the received response
•  isSafe() internally makes a call to a different Ollama model llama-guard which flags out the content as safe/ unsafe

Check the TestApiCaller.py test case for better clarity.

References

• https://mlcommons.org/2024/04/mlc-aisafety-v0-5-poc/
• https://www.ibm.com/think/tutorials/llm-guardrails
• https://ollama.com/library/llama-guard3

LangWatch Scenario with Ollama

LangWatch Scenario is a framework for a Agent testing based on pytest. Scenario runs with Openai compatible api's. Here we show how to get LangWatch running using local Llm's with Ollama.

The code test_ollama_client.py is along the same lines as the test_azure_api_gateway.py from the scenario python examples folder. 

Changes specific to Ollama being:

1. Set-up

    pip3 install langwatch-scenario 

Environment variables

    export OPENAI_API_BASE_URL=http://localhost:11434/api/
    export OPENAI_API_KEY=NOTHING

2. Create Ollama client

    ollama_client() -> OpenAI(base_url=<OLLAMA_BASE_URL>)

3. Configuring the Ollama model (gemma, etc) & custom_llm_provider ("ollama") in the Agents (UserSimulatorAgent & JudgeAgent)           

    scenario.UserSimulatorAgent(model=OLLAMA_MODEL, client=custom_client, custom_llm_provider=CUSTOM_LLM_PROVIDER)...

For better clarity see test_ollama_client.py.

4. Offline LangWatch Scenario Reporter

For every run LangWatch uploads run results to app.langwatch.ai endpoint. For a truly offline run set the LANGWATCH_ENDPOINT location: 

    export LANGWATCH_ENDPOINT= <https://YOUR_REPORTING_ENDPOINT>

There's no option to disable scenario reporting for now. Only work around is to set  to LANGWATCH_ENDPOINT to an invalid value (eg "http://localhost2333/invalid").

 

DeepEval

DeepEval helps to test and verify the correctness of LLMs. DeepEval is a framework with a suite of Metrics, Synthetic Data generation having integrations across all leading AI/ ML libraries. 

DeepEval can be used to set-up one LLM to judge the output of another LLM. This JudgeLLM set-up can be used at both the training as well as live inference stage for MlOps scenarios.

Getting started with DeepEval is simple with Ollama. 

(I) Installation

    pip install deepeval

Ollama installation was covered previously with a llama3.2 base model. 

(II) Set-Ollama model in DeepEval

# Unset the openai model - default for DeepEval     

deepeval unset-openai

# Set ollama model for DeepEval 

deepeval set-ollama "llama3.2:1b" --base-url="http://localhost:11434"  

(III) Create a JudgeLLM.py code

# Set up ollama model

model = OllamaModel(
  model="llama3.2:1b",
  base_url="http://localhost:11434",
  temperature=0.0,  # Example: Setting a custom temperature
) 

# Set up evaluation metrics 

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct based on the expected output.",
    # NOTE: you can only provide either criteria or evaluation_steps, and not both
    evaluation_steps=[
        "Check whether the facts are true"    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
   model=model, # ollama model
rubric=[
        Rubric(score_range=(0,2), expected_outcome="Factually incorrect."),
        Rubric(score_range=(3,6), expected_outcome="Mostly correct."),
        Rubric(score_range=(7,9), expected_outcome="Correct but missing minor details."),
        Rubric(score_range=(10,10), expected_outcome="100% correct."),
    ],
#    threshold=0.1   
) 

# define the test case

test_case_maths = LLMTestCase(
    input="what is 80 in words? using only 1 word.",
    actual_output="eighty",
    expected_output="eighty"
) 

 # Run the evaluation

evaluate(test_cases=[test_case_maths], metrics=[answer_relevancy]) 

(IV) Execute the JudgeLLM.py 

 deepeval test run JudgeLLM.py 

 

Codegen LLMs

One GenAI feature making headlines is coding. GenAI apps are getting better with reading and writing code (codegen) in various programming languages such as Python, Java, C++ and so on.

On the evaluation side there are all kinds of benchmarks and leaderboards that track progress on codegen. Additionally, aspects of usability, platform support, IDE integration, etc are all key factors for using codegen.

In terms of local evaluations, Ollama provides handy options. With Ollama it's easy to download and run LLMs from various providers (Llama, Gemma, etc). Most now support codegen and readily follow instructions in a chat to churn out basic level Python code:

•  Llama3.2
• Gemma3
• Codegemma 
• Falcon3
• Starcoder