Construct a Multimodal Agent for Product Ingredient Evaluation

Have you ever ever discovered your self watching a product’s elements record, googling unfamiliar chemical names to determine what they imply? It’s a standard wrestle – deciphering advanced product info on the spot could be overwhelming and time-consuming. Conventional strategies, like trying to find every ingredient individually, typically result in fragmented and complicated outcomes. However what if there was a wiser, sooner technique to analyze product elements and get clear, actionable insights immediately? On this article, we’ll stroll you thru constructing a Product Elements Analyzer utilizing Gemini 2.0, Phidata, and Tavily Net Search. Let’s dive in and make sense of these ingredient lists as soon as and for all!

Studying Goals

• Design a Multimodal AI Agent structure utilizing Phidata and Gemini 2.0 for vision-language duties.
• Combine Tavily Net Search into agent workflows for higher context and knowledge retrieval.
• Construct a Product Ingredient Analyzer Agent that mixes picture processing and net seek for detailed product insights.
• Learn the way system prompts and directions information agent habits in multimodal duties.
• Develop a Streamlit UI for real-time picture evaluation, diet particulars, and health-based recommendations.

This text was printed as part of the Information Science Blogathon.

What are Multimodal Methods?

Multimodal programs course of and perceive a number of varieties of enter information—like textual content, photographs, audio, and video—concurrently. Imaginative and prescient-language fashions, comparable to Gemini 2.0 Flash, GPT-4o, Claude Sonnet 3.5, and Pixtral-12B, excel at understanding relationships between these modalities, extracting significant insights from advanced inputs.

On this context, we deal with vision-language fashions that analyze photographs and generate textual insights. These programs mix pc imaginative and prescient and pure language processing to interpret visible info based mostly on consumer prompts.

Multimodal Actual-world Use Circumstances

Multimodal programs are reworking industries:

• Finance: Customers can take screenshots of unfamiliar phrases in on-line varieties and get prompt explanations.
• E-commerce: Buyers can {photograph} product labels to obtain detailed ingredient evaluation and well being insights.
• Schooling: College students can seize textbook diagrams and obtain simplified explanations.
• Healthcare: Sufferers can scan medical experiences or prescription labels for simplified explanations of phrases and dosage directions.

Why Multimodal Agent?

The shift from single-mode AI to multimodal brokers marks a serious leap in how we work together with AI programs. Right here’s what makes multimodal brokers so efficient:

• They course of each visible and textual info concurrently, delivering extra correct and context-aware responses.
• They simplify advanced info, making it accessible to customers who could wrestle with technical phrases or detailed content material.
• As a substitute of manually trying to find particular person elements, customers can add a picture and obtain complete evaluation in a single step.
• By combining instruments like net search and picture evaluation, they supply extra full and dependable insights.

Constructing Product Ingredient Analyzer Agent

Let’s break down the implementation of a Product Ingredient Evaluation Agent:

Step 1: Setup Dependencies 

• Gemini 2.0 Flash: Handles multimodal processing with enhanced imaginative and prescient capabilities
• Tavily Search: Gives net search integration for extra context
• Phidata: Orchestrates the Agent system and manages workflows
• Streamlit: To develop the prototype into Net-based functions.

!pip set up phidata google-generativeai tavily-python streamlit pillow

Step 2: API Setup and Configuration 

On this step, we’ll arrange the setting variables and collect the required API credentials to run this use case. 

from phi.agent import Agent
from phi.mannequin.google import Gemini # wants a api key
from phi.instruments.tavily import TavilyTools # additionally wants a api key

import os
TAVILY_API_KEY = ""
GOOGLE_API_KEY = ""
os.environ['TAVILY_API_KEY'] = TAVILY_API_KEY
os.environ['GOOGLE_API_KEY'] = GOOGLE_API_KEY

Step 3: System immediate and Directions

To get higher responses from language fashions, you’ll want to write higher prompts. This includes clearly defining the function and offering detailed directions within the system immediate for the LLM.

Let’s outline the function and obligations of an Agent with experience in ingredient evaluation and diet. The directions ought to information the Agent to systematically analyze meals merchandise, assess elements, think about dietary restrictions, and consider well being implications.

SYSTEM_PROMPT = """
You're an knowledgeable Meals Product Analyst specialised in ingredient evaluation and diet science. 
Your function is to investigate product elements, present well being insights, and establish potential issues by combining ingredient evaluation with scientific analysis. 
You make the most of your dietary information and analysis works to offer evidence-based insights, making advanced ingredient info accessible and actionable for customers.
Return your response in Markdown format. 
"""

INSTRUCTIONS = """
* Learn ingredient record from product picture 
* Bear in mind the consumer will not be educated concerning the product, break it down in easy phrases like explaining to 10 12 months child
* Establish synthetic components and preservatives
* Test towards main dietary restrictions (vegan, halal, kosher). Embrace this in response. 
* Charge dietary worth on scale of 1-5
* Spotlight key well being implications or issues
* Recommend more healthy options if wanted
* Present temporary evidence-based suggestions
* Use Search instrument for getting context
"""

Step 4: Outline the Agent Object

The Agent, constructed utilizing Phidata, is configured to course of markdown formatting and function based mostly on the system immediate and directions outlined earlier. The reasoning mannequin used on this instance is Gemini 2.0 Flash, identified for its superior skill to know photographs and movies in comparison with different fashions.

For instrument integration, we’ll use Tavily Search, a sophisticated net search engine that gives related context straight in response to consumer queries, avoiding pointless descriptions, URLs, and irrelevant parameters.

agent = Agent(
    mannequin = Gemini(id="gemini-2.0-flash-exp"),
    instruments = [TavilyTools()],
    markdown=True,
    system_prompt = SYSTEM_PROMPT,
    directions = INSTRUCTIONS
)

Step 5: Multimodal – Understanding the Picture

With the Agent elements now in place, the subsequent step is to offer consumer enter. This may be completed in two methods: both by passing the picture path or the URL, together with a consumer immediate specifying what info must be extracted from the supplied picture.

Strategy: 1 Utilizing Picture Path

agent.print_response(
    "Analyze the product picture",
    photographs = ["images/bournvita.jpg"],
    stream=True
)

Output:

Strategy: 2 Utilizing URL

agent.print_response(
    "Analyze the product picture",
    photographs = ["https://beardo.in/cdn/shop/products/9_2ba7ece4-0372-4a34-8040-5dc40c89f103.jpg?v=1703589764&width=1946"],
    stream=True
)

Output:

Step 6: Develop the Net App utilizing Streamlit

Now that we all know the way to execute the Multimodal Agent, let’s construct the UI half utilizing Streamlit. 

import streamlit as st
from PIL import Picture
from io import BytesIO
from tempfile import NamedTemporaryFile

st.title("🔍 Product Ingredient Analyzer")

To optimize efficiency, outline the Agent inference below a cached perform. The cache decorator helps enhance effectivity by reusing the Agent occasion.

Since we’re utilizing Streamlit, which refreshes the complete web page after every occasion loop or widget set off, including st.cache_resource ensures the perform isn’t refreshed and saves it within the cache.

@st.cache_resource
def get_agent():
    return Agent(
        mannequin=Gemini(id="gemini-2.0-flash-exp"),
        system_prompt=SYSTEM_PROMPT,
        directions=INSTRUCTIONS,
        instruments=[TavilyTools(api_key=os.getenv("TAVILY_API_KEY"))],
        markdown=True,
    )

When a brand new picture path is supplied by the consumer, the analyze_image perform runs and executes the Agent object outlined in get_agent. For real-time seize and the choice to add photographs, the uploaded file must be saved briefly for processing.

The picture is saved in a short lived file, and as soon as the execution is accomplished, the momentary file is deleted to unencumber sources. This may be completed utilizing the NamedTemporaryFile perform from the tempfile library. 

def analyze_image(image_path):
    agent = get_agent()
    with st.spinner('Analyzing picture...'):
        response = agent.run(
            "Analyze the given picture",
            photographs=[image_path],
        )
        st.markdown(response.content material)

def save_uploaded_file(uploaded_file):
    with NamedTemporaryFile(dir=".", suffix='.jpg', delete=False) as f:
        f.write(uploaded_file.getbuffer())
        return f.identify

For a greater consumer interface, when a consumer selects a picture, it’s prone to have various resolutions and sizes. To take care of a constant format and correctly show the picture, we will resize the uploaded or captured picture to make sure it matches clearly on the display.

The LANCZOS resampling algorithm gives high-quality resizing, notably useful for product photographs the place textual content readability is essential for ingredient evaluation.

MAX_IMAGE_WIDTH = 300

def resize_image_for_display(image_file):
    img = Picture.open(image_file)
    
    aspect_ratio = img.top / img.width
    new_height = int(MAX_IMAGE_WIDTH * aspect_ratio)
    img = img.resize((MAX_IMAGE_WIDTH, new_height), Picture.Resampling.LANCZOS)
    
    buf = BytesIO()
    img.save(buf, format="PNG")
    return buf.getvalue()

Step 7: UI Options for Streamlit 

The interface is split into three navigation tabs the place the consumer can decide his selection of pursuits:

• Tab-1: Instance Merchandise that customers can choose to check the app
• Tab-2: Add an Picture of your selection if it’s already saved.
• Tab-3: Seize or Take a stay photograph and analyze the product.

We repeat the identical logical move for all the three tabs:

• First, select the picture of your selection and resize it to show on the Streamlit UI utilizing st.picture.
• Second, save that picture in a short lived listing to course of it to the Agent object.
• Third, analyze the picture the place the Agent execution will happen utilizing Gemini 2.0 LLM and Tavily Search instrument.

State administration is dealt with by way of Streamlit’s session state, monitoring chosen examples and evaluation standing. 

def major():
    if 'selected_example' not in st.session_state:
        st.session_state.selected_example = None
    if 'analyze_clicked' not in st.session_state:
        st.session_state.analyze_clicked = False
    
    tab_examples, tab_upload, tab_camera = st.tabs([
        "📚 Example Products", 
        "📤 Upload Image", 
        "📸 Take Photo"
    ])
    
    with tab_examples:
        example_images = {
            "🥤 Power Drink": "photographs/bournvita.jpg",
            "🥔 Potato Chips": "photographs/lays.jpg",
            "🧴 Shampoo": "photographs/shampoo.jpg"
        }
        
        cols = st.columns(3)
        for idx, (identify, path) in enumerate(example_images.objects()):
            with cols[idx]:
                if st.button(identify, use_container_width=True):
                    st.session_state.selected_example = path
                    st.session_state.analyze_clicked = False
    
    with tab_upload:
        uploaded_file = st.file_uploader(
            "Add product picture", 
            sort=["jpg", "jpeg", "png"],
            assist="Add a transparent picture of the product's ingredient record"
        )
        if uploaded_file:
            resized_image = resize_image_for_display(uploaded_file)
            st.picture(resized_image, caption="Uploaded Picture", use_container_width=False, width=MAX_IMAGE_WIDTH)
            if st.button("🔍 Analyze Uploaded Picture", key="analyze_upload"):
                temp_path = save_uploaded_file(uploaded_file)
                analyze_image(temp_path)
                os.unlink(temp_path) 
    
    with tab_camera:
        camera_photo = st.camera_input("Take an image of the product")
        if camera_photo:
            resized_image = resize_image_for_display(camera_photo)
            st.picture(resized_image, caption="Captured Photograph", use_container_width=False, width=MAX_IMAGE_WIDTH)
            if st.button("🔍 Analyze Captured Photograph", key="analyze_camera"):
                temp_path = save_uploaded_file(camera_photo)
                analyze_image(temp_path)
                os.unlink(temp_path) 
    
    if st.session_state.selected_example:
        st.divider()
        st.subheader("Chosen Product")
        resized_image = resize_image_for_display(st.session_state.selected_example)
        st.picture(resized_image, caption="Chosen Instance", use_container_width=False, width=MAX_IMAGE_WIDTH)
        
        if st.button("🔍 Analyze Instance", key="analyze_example") and never st.session_state.analyze_clicked:
            st.session_state.analyze_clicked = True
            analyze_image(st.session_state.selected_example)

Necessary Hyperlinks

• You will discover the total code right here.
• Change the “” placeholder together with your keys.
• For tab_examples, you’ll want to have a folder picture. And save the pictures over there. Right here is the GitHub URL with photographs listing right here.
• If you’re fascinated with utilizing the use case, right here is the deployed App right here.

Conclusion 

Multimodal AI brokers signify a larger leap ahead in how we will work together with and perceive advanced info in our every day lives. By combining imaginative and prescient processing, pure language understanding, and net search capabilities, these programs, just like the Product Ingredient Analyzer, can present prompt, complete evaluation of merchandise and their elements, making knowledgeable decision-making extra accessible to everybody.

Key Takeaways

• Multimodal AI brokers enhance how we perceive product info. They mix textual content and picture evaluation.
• With Phidata, an open-source framework, we will construct and handle agent programs. These programs use fashions like GPT-4o and Gemini 2.0.
• Brokers use instruments like imaginative and prescient processing and net search. This makes their evaluation extra full and correct. LLMs have restricted information, so brokers use instruments to deal with advanced duties higher.
• Streamlit makes it straightforward to construct net apps for LLM-based instruments. Examples embody RAG and multimodal brokers.
• Good system prompts and directions information the agent. This ensures helpful and correct responses.

Regularly Requested Questions

The media proven on this article isn’t owned by Analytics Vidhya and is used on the Creator’s discretion.

Information Scientist at AI Planet || YouTube- AIWithTarun || Google Developer Knowledgeable in ML || Received 5 AI hackathons || Co-organizer of TensorFlow Person Group Bangalore || Pie & AI Ambassador at DeepLearningAI