Unlocking the Power of LLMs

NOTE: Hands-On AI Software Training – See Requirements & Prerequisites

Workshop Title: Unlocking the Power of LLMs for Customizable Problem-Solving via LLAMA

Presenters: Prakash Murugesan and Srihari Jayakumar, Meta

Abstract :

This workshop covers the use of LLM agents ( using Llama ) and how to integrate them into different real life applications. Participants will get hands-on experience with Llama by learning how to extend a basic agent to connect to a database for data retrieval & how to chain GenAI models together to generate animations. The workshop will also cover security and privacy considerations when working with LLM agents.

Prior knowledge or software for attendees :

  • Intermediate Python: You need to know how to set up a virtual environment, install the required packages, and jupyter-lab.
  • Git/GitHub: The content and base files for this workshop are hosted in GitHub, you need to be able to clone the repository.
  • LLM Agents: You should have played around with Chat-GPT, Llama or any other AI Agent in the past.
  • Basic SQL: One demo showcases how to translate natural language to SQL.

Technical requirements:

  • Laptop with GPU capabilities and 10GB of disk space.
  • Python 3.9, pip, and virtual environment.
  • Git/GitHub Client.
  • A Python IDE.

Pre-Workshop Requisites:

Go over the pre-workshop requisites tutorial to prepare your system for the workshop. It requires installing Ollama, and downloading ~5GB of data, therefore it is better to do it before the workshop. See https://ollama.com/download/mac and https://github.com/cgamamx/llm-workshop

Agenda

Note: If time permits, we will cover all four interactive use cases during the workshop. However, if we are short on time, demos #3 and #4 will be assigned as take-home exercises for participants to complete on their own.

  1. Pre-Workshop Requisites Check
  2. Introduction
    • LLMs and LLaMa Overview
    • What is RAG?
    • LLM Agent Architecture
    • Security and Privacy considerations
  3. Interactive Session
    • Basic Agent: Dive into a Jupyter notebook and get familiar with a basic agent. Inspect each block, make changes, and see how it affects the agent’s behavior.
    • Data Retrieval: Learn how to extend the basic agent to connect to a database for data retrieval. Discover how this approach can be applied to fetch information from third-party APIs or RESTful endpoints.
    • GenAI Model Chaining: Witness the power of chaining GenAI models together. See how Llama can create prompts for diffusion models to generate animations.
    • Web Application: Take an LLM agent from a notebook and transform it into a Flask application. Experience the process of bringing your agent to life on the web.
  4. Final Notes
    • More examples of industry applications.
    • Instructions to deploy on cloud.
    • Instructions to deploy on premises.

Presenter Bios:

Prakash Murugesan, Machine Learning Engineer, Meta

Prakash Murugesan is a Machine Learning Engineer at Meta, leading the development of Multimodal LLM foundational models for international expansion. He has been instrumental in building the AI systems powering Meta’s Ray-Ban Smart Glasses, leveraging advanced multimodal techniques to integrate vision and language for an enhanced user experience. Previously, Prakash played a pivotal role in building the ML models responsible for growing Instagram Reels from a new product to one used globally by over a billion users. With a focus on scalable machine learning solutions, Prakash continues to push the boundaries of wearable AI technologies.

Linkedin: https://www.linkedin.com/in/prakash-murugesan/

Srihari Jayakumar, Machine Learning Engineer and Researcher, Meta

Srihari Jayakumar is a machine learning engineer and researcher at Meta. Srihari’s primary research interests span computer vision (scene text recognition), multimodal large language models and NLP in general. Srihari has successfully led projects involving the development of multimodal AI capabilities for wearable devices, including the Ray-Ban Meta smart glasses. With a strong technical background in machine learning, Srihari is focused on creating innovative and impactful technology solutions.

Linkedin: https://www.linkedin.com/in/srihari-j/