Skip to main content

Vision-Language-Action Models

Vision-Language-Action (VLA): The Convergence of LLMs and Robotics​

This module explores the exciting field of Vision-Language-Action (VLA) models, where we bridge the gap between human language and robotic action.

From Voice to Action​

We will use OpenAI Whisper to enable our robot to understand voice commands. This is the first step in creating a natural and intuitive human-robot interface.

Cognitive Planning with LLMs​

The true power of VLA comes from using Large Language Models (LLMs) for cognitive planning. You will learn how to translate high-level natural language commands, such as "Clean the room," into a sequence of executable ROS 2 actions.

Capstone Project: The Autonomous Humanoid​

This module culminates in a capstone project where you will build an autonomous humanoid robot in simulation. The robot will be able to:

  1. Receive a voice command.
  2. Plan a path to a target location.
  3. Navigate through a cluttered environment, avoiding obstacles.
  4. Identify a target object using computer vision.
  5. Manipulate the object.

Week 13: Conversational Robotics​

In the final week, we will focus on conversational robotics:

  • GPT Integration: Integrate GPT models to create a conversational AI for your robot.
  • Speech and Language: Dive into speech recognition and natural language understanding.
  • Multi-modal Interaction: Explore how to combine speech, gestures, and vision for a rich interactive experience.

Assessments​

  • ROS 2 package development project
  • Gazebo simulation implementation
  • Isaac-based perception pipeline
  • Capstone: Simulated humanoid robot with conversational AI