Skip to content

Object Detection with Spoken Audio Feedback for Blind Assist - #522

Closed
tanmayskotadia wants to merge 17 commits into
tensorflow:masterfrom
tanmayskotadia:master
Closed

tanmayskotadia wants to merge 17 commits into
tensorflow:masterfrom
tanmayskotadia:master

Conversation

@tanmayskotadia

Copy link
Copy Markdown

Summary

  • Detect objects in real time using the Android device's on-device object detection model.
  • Convert detected object labels into spoken announcements.
  • Send the generated audio from the Android device to a Raspberry Pi over TCP.
  • Play the announcements through wired earphones connected to the Raspberry Pi's 3.5 mm audio jack.
  • Hide the visual detection overlays to support a voice-first experience.

Purpose

This project aims to assist visually impaired users by detecting nearby objects and announcing their names through earphones, helping users become aware of objects around them without needing to view the screen.

Architecture

Android device (object detection and text-to-speech) → TCP audio transfer → Raspberry Pi 3 (audio playback) → Wired earphones.

Testing

  • Verified that WAV audio plays through the Raspberry Pi's headphone jack.
  • Further end-to-end testing should confirm reliable object detection, audio transfer, and spoken announcements.

Added TextToSpeech functionality to provide audio warnings.
Removed unnecessary comments and simplified license information.
Added required permissions and features for camera and internet access.
Removed unnecessary comments from AndroidManifest.xml.
Added Text-to-Speech functionality to announce detected objects and improved code structure.
Added permissions for camera and internet access, and allowed discovery of Text-to-Speech engines.
Refactor CameraFragment to improve readability and maintainability by organizing constants, enhancing text-to-speech functionality, and ensuring proper resource management.
Updated CameraFragment to improve text-to-speech functionality and added Raspberry Pi audio transmission.
Removed unused variables and methods related to text-to-speech and camera lifecycle management.
Added text-to-speech functionality for object detection announcements, including initialization, synthesis, and audio transmission to Raspberry Pi.
Hide visual overlays for bounding boxes and labels while keeping detection active. Update announcement message format for detected objects.
@google-ml-butler google-ml-butler Bot added size:XL CL Change Size: Extra Large awaiting review labels Oct 18, 2026
@google-cla

google-cla Bot commented Oct 18, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL CL Change Size: Extra Large

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant