Smores - Searching while modelling perceptually degraded environments for time critical rescues

Demonstration and Discussion

Recent events have shed light on the importance of arming first responders with critical information about their environments, especially in hazardous environments like an active fire. Through smores, we aim to deliver an autonomous drone capable of mapping the environment through dense smoke and annotating points of Interests (POIs) on this map so firefighters can take informed actions and utilize resources more efficiently.

The different technical verticals of our system are:

• Real-time Mapping: Using a stereo thermal camera setup on the drone to perceive through dense smoke and recovering dense depth. Using the dense depth we can reconstruct a pointcloud representation of the environment.
• State Estimation: Fusing RGB cameras, thermal cameras, and 9 DOF IMU to predict state through dense smoke. The recovered odometry is crucial for autonomy and combining separate pointclouds of a specific scene into one combined representation.
• Autonomy: Since firefighters have extremely limited manpower, we aim to provide safe and autonomous flight so we can provide crucial information at no additional cost to firefighters.
• Identifying POIs: Recognizing and annotating points of interests on the map such as humans, open doors, and origin of fires on the reconstructed map to allow for informed planning for firefighters.

In order to deliver a solution of the highest quality, we decided to build our own platform. In addition to building the hardware and electrical subsystems, we are focusing on accelerating our depth perception pipeline to work in real-time since we are working with limited computing power (NVIDIA Jetson AGX Orin). Our coming priorities are relaying all information in real-time to the ground control station where the incident commander can utilize this information to plan a response.

My focus so far has been leading the development on the perception pipeline to recover dense depth and making a consistent 3D reconstruction of the environment. In conjuction, I have been overseeing the electrical subsystem and sensor interfacing for the build of the drone. Finally, in the coming semster, I will be focusing on accelerating existing machine learning models to work in real-time and creating models for detecting points of interests.

Project Website: Link
GitHub Code Repository: Private Repository