Gesture-based Map Interaction

2023Fall_gestures_video_zaruba.mp4

Map Exploration with Gestures

Spatial Data Scientist: Luke Zaruba

Degree Program: Masters of Geographic Information Science

Overview

Geovisualization is a discipline that has roots stretching far back to the early cartographers of the world. Now, with powerful new technologies like Artificial Intelligence (AI), a whole new set of possibilities is opened up in the field to explore new and interesting ways to interact with our representations of the world. Using hand gestures to power this interaction is just one of the many possibilities.

This project provides a simple, intuitive way to interact with maps by using AI-powered gesture recognition capabilities to interpret users’ hand gestures and movement. The project was inspired by Steve Attewell’s work using gestures, voice recognition, and other interactive means to control maps.

Supporting User Interaction

Users are able to pan the map, zoom in and out, return to the starting map extent, and exit the gesture recognition mode. In its current form, there are certainly limitations as far as the smoothness of the map movement is concerned, as well as some lag time between gestures and their detection in certain instances. It can be tricky for the AI models to accurately detect gestures when there are confusing or noisy backgrounds. Additionally, the process of how the recognition and movement results from the model gets turned into logic that controls the map is a pretty difficult task and requires a lot of fine-tuning with different parameters. Plenty of future possibilities exist to fine-tune these capabilities, expand into new functionalities, and make the system more robust.

Technology Stack

This project utilized Google’s MediaPipe library to perform on-device machine learning (ML). The user's camera gets fed into the model which then returns various outputs that can be continuously interpreted and fed into the logic that controls the map. This was implemented using TypeScript/JavaScript, along with HTML and CSS for adding structure and styling. The mapping component is done with Leaflet.

Together, the core ML and mapping application is hosted via a CodePen, making it easy to interact with the code and allow others to use and customize the tool. After getting the core tool created, the CodePen was embedded in a simple React application to make it look a little nicer and add some instructional components. The React application makes use of Material-UI for the interface design and is hosted via GitHub Pages.

Give it a Try

Click here to try the active website.

Project Advising

Dr. Eric Shook, Director, GeoCommons

Page updated

Report abuse