5 Things We Learned About AI Trainers

When explaining Map AI, there's an analogy we like to use: dashcams are the eyes of the Hivemapper network, and Map AI is the brain.

You could also flip the analogy, and describe the human brain as an incredibly powerful computer. Our brains have 100 billion neurons making 100 trillion connections, despite running on just 20 watts of power, a fraction of what a laptop charger draws. 

To help us perform complex tasks such as driving a car safely down the road, our brains need years of education and training. That is how we form all the neural pathways that help us make snap judgments based on the data we take in through our eyes and ears.

So naturally, our Map AI needs a fair bit of training too. 

Map AI needs to be taught what a stop sign looks like. It needs to be taught how to use 2D images to interpret the position of that stop sign in 3D space. It needs to be taught to look at different pictures of that stop sign and confirm that yes, they’re actually the same sign. 

This is the purpose of AI Trainers. 

When we launched the first iteration of AI Trainers in April, we envisioned them as simple, bite-sized tasks that would allow anyone in the world to share their brainpower with the Map AI and receive rewards. In this way, the Map AI would learn to efficiently build a map of the world for humans to use.

In the first three months since launch, trusted reviewers have done more than 7 million reviews, committing more than 10,000 hours of human brainpower to train the Map AI.

 We’ve learned all kinds of valuable lessons that will inform the design of the AI Trainers moving forward. Here are some of those lessons.

We quickly learned  people were doing AI Trainers for different reasons. 

Some people see AI Trainers as a microtasking opportunity, similar to Amazon Mechanical Turk or Remotasks. The microtaskers are often newcomers to the project. They do tasks for long stretches of time, and are focused on being proficient and accurate to maximize rewards. Whenever an AI Trainer runs out of tasks or starts lagging, the microtaskers let us know within minutes - underscoring the importance of a stable, reliable pipeline.

Because of how rewards are currently calculated, microtaskers often gravitate toward tasks that can be completed quickly. This makes clear that we need to set AI Trainer rewards based on the time requirements and complexity of the task. We plan to do that soon.

Another cluster is made up of existing community members - often, dashcam contributors - who just want to help the project be successful.

Dashcam contributors are willing to fit in a few minutes of AI Training during free moments of their day, as long as the games are easy enough. These users have taught us the importance of optimizing games for mobile and reducing friction so that people can quickly log on and do a few tasks while riding a train or sitting in the waiting room of a doctor’s office.

One thing these groups share is a desire to make the AI Trainer experience more gamelike. 

For example, we keep hearing how much people enjoyed the Ultimate AI Trainer contest. The contest offered special HONEY prizes, but several people told us they participated to win beta access and the “Top Trainer” badges given to the top 50 finishers.

You can expect to see the AI Trainer experience become more fun and satisfying over time, with elements such as leaderboards, personal statistics and in-app badges.

Quality is critical when building a map, and the decentralized, permissionless design of the Hivemapper network makes ensuring quality more difficult. It is critical to have strong incentives and guardrails, because a few bad actors can have a huge negative impact. 

Almost as soon as AI Trainers went live, spammers started targeting them with bot accounts. We knew it would happen (it’s unavoidable with HONEY rewards on the line) but we were still a bit surprised by the speed, scale and sophistication of these efforts. 

Although we feel good about the job we’ve done of identifying spam and ensuring that spammers receive little or no rewards, this cat-and-mouse game has taken a significant amount of effort. And at times, spikes in spam had a severe negative impact on the quality of data from AI Trainers.

For this reason, we plan to invest heavily in additional anti-spam protections, in our reputation system, and in a leveling system that will create strong incentives for sustained quality over time. 

We envision a system where the best reviewers level up, as they do in crowdsourced mapping platforms such as Waze. Once they have a strong track record of performance on the simplest tasks that have the least risk for the integrity of the map, they will earn access to more challenging, more sensitive tasks that also yield larger rewards.

As we released AI Trainers into the wild, we learned that some tasks needed a different form factor from what we had originally imagined.

In some AI Trainers, humans sort images based on whether the object was correctly or incorrectly classified by the Map AI. We discovered it was faster and more stimulating to do multiple objects at once, so we switched those games over to a “mosaic” view with 9 or even 18 objects per task.

We also learned that some tasks were better suited to mobile, and others to desktop. 

If you have a few free minutes, AI Trainers are a great way to pass the time – if the game is easy enough to play on a smaller screen. Some tasks are a better fit for a laptop or desktop computer, where users can use hotkeys and benefit from the larger screen when looking at a map. We’ve received feedback that many professional taskers prefer to work on a computer.

Although we will design tasks to be compatible with multiple form factors whenever possible, it’s likely that the simpler, lower-level tasks will be designed primarily for mobile, while the more advanced tasks will be designed primarily for desktop.

The rapid adoption of AI Trainers reinforced the importance of keeping it simple.

We want AI Trainer participants to be able to complete most tasks on their first try, without any special training. We originally planned to release training modules for each game, but we quickly realized that we didn’t really need them – as long as we designed the tasks and interfaces well.

This may sound obvious, but it’s very different from how maps are built today.

Crowdsourced mapping projects such as OpenStreetMap and Waze rely on thousands of humans to painstakingly edit and annotate maps, as shown below. The tools they use are powerful but complicated, requiring extensive learning that creates a barrier for new users. 

By entrusting much of this work to AI, and by breaking human tasks into small and simple pieces, we will enable a significantly larger pool of people to participate in mapmaking.

This may seem like it’s in conflict with the previous lesson, but it’s actually not. 

Tasks can be simple to complete, and yet deceptively challenging. Sort of like the motto of the board game Othello: “A minute to learn, a lifetime to master.” Within the Hivemapper team, everyone instantly loved one specific game called Positional Accuracy. (Note: this game is currently limited to a trusted subset of users. In the future, a version of this game could be unlocked as users level up.) 

In this game, the user needs to sanity-check the location that Map AI generated for a real-world object by looking at aerial imagery. Sometimes, it’s easy. The traffic light is standing right there, as the Map AI predicted. But sometimes you can’t see the object and you need to follow the shadows cast by poles, or look for other reference points you can see in the dashcam image. (Where’s that driveway I see in the picture?) The most exciting situation is capturing objects on brand-new roads that are still under construction; it reminds us of why it’s so important for Hivemapper to exist.

Games like this are more time consuming than some other games. They’re also really fun.

We’ve drawn inspiration from the location guessing game GeoGuessr, which now claims 40 million users. GeoGuessr has become so popular that hundreds of thousands of people follow top players such as Rainbolt, who livestreams his play sessions on Twitch and accepts challenges to find real-world locations from a single photograph on social media. 

If we do a good job, training the Map AI can be just as fun.