AI vehicle color analysis turbocharges automatic number plate recognition (ANPR)

Being able to recognize numbers and characters was an early big win for machine learning. And one of the first practical lessons for AI students is often training a simple algorithm to classify digits based on the classic EMNIST dataset of 28×28 pixel images of handwritten numbers. Today, optical character recognition (OCR) is big business with a market size in excess of US $11 billion, according to analyst firm Global Data. And high-achieving AI systems include automatic number plate recognition (ANPR) technology, which has become commonplace on road networks worldwide.

In the UK alone, based on Police data, national ANPR cameras generate in the region of 60 million vehicle number plate read records every day. And, while the importance of ANPR data in crime prevention and law enforcement shouldn’t be overlooked, being able to recognize cars, trucks, and other vehicles has a wide range of applications.

Number plate recognition systems are used by firms to log facilities access and to enable ticketless parking. ANPR data also supports intelligent traffic management systems, helping national highways authorities to accurately calculate journey times and monitor road networks in greater detail.

In Bristol, UK – where TechHQ has its headquarters – 49 ANPR cameras have been installed to deliver a clean air zone (CAZ) in the city center. According to supplier SEA – which won the contract to provide ANPR technology for the CAZ – individual cameras can monitor up to three lanes of traffic. And the computer vision system can retrieve vehicle make, model, gross weight, engine type, Euro rating and CO2 emission band from DVLA databases based on OCR information.

Adding vehicle color recognition to the ANPR mix

But that’s by no means all of the vehicle identifiers that can be gathered using roadside cameras and other image sources, as research published in the Frontiers of Computer Science shows (alternative link). The team, based in China, highlights the opportunity for systems to perform vehicle color recognition (VCR), and there are some good reasons why adding VCR to the ANPR mix is helpful.

Not all drivers are careful to keep their license plates clean, and some may cover, remove, or even – in the case of bad actors – use forged plates, all of which will disrupt the performance of conventional ANPR systems.

The idea of using vehicle color to augment other characteristics acquired using computer vision dates back to an earlier study performed in 2015. In this preliminary work, researchers in Taiwan used decision tree classification to separate road traffic images into a handful of categories based on vehicle color.

They say there are over 250 #AI (ANPR) and red-light violation detection traffic cameras operational in Namma Bengaluru.

This is your next level @blrcitytraffic.

No #helmet No 🟢 🚥! 💡👏

@btppubliceye @peakbengaluru #WearHelmet #RoadSafetypic.twitter.com/loVPGXMDLT

— TORquED .eth .lens (@DigiMaverick) June 28, 2023

Twitter video appears to show that ANPR cameras can identify helmetless riders at road junctions.

Physical characteristics such as the chemistry, color, number, and layer sequence of paint can all be used to help differentiate one vehicle from another. Since 1997, the FBI and Royal Canadian Mounted Police have teamed up to create a database of more than 9000 paint records dubbed Paint Data Query (PDQ). And the body of information can be used to identify most new vehicles sold in North America after 1973, including models marketed by foreign manufacturers as well as domestic OEMs.

VCR systems can’t match the capabilities of having a physical paint sample to hand, but – as the latest studies show – the number of vehicle colors that can be reliably detected using computer vision is increasing. For example, using a VCR method based on Smooth Modulation Neural Network with Multi-Scale Feature Fusion (SMNN-MSFF), AI experts have almost doubled the number of color categories that can be recognized remotely.

This latest VCR dataset consists of 10,091 vehicle images extracted from 100 hours of urban road surveillance video. And given the vast amount of traffic footage that is captured on highways globally, this work should be considered as a proof-of-concept rather than the end of the road. The prospects for generating more training data are surely huge.

Based on the results presented in the 2023 paper, VCR is already capable of distinguishing between different shades of vehicle colors – for example, by identifying champagne, yellow, lemon-yellow, and earthy-yellow as distinct categories. And combining VCR with ANPR data adds up to more robust vehicle identification.

As the researchers point out, the fact that vehicles typically only have a single dominant color, which isn’t easily damaged in its entirety or changed– compared with swapping a license plate – makes auxiliary VCR data worth having. And for drivers that don’t want to stand out? White was the most popular vehicle color in the group’s VCR training footage – representing more than a third of the examples in the data set. Also, considering other geographies, over a quarter of cars in the US are painted white, according to iSeeCars.com – a search engine for cars.

Some commercial ANPR systems are already capable of performing VCR, although possibly not with the same fidelity as the neural network used by the research team profiled in this article. Online demos of ANPR systems – which can be viewed on YouTube – show evidence of low-level vehicle color classification, but it’s clear that the software can’t distinguish between different shades. For example, when a dark blue car drives by it’s labeled as simply being blue, and similarly for a different model painted this time in a lighter hue.

DIY ANPR

To understand what’s involved in automatically interpreting number plate information using OCR, it’s instructive to check out some of the many coding tutorials online. Rather than start from scratch, it’s possible to use custom-trained object detection models such as YOLO – included as part of NVIDIA’s TAO Toolkit, designed to speed up the creation of computer vision AI applications – to extract license plate images from footage of vehicles.

Having narrowed down the region(s) of interest, developers can then deploy OCR routines to read out the license plate numbering and lettering and send API calls to an ANPR database to retrieve the corresponding vehicle records. Note that users will need to ensure that they are using this data lawfully. Organizations such as the UK-based Information Commissioner’s Office has guidance on data protection implications of collecting ANPR information.

GDPR and other data privacy laws may apply to people, not vehicles, but that line is easily crossed if ANPR information can be linked to the driver and matched to other PII. And organizations such as the Electronic Frontier Foundation caution that automated license plate reader data can ‘paint an intimate portrait of a driver’s life’.

If playing around with Python libraries and having fun with a Raspberry Pi is starting to look like surveillance, then it’s time to call quits on the coding lesson.

LPR on the Edge

Video analytics firm Azena – a Bosch-funded German tech start-up based in Munich, with international offices in Eindhoven and Pittsburgh – offers edge-based license plate recognition (LPR). And commercial operators wanting to reduce their exposure to data concerns may want to consider self-contained edge-based solutions that minimize information being sent to the cloud.

Azena collaborated with Czech-based FF Group – which offers a range of traffic monitoring capabilities, including a Smart City Starter Kit – on the setup. And the system now features make, model, and color recognition (MMCR) technology. FF Group first started working on make and model recognition, powered by Nvidia hardware, back in 2017. Sampling 500k cars, the algorithm was accurate in 92% of cases, and that was just in early testing.

Products can perform vehicle count, direction detection, LPR, and MMCR of traffic traveling at speeds of up to 120 km/h. The capabilities of modern video analytics are impressive. But at the same time, if you’re looking for somewhere to hide, maybe don’t drive there.