Snap and track: New AI instantly analyzes a meal’s nutritional content through image recognition

As concerns about weight management and diabetes continue to rise, an AI system developed by NYU Tandon School of Engineering researchers introduces a promising new AI-based tool in the fight against diet-related health conditions.

Users snap a photo of their meals and are instantly presented with an estimation of its calorie count, fat content, and nutritional value, which the researchers say means “no more food diaries or guesswork.”

This futuristic scenario is now much closer to reality, say the authors who pitched the concept at the sixth International Conference on Mobile Computing and Sustainable Informatics hosted by the Institute of Electrical and Electronics Engineers.

The tool uses advanced deep-learning algorithms to recognize food items in images and calculate their nutritional content, including calories, protein, carbohydrates, and fat.

“Traditional methods of tracking food intake rely heavily on self-reporting, which is notoriously unreliable,” says Prabodh Panindre, the paper’s lead author and associate research professor at NYU Tandon School of Engineering’s Department of Mechanical Engineering.

“Our system removes human error from the equation.”

The system has been deployed as a web application that works on mobile devices and the researchers describe their current system as a proof-of-concept that can be refined and expanded for broader healthcare applications very soon.

Beginning with occupational nutrition

For over a decade, NYU’s Fire Research Group, which includes Panindre and co-author Sunil Kumar, has studied critical firefighter health and operational challenges.

The tool uses advanced deep-learning algorithms to recognize food items in images and calculate their nutritional content, including calories, protein, carbohydrates, and fat.

Several research studies show that 73–88% of career and 76–87% of volunteer firefighters are overweight or obese, facing increased cardiovascular and other health risks that threaten operational readiness.

These findings directly motivated the development of their AI-powered food-tracking system.

Despite the apparent simplicity of the concept, the researchers highlight developing reliable food recognition AI has stumped developers for years. Previous attempts struggled with three fundamental challenges that the NYU Tandon team appears to have overcome.

“The sheer visual diversity of food is staggering,” says Kumar, a professor of mechanical engineering at NYU Abu Dhabi and global network professor of mechanical engineering at NYU Tandon.

“Unlike manufactured objects with standardized appearances, the same dish can look dramatically different based on who prepared it. A burger from one restaurant bears little resemblance to one from another place, and homemade versions add another layer of complexity.”

Harnessing powerful image recognition

Earlier systems also faltered when estimating portion sizes — a crucial factor in nutritional calculations. The NYU team's advance is their volumetric computation function, which uses advanced image processing to measure the exact area each food occupies on a plate.

“The new system correlates the area occupied by each food item with density and macronutrient data to convert 2D images into nutritional assessments,” the researchers explain.

“This integration of volumetric computations with the AI model enables precise analysis without manual input, solving a longstanding challenge in automated dietary tracking.”

The third major hurdle has been computational efficiency.

“Previous models required too much processing power to be practical for real-time use, often necessitating cloud processing that introduced delays and privacy concerns,” highlight the researchers.

The team used a powerful image-recognition technology called YOLOv8 with ONNX Runtime — a tool that helps AI programs run more efficiently — to build a food-identification program that runs on a website instead of as a downloadable app. People can visit it using their phone’s web browser to analyze meals and track their diet.

Optimizing for complex dishes and cuisines

When tested on a pizza slice, the system calculated 317 calories, 10 grams of protein, 40 grams of carbohydrates, and 13 grams of fat — nutritional values that closely matched reference standards.

It performed similarly well when analyzing more complex dishes such as idli sambhar, a South Indian specialty featuring steamed rice cakes with lentil stew, for which it calculated 221 calories, 7 grams of protein, 46 grams of carbohydrates and just 1 gram of fat.

“One of our goals was to ensure the system works across diverse cuisines and food presentations,” says Panindre.

“We wanted it to be as accurate with a hot dog — 280 calories according to our system — as it is with baklava, a Middle Eastern pastry that our system identifies as having 310 calories and 18 grams of fat.”

The researchers solved data challenges by combining similar food categories, removing food types with too few examples, and giving extra emphasis to certain foods during training. These techniques helped refine their training dataset from countless initial images to a more balanced set of 95,000 instances across 214 food categories.

The technical performance metrics are significant: the system achieved a mean Average Precision (mAP) score of 0.7941 at an Intersection over Union (IoU) threshold of 0.5. For non-specialists, this means the AI can accurately locate and identify food items approximately 80% of the time, even when they overlap or are partially obscured.