In short
Calorie counting from a photo happens in three steps: food recognition, portion estimation and database lookup. For clearly visible, separated dishes the estimate is usually workable; for food buried in sauce, stacked, or inside closed containers the error grows sharply. That is why serious apps never save the result directly — they open a correction screen first.
Three problems, not one
Turning a meal photo into a calorie figure means solving three separate problems, and each one carries its own margin of error. Thinking about them separately is the most practical way to judge how much to trust the number you get.
The steps run in sequence, so an error in the first one propagates. If the model reads bulgur as rice, the calorie figure will be wrong even when the portion estimate was perfect.
- Food recognition: working out what is on the plate. On mixed plates each component has to be found separately.
- Portion estimation: converting the area in the image into real weight, using plate diameter, cutlery and bowl depth as references.
- Database lookup: multiplying the recognised food and estimated weight by the values in a food composition table.
The hard part is portion, not recognition
Vision models identify dish types remarkably well today. The genuinely hard problem is going from a two-dimensional photo to a three-dimensional volume: two bowls of soup shot from the same angle can be 200 ml and 350 ml and look almost identical in the picture.
That is why portion estimation leans on reference objects. A standard dinner plate is roughly 26-28 cm, a teaspoon is 5 ml, a tablespoon is 15 ml. The model finds these known dimensions in the frame and builds a scale from them. With no reference in shot — a tight close-up where the plate rim is cropped out — the estimate degrades noticeably.
- Get the whole plate in frame; shoot from above rather than close up.
- On mixed plates, keep the components separated where you can.
- Declare sauces and oil separately — invisible olive oil can double the calories in a salad.
Fat is the calorie you cannot see
The single largest source of error in photo estimation is added fat. One tablespoon of olive oil is about 120 calories and is usually invisible in the picture. The same grilled chicken breast carries 240 extra calories once it has been pan-fried in two tablespoons of oil.
This is not a flaw in the method so much as a limit of physics: no model can know what the camera did not see. Eating out makes the uncertainty worse, because you do not know how much fat went into the pan either.
What the margin of error actually means
The goal of food tracking is not perfect accuracy — it is a consistent measure. On a 2,000-calorie daily target, being 100 calories out on one meal is a 5% deviation and it will not distort the weekly trend. Systematic error will: undercounting fat every single day compounds into thousands of calories a month.
The practical consequence is that you should watch the weekly average produced by one consistent method, not the individual number. If the scale is not moving, question your logging habit before you lower the target — there is almost always one item you are not counting.
Where it works and where it does not
The value of the method comes less from its accuracy than from its sustainability. In a classic calorie app, logging one meal takes minutes and most people stop within a fortnight. A log that takes three seconds and carries a 10% margin beats a perfect log that never gets kept.
| Situation | Estimate quality |
|---|---|
| Single dish, separated, clearly visible | Good |
| Mixed plate, components distinguishable | Fair to good |
| Dish covered in sauce or cream | Poor |
| Closed container, wrap, filled pastry | Very poor |
| Packaged product with a readable barcode | Exact |
Frequently asked questions
Is weighing food more accurate than photographing it?
Yes — a weighed entry is always more accurate. But weighing adds a few minutes of work every day, and most people abandon it after a few weeks. The most accurate method is the one you keep doing; weighing at home and photographing when you eat out is a sensible balance for most people.
Why do two photos of the same meal give different numbers?
Because the angle, the lighting and the reference objects in frame changed, so the portion estimate changed with them. For more consistent results, always shoot the whole plate from above and keep a similar distance.
Does it work offline?
Analysis needs a connection, because the image is processed on a server. Meals you have already logged and your past data remain viewable offline.
Sources
- 1.
Thames Q, Karpur A, Norris W, Xia F, Panait L, Weyand T, Sim J Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
- 2.
Lu Y, Stathopoulou T, Vasiloglou MF, Pinault LF, Kiley C, Spanakis EK, Mougiakakou S goFOODTM: An Artificial Intelligence System for Dietary Assessment of Food Images. Sensors, 20(15):4283, 2020.
- 3.
Vasiloglou MF, Mougiakakou S, Aubry E, et al. A Comparative Study on Carbohydrate Estimation: GoCARB vs. Dietitians. Nutrients, 10(6):741, 2018.