GIFSQ ID: FTN-0093
Big Data Analytics in Food Safety and Quality
Unit 1: What Big Data Means in Food
Note
Unit 1: What Big Data Means in Food
A question to start with
A bakery makes the same loaf every day. One morning the loaves come out pale and flat. The flour delivery note looks normal. The oven temperature log looks normal. The mixing time looks normal. Nothing on its own explains the failure. Then the quality officer puts three records side by side: the flour moisture readings from the last ten deliveries, the oven temperature readings taken every minute through the bake, and the dough mixing times from the last month. Seen together, a pattern appears. The flour moisture has been creeping up a little with each delivery. The oven temperature dips a little whenever the dough is wetter. The mixing time has stayed the same, so the wetter dough is undermixed. No single reading was wrong. The story was in the combination.
That is what big data means in food. It is not one large file. It is the habit of joining many small records together and reading the pattern they make. A food business already collects records every day: delivery notes, laboratory results, temperature logs, cleaning checklists, customer comments, sales records, and storage readings. Each record on its own answers a small question. Joined together, they can answer bigger questions: why did this batch fail, which supplier is drifting, which line needs attention first, and what will probably go wrong next week if nothing changes.
This unit explains the idea in plain words. It sets out what counts as data, what makes data "big", and how food safety and quality work changes when the records are joined up. Later units go into the detail: where the records come from, how to clean them, how to find patterns, how to predict problems, and how to turn all of it into decisions people can act on.
Questions to think about
- A bakery has ten years of paper temperature logs in a cupboard. Is that big data, or is it only a large pile of paper?
- If a sensor records oven temperature every minute, what question can it answer that a single reading at the end of the bake cannot?
- A laboratory result says a batch of milk is safe. A delivery record says the milk arrived warm. Which record should the quality officer trust, and why might the answer be both?
Think about the bakery story before you read on
The pile of paper is not big data yet. Paper records that nobody can search, sort, or join together cannot answer new questions quickly. Big data starts when records are stored in a form a computer can read, search, and join: a table, a database, a file with the same columns every time. The minute-by-minute sensor can show a dip in the middle of the bake that a single end reading would miss. The lab result and the warm delivery can both be true at once: the milk may have passed the test, and the warm journey may still have shortened its keeping life. The joined story matters more than either record alone.
Data, information, and knowledge
Data are recorded facts: a temperature reading, a weight, a date, a batch number, a laboratory result, a tick on a checklist. A single reading is data. Information is data that has been organised so it answers a question: the average oven temperature during a bake, the highest moisture reading in a delivery, the number of batches that failed last month. Knowledge is what a person does with information: the decision to change the mixing time, to call a supplier, or to hold a batch.
The three levels matter because food businesses often stop at the first one. They collect readings and file them. The readings sit in notebooks, inboxes, and separate computer files. Nobody turns them into information, so nobody gains the knowledge. Big data work is the bridge between the three levels: collect the facts, organise them so questions can be asked, and use the answers to make better decisions.
A useful test is to ask, for any record: what question does this answer, and who acts on the answer? A temperature log that nobody reads answers no question. A laboratory result that reaches the production manager a week late answers the question too late. Data earns its keep only when it reaches the right person in time to act.
Quick self-check
Question: A cold store records temperature every hour and files the readings. Nobody ever looks at them. Is that data, information, or knowledge?
It is data only. The readings are facts, but they have not been organised into an answer, and nobody has used them to decide anything. The moment someone plots the readings and sees a warm spell each afternoon, it becomes information. The moment someone fixes the door seal that causes the warm spell, it becomes knowledge in action.
Question: What is the difference between a fact and an answer?
A fact is one recorded value, such as 4 degrees in a cold store at noon. An answer puts facts together: the cold store averaged 6 degrees this week, which is warmer than the limit the business set. Facts are the raw material. Answers are facts organised around a question.
What makes data "big"
People use the word big in three ways, and all three show up in food work. The first is volume: there is a lot of it. A line sensor that records every minute makes more than a thousand readings a day. A laboratory that tests every batch adds hundreds of results a month. A chain of shops that records every sale adds thousands of lines a day. Volume alone is not the point, but it changes the work: paper and memory cannot hold it, and a computer must.
The second is speed: data arrives fast, and sometimes it must be used fast. A temperature sensor in a cold store sends readings all day. A metal detector on a packing line signals the moment it finds something. A complaint comes in while the batch is still on the shelf. When data arrives this fast, waiting for a weekly report is too slow. The system must watch as the data arrives and raise the alarm in time.
The third is variety: data comes in many shapes. Numbers from instruments, words from customer comments, photographs from a packing line, checklists ticked on a phone, laboratory reports as tables, and delivery notes as text. Joining numbers with words and pictures is harder than joining one table to another, and it is also where some of the best answers hide. A rise in customer comments about a strange taste may line up with a small change in a laboratory result that nobody flagged on its own.
There is a fourth idea that matters as much as the first three: truthfulness. Data can be wrong. A sensor can drift. A person can type a batch number wrong. A laboratory result can be recorded against the wrong sample. A clock can be set wrong, so the time on every reading is off by an hour. Big data that is wrong at scale misleads at scale. That is why cleaning and checking data, the subject of Unit 3, is not a side task. It is the foundation everything else stands on.
The diagram shows the four marks in order. The steps point to the diagram: a food data project that forgets any one of the four will struggle, and the fourth mark, truthfulness, decides whether the other three are worth having.
Quick self-check
Question: A packing line camera takes a photo of every pack. Which of the four marks does that show?
Variety and volume. Photos are a different shape of data from numbers, so that is variety. One photo per pack, all day, is a large number of records, so that is volume.
Question: Why is a wrong sensor reading worse when there are thousands of readings?
Because the wrong readings pile up and shape the average, the chart, and the decision. One wrong reading in a notebook is easy to spot. A thousand wrong readings from a drifting sensor look like a real pattern, and people may act on a pattern that never happened.
Structured, semi-structured, and unstructured records
Not all food records look the same. Structured data sits in neat rows and columns, with the same fields every time: a laboratory table with batch number, test name, result, unit, and date; a temperature table with time, sensor name, and reading. This is the easiest data to search, sort, and join, and most food safety work starts here.
Semi-structured data has some organisation but not a fixed table: a delivery note with a heading, a supplier name, and a list of items; a checklist on a phone with tick boxes and a comment field; a laboratory report with a table plus a free-text note at the bottom. The organised parts can be read by a computer, and the free-text parts need a person or a text tool to read them.
Unstructured data has no fixed shape: a customer comment, a photograph of a damaged pack, a voice note from a driver, a video of a line stoppage. These records carry real information, and they are the hardest to join with tables. A comment that says "the milk tasted sour" is useful only if someone links it to the batch number, the date, and the shop. That linking work is a large part of real food data projects.
A good habit is to ask, for each record: what is the shape, and what is the link? The link is usually a batch number, a date and time, a supplier name, or a product code. Records without a link are hard to join, however good they are on their own. A photograph of a swollen pack with no date and no batch number is a warning with no address. The same photograph with a batch number and a time becomes evidence that can be traced and acted on.
Quick self-check
Question: A laboratory table lists batch, test, result, and date. A customer email says "the biscuits tasted stale". Which is structured, and which is unstructured?
The laboratory table is structured: same fields, same order, every row. The email is unstructured: free words with no fixed fields. The email becomes far more useful when someone adds the batch number and the purchase date to it.
Question: Why is a batch number so important in food data?
Because it is the link that joins records from different places: the delivery, the production run, the laboratory test, the storage log, and the customer comment. Without a shared link, each record stays separate, and the joined story cannot be built.
How big data changes safety and quality work
Before joined-up records, quality work was mostly reactive and sample-based. A batch was tested, a result came back, and a decision was made about that batch. That work still matters, and testing will never go away. Big data adds three new abilities on top of it.
First, it widens the view. Instead of judging one batch, the quality team can see all batches from one supplier over a season, or all lines in a factory over a month, and spot which one is drifting. A drift that is invisible in one result shows up clearly in a line of fifty results.
Second, it shortens the time between a problem and the response. A sensor that watches continuously can raise an alarm while the batch is still on the line, not after the laboratory result comes back the next day. A dashboard that updates through the day can show a rising temperature in a cold store before the food inside is harmed.
Third, it connects causes to effects. A quality problem often has more than one cause: wetter flour plus a cooler oven plus a shorter mix. Joined records let the team test each possible cause against the data, instead of guessing or blaming the most recent change. That is the difference between fixing a symptom and fixing a cause.
None of this replaces the analyst, the production manager, or the food safety plan. Data suggests, people decide. A chart cannot taste a product, smell a spoiled sample, or walk a line and see a blocked drain. What data does is give those people a wider view, a faster warning, and a clearer test of their ideas.
Quick self-check
Question: Why does seeing fifty results in a line help more than seeing one result?
One result can be high or low by chance. Fifty results in a line show the direction: steady, rising, falling, or swinging wildly. A slow rise over weeks is a drift that needs fixing, and it is invisible in any single result.
Question: A dashboard shows a cold store warming up. Who should act, and what should they do first?
The person on duty for that store should act first, by checking the store in person: is the door shut, is the cooling unit running, is the sensor itself at fault? The dashboard gives the warning. The person confirms the cause before moving food or changing settings.
A simple picture of the whole system
A food data system has five stages, and every later unit in this subject covers one or more of them. First, records are collected: sensors, laboratory instruments, checklists, delivery notes, and sales records. Second, records are stored in one place where they can be found, with the same names for the same things. Third, records are cleaned and organised: wrong values fixed, missing values handled, units made consistent. Fourth, records are analysed: counted, plotted, compared, and modelled. Fifth, the results are shown and used: a chart, an alert, a report, a decision.
The stages feed each other. A problem found in analysis often points back to a problem in collection: a sensor that needs calibrating, a checklist that needs a clearer question. A decision made in stage five often creates a new collection need in stage one: the team starts recording a value they never recorded before. The system is a loop, not a straight line.
Small businesses can run this loop with simple tools: a shared spreadsheet program, a folder of tables with the same columns, a whiteboard chart updated each day. Large businesses use databases and dashboards, but the logic is the same. The size of the tool is less important than the discipline: same names, same units, same batch links, and a regular habit of looking at the joined picture.
The diagram shows the five stages in order. The steps point to the diagram: a weakness in an early stage shows up as a wrong answer in a later stage, so fixing the early stage is often the fastest way to improve the answer.
Quick self-check
Question: A team buys dashboard software before it fixes its batch numbers. What will probably go wrong?
The dashboard will show pretty charts built on records that cannot be joined. Batches will be spelled three different ways, so the same batch looks like three batches. The team should fix names and links first, then buy the display tool.
Question: Why is the system a loop and not a straight line?
Because each stage teaches the team about the others. Analysis shows which collection steps are weak. Decisions create new questions that need new records. The loop keeps improving, while a straight line would stop at the first report.
What big data cannot do
Big data has limits, and naming them early saves disappointment. First, data cannot fix a bad process on its own. If a recipe is wrong, or a cold store door does not seal, no amount of charting will bake the bread or chill the milk. Data shows the problem clearly; people and engineering fix it.
Second, data cannot prove a cause by itself. Two things can rise together without one causing the other. Complaints about a product may rise in the same month that a new supplier starts, but the cause may be a hot season, a longer storage time, or a change in packaging. Proving a cause needs a careful test: change one thing, hold the rest steady, and watch what happens. Unit 4 explains how to compare groups fairly, and Unit 5 explains how to test a prediction before trusting it.
Third, data cannot replace tasting, smelling, and looking. A laboratory result and a sensor reading are proxies: they stand in for the food, but they are not the food. A trained panel that tastes and smells a sample can catch faults that no instrument measures. The best systems pair instrument data with human sense checks.
Fourth, more data is not always better data. A thousand readings from a broken sensor are worse than ten readings from a good one. A long customer comment file with no batch numbers is harder to use than a short table with clear links. The question is never "how much data do we have?" It is "which question can this data answer, and can we trust the answer?"
Quick self-check
Question: Complaints rise in the same month a new flour supplier starts. Does that prove the supplier caused the complaints?
No. The two events happened together, but the cause could be something else that changed in the same month: the season, the storage time, the packaging, or the recipe. A fair test would compare batches that differ only in the supplier, with everything else held steady.
Question: A team records oven temperature every second instead of every minute. Will the bread improve?
Not by itself. More readings give a finer picture of the oven, but the bread improves only when someone uses that picture to change the oven setting, the recipe, or the process. Recording is not fixing.
People, roles, and trust
A food data system is a team effort. The production team collects most of the records: temperatures, times, weights, and checklists. The laboratory team adds test results and checks that instruments are calibrated. The quality team joins the records, checks them, and turns them into charts and alerts. The managers decide what to change and make sure the change happens. The suppliers and customers add records from outside the factory: delivery notes, certificates, and comments.
Trust is the thread that holds the team together. If the production team believes the data will be used to blame them, they will stop recording problems. If the laboratory team is rushed, results will be typed in a hurry and mistakes will creep in. If managers ignore the charts, the team will stop updating them. The rules that build trust are simple: record honestly, check kindly, share the charts openly, and use the data to fix processes, not to hunt for people to blame.
Privacy and care matter too. Customer comments may carry names and contact details. Supplier records may carry prices and contract terms. Staff records may carry shift patterns. The data system should keep personal and commercial details separate from the quality records where possible, limit who can see them, and keep them only as long as they are needed.
Quick self-check
Question: Why would a worker stop writing down a problem if the data is used to blame people?
Because honest records would then bring punishment, so the safe choice is to hide the problem. The data gets cleaner on paper and worse in reality, and the real problem keeps hurting the food. Data used to fix processes invites honesty; data used to blame invites silence.
Try it yourself
Pick one food you know well: bread, milk, yoghurt, jam, biscuits, or juice. On one page, list every record that already exists for that food in a small business: delivery notes, temperatures, times, weights, laboratory tests, checklists, customer comments, and sales notes. For each record, write three things: who makes it, where it is kept now, and what question it could answer if it were joined with one other record. Then circle the three joins that would answer the most useful questions. Those three joins are the start of a big data project for that food, and they need no new equipment at all.
Key ideas to carry forward
- Big data in food means joining many small records together and reading the pattern they make.
- Data are recorded facts; information is data organised to answer a question; knowledge is the decision made from the answer.
- Big data is marked by volume, speed, variety, and truthfulness; the last one decides whether the rest are worth having.
- Structured data sits in neat tables; semi-structured data mixes tables with free text; unstructured data is words, pictures, and voice, and it needs a link to be useful.
- A batch number, a date and time, or a product code is the link that joins records from different places.
- A food data system runs in five stages: collect, store, clean and organise, analyse, and show and use, and the stages form a loop.
- Data widens the view, shortens the time to respond, and connects causes to effects, but it cannot fix a process, prove a cause alone, or replace tasting and looking.
- Trust decides whether the system works: honest records, shared charts, and data used to fix processes, not to blame people.
Glossary
Batch
A quantity of food made or handled under the same conditions at one time, given one number so it can be traced. Food tech example: one morning of biscuit production is a batch, and every test and temperature reading for that morning carries the same batch number.
Batch number
The code that identifies one batch and links all its records together. Food tech example: batch number 214 links the flour delivery, the mixing time, the oven log, and the laboratory result for one run of bread.
Big data
Records that are large in amount, fast in arrival, varied in shape, and in need of checking, stored so a computer can search and join them. Food tech example: minute-by-minute oven readings plus laboratory results plus customer comments, joined by batch number, form big data for a bakery.
Dashboard
A screen that shows the most important numbers and charts together, updated often, so a team can see the state of a process at a glance. Food tech example: a dashboard shows cold store temperatures, open alerts, and today's test results on one screen.
Data
Recorded facts such as a temperature, a weight, a date, or a test result. Food tech example: a reading of 4 degrees in a cold store at noon is one piece of data.
Database
An organised store of tables where records are kept with fixed names and links, so they can be searched and joined. Food tech example: a laboratory database holds every test result by batch number, test name, and date.
Dataset
A set of related records kept together for one purpose. Food tech example: a dataset of flour moisture results for one season lets the team compare deliveries across months.
Information
Data that has been organised so it answers a question. Food tech example: the average moisture of flour deliveries this month is information made from many single readings.
Knowledge
What a person does with information: a decision or an action based on the answer. Food tech example: changing the mixing time after seeing that wetter flour needs a longer mix is knowledge in action.
Laboratory information management system
A computer system that tracks samples from arrival to report, so every result can be traced back to its sample and method. Food tech example: the system shows which sample a protein result belongs to and who tested it.
Record
One written or stored entry of a fact, such as one line in a log or one row in a table. Food tech example: one line in the oven log is a record of the temperature at one time.
Sensor
A device that measures a physical value, such as temperature or humidity, and records or sends the reading. Food tech example: a sensor in a cold store measures the air temperature every hour.
Structured data
Data in neat rows and columns with the same fields every time. Food tech example: a laboratory table with batch number, test name, result, and date is structured data.
Traceability
The ability to follow a food, a batch, or a result back through every step it passed through. Food tech example: a batch number lets a jam maker trace a jar back to the fruit delivery and the cooking run.
Unstructured data
Data with no fixed shape, such as free words, photographs, or voice notes. Food tech example: a customer email saying the biscuits tasted stale is unstructured data until a batch number is added to it.
Variety
The range of shapes data comes in: numbers, words, pictures, checklists, and tables. Food tech example: a factory holds numbers from sensors, words from comments, and photos from the packing line, which is variety.
Volume
The amount of data: how many records arrive and how much storage they need. Food tech example: a sensor recording every minute makes a large volume of readings in a week.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 2: Where Food Data Comes From
Note
Unit 2: Where Food Data Comes From
A question to start with
A yoghurt maker gets a complaint: a tub tastes sour two days before its date mark. The quality officer starts asking where the records are. The milk delivery note is in the office. The pasteurisation chart is on the production wall. The cooling log is in a notebook by the cold store. The laboratory result is in an email. The culture supplier certificate is in a filing cabinet. The shop fridge temperature was never recorded at all. Each record exists, but each one lives in a different place, in a different shape, and under a different name for the same batch. The sour tub is not a mystery of science. It is a mystery of scattered records.
Every food business is already a data business, whether it knows it or not. Records are made at every step: on the farm, at delivery, in production, in the laboratory, in storage, in transport, on the shelf, and in the home of the person who eats the food. This unit maps those sources one by one, so that later units can join them. It also explains the difference between records made by machines, records made by people, and records made outside the business, and what each kind can and cannot be trusted to say.
Questions to think about
- A delivery driver writes the arrival time on a paper note, and the cold store sensor records the same arrival time automatically. If the two times differ by an hour, which one do you trust, and how would you check?
- A laboratory tests one tub from a batch of ten thousand. What does that one result say about the other tubs, and what does it not say?
- A customer writes that a biscuit tasted stale. What extra facts would turn that comment from a complaint into useful data?
Think about the yoghurt story before you read on
The two arrival times differ because one clock is wrong, one record is written late, or the sensor and the driver mean different moments: the truck arriving, the milk being unloaded, or the milk entering the tank. The check is to look at a third record, such as the gate log or the tank temperature, and to compare the clocks. One laboratory tub describes the batch only if the batch was well mixed and the tub was taken fairly, which is why sampling matters. The stale biscuit comment becomes data when a batch number, a purchase date, and a shop are added to it, because then it can be joined to the production and storage records for that batch.
The chain of sources: from field to fork
Food moves through a chain, and data is made at every link. On the farm or at the fishing boat, records cover the raw material: the variety, the feed, the harvest date, the catch area, the weather, and the use of water. At delivery, records cover the supplier, the vehicle, the arrival time, the temperature on arrival, the weight, and the papers that travel with the load. In production, records cover the recipe, the mixing time, the cooking temperature and time, the cooling time, the packing time, and the checks made on the line. In the laboratory, records cover the samples, the methods, the results, and the checks on the instruments. In storage and transport, records cover the temperature, the humidity, the time in store, and the movement between stores. On the shelf, records cover the shop temperature, the date marks, and the sales. After sale, records cover customer comments, complaints, and returns.
No single business sees the whole chain. A factory sees its own links clearly and its suppliers and customers only through the papers and records they share. That is why the links between businesses matter as much as the records inside one business. A batch number that stays the same from the supplier to the shop lets every link add its records to the same story. A batch number that changes at each handover breaks the story into pieces.
Quick self-check
Question: Why does a batch number need to stay the same from supplier to shop?
So the records from each link can be joined. If the supplier calls a load one number, the factory calls it another, and the shop calls it a third, the delivery record, the production record, and the complaint cannot be matched. One shared number keeps the chain whole.
Question: Which link in the chain usually has the least data, and why does that matter?
Often the shelf and the home have the least data, because nobody records the shop fridge temperature every hour or the temperature in a shopping bag. That matters because food can spoil after it leaves the factory, and without those records the factory cannot tell whether the fault started with them or later in the chain.
Machine records: sensors and instruments
Machine records come from devices that measure without a person writing the value down. Temperature sensors in cold stores, ovens, and pasteurisers record readings on a set schedule. Humidity sensors record moisture in the air of a store or a drying room. Pressure sensors record the pressure in a retort or a packing machine. Flow meters record how much liquid passes a point. Weighing scales record the weight of ingredients and packs. Metal detectors and checkweighers on a packing line record every pack that passes and flag the ones that fail. Cameras record pictures of packs, labels, and seals.
Machine records are strong in two ways: they are regular, and they do not get tired or forget. A sensor records every minute through the night, when no person is watching. They are weak in two ways as well: they can drift out of calibration, and they measure only what they are placed to measure. A temperature sensor in the middle of a cold store does not know the temperature by the door, where warm air enters. A sensor that has drifted by two degrees will record a neat, regular, wrong series of numbers.
The gas composition of stored air is another machine record in food storage. Sensors can track oxygen and carbon dioxide, written CO₂, in a sealed store. As fruit respires, it uses oxygen and gives out CO₂ and water, written H₂O, as vapour. Watching those gases tells the store keeper how the fruit is breathing and whether the store air needs changing. The same idea applies to water activity meters and pH meters in the laboratory: the instrument gives a number, and the number is only as good as the calibration and the sample.
Quick self-check
Question: A cold store sensor records 4 degrees every hour for a week. The food near the door spoils. What went wrong?
The sensor measured the middle of the store, not the door. The reading was true for its own position and untrue for the food by the door. The fix is to place sensors where the food is most at risk, not only where it is easy to mount them.
Question: How can a team tell that a sensor has drifted?
By checking it against a second, calibrated instrument placed beside it, and by looking at the pattern: a slow, steady shift in every reading, with no change in the food or the process, points to drift. A sudden jump points to a real event or a fault.
People records: checklists, notes, and taste
People records come from observation and judgement. A worker ticks a cleaning checklist, notes that a drain smells odd, or records that a delivery looked dirty. A production operator writes the mixing time, the batch weight, and the reason for a line stop. A laboratory analyst writes the sample number, the method, and the result. A trained taster writes scores for flavour, texture, and smell. A driver writes the time of arrival and the condition of the load.
People records are strong where machines are weak: they notice what no sensor measures. A worker sees a torn seal, hears a strange noise from a pump, or smells a sour note in a sample. They are weak where machines are strong: they are irregular, they depend on memory and honesty, and they vary from person to person. Two tasters can score the same biscuit differently. Two workers can describe the same cleaning job in different words.
The way to make people records stronger is to design the form before the work starts: fixed choices instead of free text where possible, clear scales for taste scores, a place for the batch number on every sheet, and a time written at the moment of the check, not at the end of the shift. A checklist that asks "was the floor clean?" will get different answers from different people. A checklist that asks "was the floor swept, washed, and free of standing water?" gives each person the same test to apply.
Quick self-check
Question: Why should a checklist use fixed choices instead of free text?
Because fixed choices can be counted and compared. "Clean" means different things to different people, while a tick against "swept, washed, no standing water" means the same thing every time. Free text still has a place for the unusual thing, but the routine check should be a fixed choice.
Question: A taster scores a yoghurt as sour. A pH meter reads normal. Which is right?
Both may be right about different things. The meter measures acidity as a number, while the taster senses flavour, which also depends on other compounds, temperature, and the taster's own sense. The two records disagree, and the disagreement itself is useful: it says the sour note may not come from acidity alone, and more tests are needed.
Laboratory records: the anchor of safety and quality
Laboratory records are the anchor because they are the most controlled measurements a food business has. The laboratory tests composition: moisture, protein, fat, ash, and carbohydrate. It tests safety: microbes and their counts, toxins, residues, and heavy metals. It tests quality and keeping power: acidity, pH, fat breakdown, colour, and texture. Each result carries a sample number, a method name, a unit, a date, and the name of the person who tested it.
Laboratory records have limits that the team must respect. A laboratory tests a sample, not a whole batch, so the result describes the batch only as well as the sample represents it. A laboratory result also takes time: a microbe count needs hours or days to grow, so the food may have moved on before the result arrives. And a laboratory can only find what it tests for: a test for one toxin says nothing about another toxin that was never tested.
The laboratory also produces records about itself: calibration checks, blank tests, duplicate tests, and reference samples. These quality control records are data about the data. They tell the team whether the instruments were working and whether the results can be trusted. A result without its quality control record is a number without a witness.
The diagram shows the five parts of a good laboratory record. The steps point to the diagram: a record missing any one part is harder to trust, harder to join to other records, and harder to defend if the batch is questioned later.
Quick self-check
Question: A laboratory result says a batch is safe, but there is no record of the calibration check that day. Should the team trust the result?
They should treat it with care, not as certain. Without the calibration record, there is no proof the instrument was reading true that day. The team should repeat the check, look at duplicate results, and ask the laboratory to show the missing record before relying on the result.
Question: Why does a laboratory test take time, and why does that matter for decisions?
Because many tests need the sample to be prepared, reacted, or grown before a reading can be taken. That delay matters because the food may be packed and shipped before the result arrives, so the team needs faster checks on the line as well as the slower laboratory confirmation.
Records from outside the business
A large part of food data comes from outside the factory walls. Suppliers send certificates, delivery notes, and test results for their own materials. Transport companies send temperature logs from their vehicles and times of arrival and departure. Shops send sales records, shelf temperatures where they are measured, and customer comments. Customers send complaints, returns, and reviews. Public bodies publish food safety alerts, guidance, and recall notices. Trade and research bodies publish methods and studies.
Outside records are valuable because they cover the links the business cannot see itself. They are also harder to control. A supplier certificate may arrive as a scanned paper page, with the batch number handwritten. A transport log may use a different time zone or a different unit. A customer comment may name a product but not a batch. The work of joining outside records is mostly the work of matching names, units, times, and batch numbers, which is the subject of Unit 3.
Sales records deserve a special note. A shop that records every sale by product, date, and time gives the factory a picture of demand and a way to trace a batch: which shops received it, and when. That picture is also the fastest way to find the shops that need to be told if a batch must be held. A business that knows where its batches went can act in hours. A business that does not know must guess, and guessing is slow and wide.
Quick self-check
Question: A supplier certificate arrives as a scanned page with a handwritten batch number. What is the risk?
The batch number may be read wrong when it is typed into the system, and then the certificate is linked to the wrong batch. The fix is to check the typed number against the delivery note and the physical label, and to ask the supplier for a typed record where possible.
Question: Why are sales records useful for safety, not only for selling?
Because they show where each batch went and when. If a batch must be held, the sales record tells the team exactly which shops received it, so the hold can be narrow and fast instead of wide and slow.
Records made by customers and the public
Customer records are the noisiest and the most honest source a food business has. A complaint says what the customer experienced, in their own words, without the filter of a checklist. A review says what many customers think, over time. A return says the customer was unhappy enough to bring the food back. A social media post says the experience spread to other people.
The noise comes from the shape: free words, no batch number, no purchase date, sometimes no product name, and feelings that run ahead of facts. A customer who says "the milk was off" may have stored it warm at home, or may have bought a tub from a batch that was fine when it left the factory. The signal comes from the pattern: many comments about the same product, the same taste fault, in the same week, point to a real problem even before the laboratory confirms it.
The right way to use customer records is to collect the missing links at the moment of the complaint: product name, batch number from the pack, purchase date, shop, and how the food was stored at home. A short set of questions, asked kindly at first contact, turns a complaint into data. A complaint handled well also keeps the customer, which is a second reason to treat it with care.
Quick self-check
Question: Ten customers complain about the same biscuit in one week. The laboratory tests pass. What should the team do?
Keep looking. The laboratory may be testing for the wrong fault, the sample may not represent the complained batches, or the fault may be in storage or transport after the factory. The team should check the batch numbers on the complaints, the storage logs, and the taste panel, and not close the case on one passing test.
Time, place, and identity: the three links that join everything
Every record needs three links to be joinable: when, where, and what. When is the date and time, written in one format, with the clock set right. Where is the place: the farm, the store, the line, the shop. What is the identity: the batch number, the product code, the supplier name, the sample number. A record missing any of the three is hard to place in the story.
Time is the link that fails most often. Clocks drift. One device uses a 12-hour clock and another a 24-hour clock. One record writes the date as day then month, and another writes month then day. A sensor in a vehicle may use a different time standard from the factory clock. The fix is a rule for the whole business: one date format, one time format, clocks checked regularly, and times written at the moment of the event, not later from memory.
Place is the link that fails when names vary. "Cold store 1", "Store 1", and "the big fridge" may all mean the same room to the people who work there, but a computer sees three different places. A fixed list of place names, used by everyone, solves it. Identity fails when batch numbers are typed differently: with or without a leading zero, with a space, with a letter changed. A fixed format for batch numbers, printed on labels and scanned where possible, solves most of it.
Quick self-check
Question: One record writes 03-04 and another writes 04-03 for the same delivery. What is the problem?
One writer means the third of April and the other means the fourth of March, or the other way round, and nobody can tell which. The fix is one date format for the whole business, such as year-month-day, used everywhere.
Question: Why is scanning a batch number better than typing it?
Because typing adds mistakes: a missed zero, a swapped letter, a space in the wrong place. Scanning copies the number exactly as printed, so the link between records stays true.
How often to record, and how much to keep
How often a record is made decides what questions it can answer. A temperature recorded once a day can show a slow drift across a season. It cannot show a warm spell that lasts two hours each afternoon. A temperature recorded every minute can show the warm spell, and it also makes a large pile of numbers to store and check. The right frequency is the one that matches the question: record often enough to catch the event that matters, and no more often than the team can store and use.
How long to keep records is a second decision. Laboratory results and traceability records are kept for a long time, because a question about a batch can arrive months after the batch was sold. Minute-by-minute sensor readings may be kept in full for a shorter time and then summarised: the daily high, low, and average kept, the minute values archived or discarded. The rule is to keep the summary and the link to the original, so the story can be rebuilt if it is ever needed.
Storage itself needs care. Records kept in one place, with backups, survive a broken computer or a spilled cup of tea. Records kept only on one laptop, or only on paper in one cupboard, do not. The business should be able to answer, for any record: where is it kept, who can find it, and what happens if that place fails?
Quick self-check
Question: A cold store temperature is recorded once a day at noon. A warm spell happens every afternoon. Will the record show it?
No. A noon reading misses the afternoon entirely. The team should record more often, or move the reading time, or add a sensor that records through the day, so the warm spell is caught.
Try it yourself
Choose one product and draw its chain on a page: farm or source, delivery, production, laboratory, storage, transport, shop, and customer. Under each link, write the records made there, who makes them, and where they are kept. Then draw a line between any two records that share a batch number, a date, or a product code. Count the records with no line at all. Those are the orphans: good records that cannot be joined to the story. Pick one orphan and write the single change that would give it a link, such as adding a batch number box to a form, or writing the time at the moment of the check.
Key ideas to carry forward
- Data is made at every link of the food chain, from the farm to the customer, and the links between businesses matter as much as the records inside one business.
- Machine records are regular and tireless, but they can drift, and they measure only where they are placed.
- People records notice what no sensor measures, but they vary from person to person, so forms need fixed choices and clear scales.
- Laboratory records are the anchor of safety and quality, and their quality control records are the proof that the results can be trusted.
- Records from outside the business cover the links the factory cannot see, and they need matching of names, units, times, and batch numbers.
- Customer comments are noisy and honest: one comment is a story, many comments about the same fault in the same week are a signal.
- Every record needs three links to be joinable: when, where, and what, and time is the link that fails most often.
- Record often enough to catch the event that matters, keep records long enough to answer late questions, and store them where they can be found and backed up.
Glossary
Calibration
Checking a measuring device against a known standard and adjusting it so its readings are true. Food tech example: a thermometer is checked in iced water and in boiling water so its readings can be trusted in a cold store.
Certificate
A document from a supplier that states what a material is and what tests it passed. Food tech example: a flour supplier sends a certificate with the protein and moisture results for the delivery.
Checklist
A list of checks to be done and ticked off in order, so nothing is missed. Food tech example: a cleaning checklist lists each surface to be washed and signed off before production starts.
Cold chain
The unbroken series of chilled steps that keeps chilled food cold from production to sale. Food tech example: milk moves from the chilled tank to the chilled truck to the chilled shop cabinet, and each step is recorded.
Complaint
A message from a customer saying a food was not as expected. Food tech example: a customer reports a sour yoghurt, and the batch number on the tub links the complaint to the production records.
Delivery note
A paper or electronic note that travels with a delivery, listing the supplier, the items, the weights, and the batch numbers. Food tech example: the delivery note for a flour load lists the supplier, the weight, and the batch number.
Flow meter
A device that measures how much liquid or gas passes a point in a pipe. Food tech example: a flow meter records how much milk entered a pasteuriser during a run.
Humidity sensor
A device that measures moisture in the air. Food tech example: a humidity sensor in a drying room shows whether the air is dry enough to dry fruit evenly.
Metal detector
A device on a packing line that signals when metal is found in a pack. Food tech example: a metal detector rejects a pack of biscuits that contains a metal fragment.
pH meter
An instrument that measures how acidic or alkaline a sample is. Food tech example: a pH meter checks yoghurt acidity during fermentation.
Quality control record
A record that shows a test or an instrument was checked, such as a calibration check or a duplicate test. Food tech example: a blank test run beside a real sample shows the laboratory method was clean that day.
Respiration
The process by which living plant tissue uses oxygen and gives out carbon dioxide and water vapour. Food tech example: stored apples respire, so a sealed store gains CO₂ and H₂O vapour, and the store air must be managed.
Sales record
A record of what was sold, where, and when, usually by product and date. Food tech example: a sales record shows which shops received one batch of yoghurt and on which dates.
Sample
A small portion taken to represent a larger amount for testing. Food tech example: one tub taken from a batch of yoghurt is the sample sent to the laboratory.
Supplier
A business that provides raw materials, ingredients, or packaging to a food business. Food tech example: a flour supplier delivers wheat flour to a bakery each week.
Taste panel
A group of trained people who taste and score food for flavour, texture, and smell under set conditions. Food tech example: a taste panel scores biscuits for crispness and flavour before a new recipe is approved.
Time standard
The agreed clock and date format used for all records in a business. Food tech example: every device in a factory is set to the same time, so a sensor reading and a checklist tick can be matched.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 3: Cleaning and Organising Food Data
Note
Unit 3: Cleaning and Organising Food Data
A question to start with
A quality officer opens a spreadsheet of flour moisture results for the season. The first thing she sees is a row that says the moisture was 140 percent, which is impossible: a flour cannot hold more water than its own weight and still be flour. A few rows down, the same supplier is spelled three different ways. Further down, the same batch appears twice, once with a result of 13 percent and once with 14 percent. Some rows have no date. Some temperatures are in one scale and some in another. The officer has two choices: clean the table first, or analyse it as it is. If she analyses it as it is, the average will be wrong, the supplier comparison will be wrong, and the chart will show a spike that never happened. Cleaning is not tidying for its own sake. It is the work that makes every later answer true.
Dirty data is normal. It is not a sign of a bad business. It is what happens when real people and real machines record real events in a hurry. This unit explains how to find the dirt, how to fix it, how to organise the clean records into tables that can be joined, and how to prove that the cleaning was done honestly. The rule that runs through the unit is simple: never change a value silently, and never throw a record away without a note saying why.
Questions to think about
- A moisture result reads 140 percent. Is that a typing error, a wrong unit, or a broken instrument, and how would you tell the difference?
- The same batch appears twice with two different results. What are the possible reasons, and what should you do before you delete one of them?
- A table has no date on some rows. Can those rows still be used, and for which questions?
Think about the flour table before you read on
A value of 140 percent is most likely a typing error, such as 14.0 typed as 140, but it could also be a result in different units or a sensor fault. The check is to look at the original record, the neighbouring rows, and the instrument log. A batch that appears twice may be a true duplicate, a repeat test, or two different samples from the same batch, and the answer is in the sample numbers and the times, not in a guess. Rows without a date can still be used for questions that do not need time, such as the range of results, but they cannot be used for a chart over time.
What dirty data looks like
Dirty data takes a few familiar shapes. Missing values are the first: a blank cell where a temperature should be, a laboratory result with no unit, a delivery with no batch number. Duplicates are the second: the same record typed twice, the same batch under two spellings, the same reading sent twice by a sensor. Wrong values are the third: a temperature of 140 degrees in a cold store, a negative weight, a pH of 20, which cannot happen on the usual pH scale. Inconsistent names are the fourth: "Flour A", "flour a", and "FLOUR-A" for the same material. Mixed units are the fifth: some weights in grams and some in kilograms, some temperatures in one scale and some in another, some dates in day-first order and some in month-first order. Outliers are the sixth: values that sit far from the rest, which may be true or may be errors, and which need checking before they are believed or removed.
Each shape needs a different fix, and the fix starts with seeing the dirt. The first cleaning step is always to look: count the rows, count the blanks in each column, list the distinct spellings of each name, find the highest and lowest values, and plot the numbers. A table that has never been looked at in this way is a table that has never been cleaned.
Quick self-check
Question: A pH reading is written as 20. What is wrong, and what should you do first?
The usual pH scale runs from 0 to 14, so 20 cannot be a true pH. The first step is to check the original record: was it 2.0 with the point missed, was it a different measurement typed in the pH column, or was the meter faulty? Do not change it to 2.0 on a guess. Check first, then decide.
Question: Why is listing the distinct spellings of a supplier name a cleaning step?
Because the same supplier spelled three ways looks like three suppliers to a computer, so a comparison by supplier will be wrong. Listing the spellings shows the variants, and then one standard spelling can be chosen and applied to all rows.
Finding errors: ranges, rules, and looking at the shape
The fastest way to find wrong values is to set a sensible range for each measurement and flag everything outside it. A cold store temperature should sit inside a narrow band. Moisture in flour should sit inside a band that flour can actually hold. A weight should be positive. A date should fall inside the season the table covers. Any value outside its range is not automatically wrong, but it is automatically worth checking.
Rules between columns catch a second kind of error. If the packing date is earlier than the production date, one of the two dates is wrong. If the weight of a pack is greater than the weight of the whole batch, the weight is wrong. If a laboratory result is present but the sample number is blank, the result cannot be linked to a batch. These cross-checks are called validation rules, and they can be written down once and run every time new records arrive.
Looking at the shape of the numbers catches a third kind. A chart of temperatures over a day should rise and fall smoothly. A single spike in the middle of a smooth line is suspicious. A chart of laboratory results over a season should cluster around a middle value with a spread. A result far from the cluster, with nothing around it, is an outlier that needs a check. The eye is good at spotting a spike, and the chart is the cheapest checking tool there is.
Quick self-check
Question: A temperature chart shows a smooth line with one sudden spike to a very high value for a single minute, then back to smooth. What does that suggest?
It suggests a brief fault, such as a door opened, a sensor knocked, or a single wrong reading, rather than a real warming of the whole store. The check is to look at the neighbouring sensors, the door log, and the readings before and after the spike.
Question: A packing date is earlier than the production date for the same batch. What does that tell you?
That at least one of the two dates is wrong, because a batch cannot be packed before it is made. The fix is to check the original records for both dates and correct the one that is wrong, with a note saying what was changed and why.
Fixing, flagging, and the cleaning log
There are three honest ways to handle a dirty value: fix it, flag it, or remove it, and each one needs a note. Fix it when the true value can be proved from the original record: a typed 140 that the paper sheet shows as 14.0, a supplier name corrected to the standard spelling, a unit converted from grams to kilograms with the conversion written down. Flag it when the value is suspicious but cannot be proved wrong: keep the value, add a mark that says it needs checking, and leave it out of the main analysis until it is checked. Remove it when it is proved wrong and cannot be fixed, or when it is a true duplicate of another record: take it out of the working table, keep it in a separate list of removed records, and write the reason.
The cleaning log is the record of all three actions. Each line of the log says: which record, which field, what was wrong, what was done, who did it, and when. The log is what makes cleaning honest. Without it, nobody can tell whether a value was corrected from evidence or changed to make a chart look better. With it, any person can rebuild the clean table from the raw table and get the same result. That rebuild test is the standard for good cleaning: the raw records plus the log should produce the clean table exactly.
A second rule protects the raw data: never overwrite the original. The raw table stays as it arrived, mistakes and all. Cleaning makes a new, clean copy. If a fix turns out to be wrong, the original is still there to go back to. This is the same discipline as in the laboratory: the notebook entry stays, and a correction is written beside it with a date and a signature, never rubbed out.
The diagram shows the cleaning steps in order. The steps point to the diagram: skipping the log is the most common mistake, because a clean table without a log cannot be checked or rebuilt by anyone else.
Quick self-check
Question: Why should the raw table never be overwritten?
Because the raw table is the evidence. If a fix turns out to be wrong, the team needs the original to go back to. If an auditor asks how a value was changed, the raw table plus the log gives the answer. Overwriting destroys the evidence.
Question: A value looks wrong but you cannot prove what the true value is. What should you do?
Flag it and leave it out of the main analysis until it is checked. Keep the value in the table with a flag, note the doubt in the log, and say in the report that one value was flagged and excluded. Guessing a replacement value would invent data.
Missing values: what to do with blanks
Missing values are blanks where a number or a name should be, and they need a decision, not a guess. The first question is why the value is missing. A sensor may have been off. A worker may have skipped a line. A laboratory may not have run a test. A form may not have had a box for the value at all. The reason matters because it decides what the blank means: a sensor that was off tells nothing about the temperature, while a test that was never run tells nothing about the batch.
There are three honest treatments. Leave the blank and analyse only the records that have the value, saying clearly how many records were left out. Fill the blank with a careful estimate only when the method is sound and stated, such as carrying a daily average forward for a short gap in a smooth temperature series, and never for a laboratory result, which must never be invented. Or drop the row for questions that need the missing value, while keeping it for questions that do not. What must never happen is filling a blank with a silent guess and then reporting the result as if the data were complete.
A useful habit is to count and report the missing values for each column before any analysis: how many blanks, out of how many rows, and why. A table with a few blanks in one column is fine. A table where half the dates are missing cannot support a chart over time, and the team should say so rather than draw a chart that pretends otherwise.
Quick self-check
Question: A temperature sensor was off for two hours. Can you fill the gap with the average of the readings before and after?
For a smooth, slow-changing store temperature, a careful estimate may be reasonable for a chart, if the method is stated and the gap is marked. For a laboratory result or a safety decision, never. The key is to state the method and mark the gap, so nobody mistakes an estimate for a measurement.
Question: Why must a missing laboratory result never be filled with a guess?
Because a laboratory result is evidence about safety and quality. A guessed result could hide a real problem or condemn a good batch. If the test was not run, the honest statement is that the test was not run, and the batch must be judged on the evidence that does exist.
Organising tables: one fact in one place
Cleaning fixes the values. Organising fixes the shape. A good table has one row per thing and one column per fact: one row per batch, with columns for batch number, product, date, and each test result; or one row per reading, with columns for time, sensor, and value. A table that mixes two things in one row, such as a batch row that also holds three separate test results in one cell, is hard to count, hard to plot, and hard to join.
Three rules make tables joinable. First, give every table a key: a column that identifies each row, such as a batch number or a reading number, with no duplicates and no blanks. Second, use the same name for the same thing in every table: the batch column is called the same thing in the production table and the laboratory table, so the two can be matched. Third, keep one fact in one place: the supplier name lives in the supplier table, and other tables carry only the supplier code, so a change to the name is made once, not in fifty places.
Units and formats belong in this organising step too. Pick one unit for each measurement and convert all values to it, with the conversion noted in the log. Pick one date format and one time format and apply them everywhere. Pick one spelling for each name and apply it everywhere. The clean, organised tables that result are the input to every analysis in Units 4 and 5, and the time spent here pays back at every later step.
Quick self-check
Question: A table has one row per batch, and one cell holds three moisture results separated by commas. What is wrong with that shape?
The cell holds three facts, so the table cannot be counted or plotted properly. The average will be wrong, and a chart will miss two of the three values. The fix is to give each result its own row, or its own column with a clear name, so each fact sits in its own cell.
Question: Why should the supplier name live in only one table?
So a change or a correction is made once. If the name is typed into fifty rows across five tables, one correction will miss some rows, and the same supplier will again look like several suppliers.
Joining tables: matching on keys
Joining is the step where the separate tables become one story. A production table holds batch number, product, date, and mixing time. A laboratory table holds batch number, test name, and result. A delivery table holds batch number, supplier, and arrival temperature. Joining on the batch number puts the mixing time, the test result, and the arrival temperature on one row for each batch, and that joined row is what the analysis reads.
Joins fail for the same reasons cleaning matters: keys spelled differently, keys missing, keys duplicated. A batch number with a leading zero in one table and without it in another will not match. A batch that appears twice in one table will double when joined, so one batch will count as two. The cleaning step must fix the keys before the join, and the join itself must be checked: how many rows matched, how many did not, and why. A join that silently drops half the batches is worse than no join, because the analysis will be based on the half that happened to match.
A useful check after every join is to count. Count the rows before, count the rows after, count the rows that found no match, and look at a few of the unmatched rows by hand. If the unmatched rows are all from one supplier or one week, the cause is probably a naming or format difference in that group, and it can be fixed. If the unmatched rows are scattered, the cause may be missing batch numbers, and the team should fix the collection step that left them blank.
Quick self-check
Question: A join on batch number returns fewer rows than the production table had. What has happened?
Some production batches found no matching laboratory row, so they were dropped from the joined table. The team should list the dropped batches and find out why: no test was run, the batch number differs, or the laboratory table uses a different key.
Question: A batch appears twice in the laboratory table. What happens when it is joined to the production table?
It will appear twice in the joined table, so it will be counted twice in every average and chart. The team should find out why it appears twice: a repeat test, a duplicate entry, or two samples, and handle it before analysing.
Outliers: true, wrong, or telling
An outlier is a value that sits far from the rest. The first job is to decide which of three things it is. It may be wrong: a typing error, a wrong unit, a broken sensor. It may be true and important: the one batch that really did overheat, the one delivery that really was warm, the one day the line really did stop. Or it may be true and ordinary: a value at the edge of normal variation that looks far only because the rest of the data clusters tightly.
The decision needs evidence, not a rule of thumb. Check the original record. Check the neighbouring readings from other sensors and other records at the same time. Check whether the same batch shows the same odd value in a second test. Ask the people who were there: was something different that day? A value that is confirmed by a second record and by the people on the shift is a true event, and removing it would hide the very thing the team needs to see.
If an outlier is kept, say so in the report, and show the analysis with and without it where it matters. If an outlier is removed, say so in the log, with the reason and the evidence. The worst choice is to remove outliers silently to make a chart look smooth, because the smooth chart then hides the real events, and the next real event will be missed the same way.
Quick self-check
Question: A single flour moisture result is far higher than all the rest, and the delivery note says it rained while the load was uncovered. Should the value be removed?
No. The note gives a real reason for a real value, so the outlier is evidence, not an error. Keep it, note the reason, and use it to argue for covering loads in rain. Removing it would hide a lesson the business paid for.
Question: Why show an analysis with and without an outlier?
So the reader can see whether the answer depends on one value. If the answer is the same either way, the outlier does not decide the matter. If the answer changes, the reader knows the result is fragile and the outlier needs more checking.
Water, moisture, and units: a worked example
Moisture is a good example of why units and definitions matter in cleaning. Moisture is water, written H₂O, held in a food. A moisture result can be reported on a wet basis, as a share of the total weight including water, or on a dry basis, as a share of the weight without water. The same flour can give two different numbers under the two definitions, and a table that mixes them will look as if the flour changed when only the definition changed.
A cleaning pass on a moisture table should ask: which basis is each result on, and is it written in the table? Are all results in the same unit, percent, and not a mix of percent and fractions? Are the temperatures and times of the drying test recorded, so a result can be checked? Is the sample number present, so the result can be linked to its batch? Each question is a check, and each check that fails is a fix or a flag in the log.
The same care applies to any measurement with more than one definition: fat as a share of the whole food or of the dry matter, acidity as one acid or another, microbe counts as counts per gram or per millilitre. The definition is part of the value. A number without its definition is a number without a meaning.
Quick self-check
Question: Two moisture results for the same flour read 13 percent and 15 percent. One is on a wet basis and one on a dry basis. Are they in conflict?
Not necessarily. The dry basis number is always higher than the wet basis number for the same sample, because the dry weight is smaller. The two numbers may describe the same flour under different definitions, so the first step is to convert them to one basis before comparing.
Try it yourself
Take a small table of twenty rows that you make yourself, with columns for batch number, date, product, temperature, and one laboratory result. Now dirty it on purpose, in a copy: blank three cells, duplicate two rows, misspell one supplier or product name twice, type one temperature with an extra zero, and write two dates in a different format. Then clean the copy, step by step, using the five steps in the diagram above, and keep a cleaning log as you go. When you finish, compare the clean copy with the original. They should match exactly, apart from the deliberate errors you fixed, and the log should let a classmate rebuild your clean table from your dirty copy without asking you anything.
Key ideas to carry forward
- Dirty data is normal: missing values, duplicates, wrong values, inconsistent names, mixed units, and outliers appear in every real food business.
- The first cleaning step is to look: count rows, count blanks, list spellings, and find the highest and lowest values.
- Ranges, rules between columns, and charts catch different kinds of error, and all three should be used.
- Fix with proof, flag with doubt, or remove with reason, and write every action in a cleaning log.
- Never overwrite the raw table: keep it untouched, and make the clean copy from it.
- A missing value needs a decision, not a silent guess, and a missing laboratory result must never be filled with an invented number.
- Good tables have one row per thing, one column per fact, one key per table, and one name for the same thing everywhere.
- Joins on keys turn separate tables into one story, and every join must be checked by counting matches and looking at the rows that did not match.
- An outlier may be wrong, true and important, or true and ordinary, and the decision needs evidence from the original record and from neighbouring records.
- A number without its unit and its definition, such as the basis of a moisture result, has no clear meaning.
Glossary
Blank
An empty cell where a value should be. Food tech example: a blank in the temperature column means no reading was recorded for that time.
Cleaning log
A written record of every change made while cleaning data: which record, what was wrong, what was done, and by whom. Food tech example: the log shows that a moisture value of 140 was corrected to 14.0 from the paper sheet.
Cross-check
A check that compares two related fields to see whether they agree. Food tech example: checking that the packing date is not earlier than the production date for the same batch.
Duplicate
A record that appears more than once when it should appear once. Food tech example: the same laboratory result typed twice makes one batch look like two batches.
Flag
A mark added to a value to show it is doubtful and needs checking. Food tech example: a temperature reading far outside the usual range is flagged and left out of the main chart until it is checked.
Join
Combining two tables by matching a shared key, so the facts from both sit on one row. Food tech example: joining the production table and the laboratory table on batch number puts the mixing time and the test result together.
Key
A column that identifies each row in a table, such as a batch number, with no duplicates and no blanks. Food tech example: the batch number is the key of the production table.
Missing value
A value that should be present but is not recorded. Food tech example: a laboratory result with no date is a missing value for the date column.
Outlier
A value that sits far from the rest of the values. Food tech example: one moisture result far above all the others is an outlier that may be an error or a real wet delivery.
Range check
A check that a value sits inside the sensible range for that measurement. Food tech example: a cold store temperature outside its set band is flagged by a range check.
Raw data
Data exactly as it arrived, before any cleaning, kept untouched. Food tech example: the sensor file as downloaded is the raw data, and cleaning works on a copy of it.
Unit
The standard amount a measurement is reported in, such as grams, degrees, or percent. Food tech example: a moisture result needs its unit, percent, to be read correctly.
Validation rule
A written rule that a record must pass to be accepted, such as a date order or a value range. Food tech example: a rule that a weight must be positive rejects a negative weight at entry.
Wet basis
A way of reporting a share as part of the total weight including water. Food tech example: moisture on a wet basis is the weight of water, H₂O, divided by the total weight of the flour.
Dry basis
A way of reporting a share as part of the weight without water. Food tech example: moisture on a dry basis is the weight of water, H₂O, divided by the weight of the flour after drying.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 4: Finding Patterns: Descriptive and Diagnostic Analytics
Note
Unit 4: Finding Patterns: Descriptive and Diagnostic Analytics
A question to start with
A biscuit factory makes the same product on two lines. Line A has a few more broken biscuits than Line B, and nobody knows why. The quality team puts a month of records on the table: the breakage counts, the baking temperatures, the dough moisture, the flour supplier, the operator on shift, and the time of day. The first chart shows the average breakage per line, and Line A is indeed higher. The second chart splits the breakage by time of day, and the difference almost disappears in the morning and grows wide in the afternoon. The third chart splits by flour delivery, and the afternoon gap sits mostly on one delivery. The story changes with each split: it is not "Line A is worse". It is "one flour delivery, baked in the afternoon, on the warmer line, breaks more". That change in the question is descriptive and diagnostic analytics at work: first describe what happened, then ask why it happened, by splitting the data in ways that test each possible cause.
This unit covers the two kinds of looking back. Descriptive analytics answers: what happened, how much, how often, and where. Diagnostic analytics answers: why did it happen, and which possible cause fits the data best? Both use the clean, joined tables from Unit 3, and both are done before any prediction is attempted, because a prediction built on an undescribed past is a guess.
Questions to think about
- A factory reports that its average breakage is low. Why might the average hide a real problem?
- Two charts show the same data, one by line and one by time of day. Why can they tell different stories?
- A supplier change and a season change happen in the same week. How can the team tell which one caused a rise in complaints?
Think about the biscuit story before you read on
An average hides a problem when one group is bad and another is good: the average sits in the middle and looks fine. Splitting by line, by time, or by supplier shows the groups separately, and the bad group stands out. Two charts can tell different stories because each split tests a different possible cause, and the true cause shows up in the split that matches it. When two changes happen together, the team needs batches that differ in only one of them, or a longer period where the two changes do not line up, before it can separate their effects.
Descriptive analytics: counting, averaging, and showing the spread
Descriptive analytics starts with counting. How many batches were made, how many failed, how many complaints arrived, how many deliveries were late? Counts turn a pile of records into a picture of size. A count on its own is rarely enough, so the next step is to summarise the numbers: the average, which is the sum divided by the count; the middle value, which is the value in the middle when all values are placed in order; and the range, which is the distance from the lowest to the highest value.
The average is useful and easy to misread. A few very high values can pull the average up even when most values are low. The middle value, also called the median, is not pulled in that way, so comparing the average and the middle value tells the team whether a few extreme values are shaping the picture. The spread matters as much as the centre: two lines can have the same average breakage, while one line is steady every day and the other swings from near zero to very high. The steady line has a small spread and the swinging line has a large spread, and they need different fixes.
Rates are often more useful than raw counts. Ten failed batches out of a hundred is a different story from ten failed batches out of a thousand. A rate puts the count against the total it came from, so lines and months of different sizes can be compared fairly. A chart of the failure rate by week shows whether the problem is growing, shrinking, or steady, and a chart of the rate by supplier shows which supplier needs attention first.
Quick self-check
Question: Line A made 100 batches and 10 failed. Line B made 1000 batches and 10 failed. Which line has the worse problem?
Line A. Both lines had 10 failures, but Line A failed one in ten batches while Line B failed one in a hundred. The rate, not the count, is the fair comparison.
Question: The average moisture is normal, but the middle value is lower and a few values are very high. What does that suggest?
That a few high values are pulling the average up while most batches are drier than the average suggests. The team should look at the high batches separately, because the average hides them.
Charts that answer questions
A chart is an argument made visible, and the choice of chart should follow the question. A line chart over time answers: is this rising, falling, or steady? A bar chart by group answers: which group is highest or lowest? A histogram, which is a bar chart of how many values fall in each band, answers: what does the spread look like, and are there two humps where one was expected? A scatter plot, with one measurement on each axis and one dot per batch, answers: do these two things move together?
Each chart needs the same care as a table: clear labels, the unit on the axis, the time period named, and the number of records behind it. A chart without a unit can be read in two ways. A chart without the count can hide that it is based on three batches. A chart with a squeezed axis can make a tiny change look large, and a chart with a stretched axis can make a large change look tiny. The honest chart starts its axis at a sensible place, uses the same scale when comparing two charts, and says how many records it shows.
A control chart is a special line chart used in quality work. It plots each result in time order, with a centre line at the usual level and two boundary lines above and below it. A result outside the boundaries is a signal to look. A run of results all on one side of the centre, or a steady climb across many points, is also a signal, even if no single result crosses a boundary. The control chart turns "is this normal variation or a real change?" from a feeling into a visible test.
The diagram shows the chart choice for each question. The steps point to the diagram: a chart chosen for the wrong question will answer a question nobody asked, however pretty it looks.
Quick self-check
Question: You want to know whether oven temperature and biscuit colour move together. Which chart do you draw?
A scatter plot, with temperature on one axis and colour on the other, and one dot per batch. If the dots rise together, higher temperature goes with darker colour. If they form a cloud with no direction, the two do not move together in this data.
Question: A control chart shows seven results in a row above the centre line, all inside the boundaries. Is that a signal?
Yes. A run of points on one side suggests the process has shifted up, even though no single point crossed a boundary. The team should look for what changed before the run began.
Splitting the data: the heart of diagnosis
Diagnosis is mostly splitting. The team takes the whole set of batches and splits it by one possible cause at a time: by line, by supplier, by shift, by time of day, by recipe, by storage time. In each split, it compares the groups: does one group fail more, taste worse, or vary more than the others? A split that shows a large difference points to a cause worth testing. A split that shows no difference rules a cause out, which is also progress.
The power of splitting is that it stops the team from arguing about opinions. Instead of debating whether the night shift is careless, the team splits by shift and looks. Instead of blaming a supplier on a feeling, the team splits by supplier and looks. The data may confirm the feeling, or it may show that the feeling was wrong and the true cause sits elsewhere. Either way, the next step is clearer.
Splitting has a trap: small groups. A group with three batches can look very good or very bad by chance. A group with three hundred batches gives a steadier picture. The team should always show the count behind each group, and treat a small-group difference as a lead to test, not as a verdict. A second trap is splitting too many ways at once: split by line, by shift, by supplier, and by week together, and each cell holds one or two batches, and the picture dissolves into noise. Split one way at a time, and keep the groups large enough to read.
Quick self-check
Question: Splitting by shift shows the night shift fails more. Splitting by line shows no difference. What should the team do next?
Follow the shift lead: split the night shift further, by operator, by product, or by week, and check whether the difference stays. Also check whether the night shift handles a different product or a different supplier, because the true cause may be the product, not the shift.
Question: A group with two batches has a failure rate of one in two. Should the team act on it?
Not as a verdict. Two batches are too few to judge a rate: one failure more or less swings the rate wildly. Treat it as a lead, gather more batches, and look at the same group over a longer period before acting.
Correlation: moving together is not causing
When two measurements move together, they are correlated. As oven temperature rises, biscuit colour darkens. As storage time lengthens, vitamin C in juice falls. As flour moisture rises, dough stickiness rises. Correlation is easy to see on a scatter plot: the dots form a rising or falling band rather than a cloud.
Correlation is not proof of cause. Two things can move together because one causes the other, because both are caused by a third thing, or by chance. Ice cream sales and sunburn rise together in summer, but ice cream does not cause sunburn: the hot season causes both. In food, the third thing is often the season, the storage time, or a change in raw material that affects two measurements at once. A correlation is a lead, and the test of the lead is to change one thing while holding the rest steady, and to see whether the other thing follows.
The strength of a correlation matters too. A tight band of dots means the two measurements move closely together. A wide cloud with a slight tilt means they move together weakly, and knowing one tells little about the other. A correlation found in one season or one factory may not hold in another, so the team should check whether the pattern repeats in a second set of data before trusting it.
Quick self-check
Question: Complaints about stale biscuits rise in the same weeks that delivery times lengthen. Does that prove slow delivery causes staleness?
No. Both may be caused by a third thing, such as hot weather that slows traffic and also dries biscuits faster, or a busy season that strains both transport and storage. The test is to compare batches with the same storage time but different delivery times, and see whether the complaint rate still differs.
Question: A scatter plot shows a wide cloud with a slight upward tilt. What does that mean?
The two measurements move together weakly. Knowing one gives only a little information about the other, so the team should not use one to predict the other, and should look for a stronger cause.
Comparing groups fairly
Diagnosis often comes down to comparing two groups: the batches before a change and after a change, the batches from supplier A and supplier B, the batches baked at one temperature and at another. A fair comparison needs the groups to be alike in everything except the thing being tested. If supplier A delivers in the cool season and supplier B in the hot season, a comparison of the two suppliers is also a comparison of two seasons, and the result cannot be pinned on the supplier.
The way to compare fairly is to match the groups: compare batches from the same season, the same line, and the same product, differing only in the supplier. Or compare the same line before and after a change, with the recipe and the raw material held steady. When a fair match is not possible from past records, the team should run a small planned test: make a set of batches that differ only in the thing under test, record the results, and compare those. Past data can suggest the cause. A planned test can confirm it.
Averages alone can mislead a comparison. Two groups can have the same average and different spreads, or the same spread and different shapes. The team should compare the middle values, the spreads, and the charts, not only the averages. It should also ask whether the difference is large enough to matter in practice: a difference of a fraction of a percent in moisture may be real but too small to change the product, while a difference of several percent will change the dough and the bake.
Quick self-check
Question: Why is comparing supplier A in winter with supplier B in summer unfair?
Because the season differs as well as the supplier, so any difference in the results could come from the season, the supplier, or both. A fair comparison holds the season steady and changes only the supplier.
Question: Two groups have the same average breakage but one varies much more. Which group needs attention?
The varying group, because its worst days are far worse, and the variation itself suggests the process is not under control. The steady group with the same average is predictable, and its average can be lowered by a steady improvement.
Pareto and the few causes that matter most
Not all causes matter equally. In many food businesses, a small number of fault types cause most of the failures: a few defect types cause most of the rejected packs, a few suppliers cause most of the late deliveries, a few complaint types cause most of the complaints. A Pareto chart shows this by ranking the fault types from the most common to the least, with bars for the count of each and a line for the running total. The chart often shows that fixing the top two or three fault types would remove most of the problem, while fixing the long tail of rare faults would take great effort for little gain.
The Pareto chart is a guide to where to start, not a rule to ignore the rare faults. A rare fault can still be serious for safety: a single case of a harmful contaminant matters even if it is rare. The chart should be read with safety in mind: sort by count for quality and efficiency, and always look separately at the serious faults, however few they are. A fault that harms people is never a low priority because it is rare.
Quick self-check
Question: A Pareto chart shows that three defect types cause most rejected packs. What should the team do first?
Work on the top three defects first, because fixing them removes the largest share of the problem. At the same time, check the rare defects separately for any safety risk, because a rare safety fault still needs fixing even if it is low on the chart.
Root cause thinking: asking why five times
Diagnostic analytics ends with a cause that can be fixed, and the way to reach it is to keep asking why. A batch failed: why? The oven temperature dipped. Why did it dip? The gas supply faltered. Why did it falter? A valve was partly blocked. Why was it blocked? It had not been cleaned on schedule. Why was it not cleaned? The cleaning checklist had no line for that valve. Each answer moves the cause one step back, from the symptom to the system, and the fix at the last step, adding the valve to the checklist, prevents the failure from returning.
The data supports each step: the temperature log proves the dip, the maintenance log proves the missed cleaning, the checklist proves the missing line. Without data, each why is a guess. With data, each why is a check. The team should stop asking why when the answer points to a fix the team can actually make and control. "The weather was hot" is not a fixable cause. "The store door was left open because there is no closer on it" points to fitting a door closer, which is a fix.
Quick self-check
Question: A batch failed because the dough was undermixed. Is "the operator was careless" a good root cause?
No. It blames a person and points to no fix. A better chain asks why the dough was undermixed: was the time too short, was the mixer loaded too full, was the timer unclear, was the training missing? Each of those points to a fix in the process, the equipment, or the instruction.
Try it yourself
Take a small set of made-up batch records for one product, twenty batches, with columns for batch, line, shift, supplier code, moisture result, and a pass or fail mark. Make the table yourself, and build in one pattern: for example, make the afternoon batches from one supplier fail more often. Then describe the data: count the passes and fails, work out the failure rate, and draw a bar chart of the failure rate by line, by shift, and by supplier, on paper. Which split shows the pattern most clearly? Write one paragraph saying what happened, and one paragraph saying why you think it happened, and what test you would run to confirm it. Swap your table with a classmate and see whether they find the same pattern from your charts.
Key ideas to carry forward
- Descriptive analytics answers what happened, how much, how often, and where, using counts, averages, middle values, ranges, spreads, and rates.
- An average can hide a problem: compare the average with the middle value, look at the spread, and use rates so groups of different sizes can be compared fairly.
- Choose the chart by the question: line charts for time, bar charts for groups, histograms for spread, scatter plots for two things moving together, and control charts for normal variation against real change.
- Diagnosis is mostly splitting: split by one possible cause at a time and compare the groups, showing the count behind each group.
- Correlation means two things move together, and it is a lead, not proof of cause: a third thing may cause both.
- A fair comparison holds everything steady except the thing being tested, and a planned test can confirm what past data only suggests.
- A Pareto chart shows that a few fault types often cause most failures, and fixing the top few gives the largest gain, while serious safety faults are never low priority because they are rare.
- Asking why, step by step, moves the cause from the symptom to the system, and the chain should end at a fix the team can make and control.
Glossary
Average
The sum of a set of values divided by how many values there are. Food tech example: the average moisture of ten flour deliveries is the sum of the ten results divided by ten.
Bar chart
A chart with bars that show the size of each group. Food tech example: a bar chart shows the failure rate for each supplier side by side.
Control chart
A line chart that plots results in time order with a centre line and boundary lines, to show normal variation against real change. Food tech example: a control chart of biscuit weights shows when the packing line drifts off its usual level.
Correlation
Two measurements moving together, rising together or one rising as the other falls. Food tech example: as oven temperature rises, biscuit colour darkens, so temperature and colour are correlated.
Descriptive analytics
Analysis that describes what happened: counts, averages, spreads, and charts of past data. Food tech example: a monthly report of batches made, batches failed, and the failure rate is descriptive analytics.
Diagnostic analytics
Analysis that asks why something happened, by splitting data and comparing groups to test possible causes. Food tech example: splitting failures by supplier and by shift to find which one explains the rise is diagnostic analytics.
Histogram
A bar chart that shows how many values fall in each band, showing the shape and spread of the data. Food tech example: a histogram of pack weights shows most packs near the target and a few far from it.
Line chart
A chart that joins points in time order to show a rise, a fall, or a steady level. Food tech example: a line chart of cold store temperature through a week shows the warm spell each afternoon.
Median
The middle value when all values are placed in order. Food tech example: the median moisture of nine deliveries is the fifth value when they are ordered from lowest to highest.
Pareto chart
A chart that ranks fault types from the most common to the least, with a line for the running total. Food tech example: a Pareto chart of rejected packs shows that three defect types cause most rejections.
Range
The distance from the lowest value to the highest value in a set. Food tech example: the range of moisture results for a season is the highest result minus the lowest.
Rate
A count put against the total it came from, so groups of different sizes can be compared. Food tech example: ten failures out of a hundred batches is a rate of one in ten.
Root cause
The underlying cause of a problem that, when fixed, stops the problem from returning. Food tech example: a missing line on a cleaning checklist is the root cause of a blocked valve that made an oven dip.
Scatter plot
A chart with one measurement on each axis and one dot per item, showing whether two things move together. Food tech example: a scatter plot of oven temperature against biscuit colour shows the dots rising together.
Spread
How widely values are scattered around their centre. Food tech example: two lines can have the same average breakage, while one is steady and the other swings widely, which is a difference in spread.
Variation
The natural differences between repeated measurements or batches. Food tech example: pack weights vary a little from pack to pack even when the line is running well, which is normal variation.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 5: Predicting Problems: Predictive Analytics in Safety and Quality
Note
Unit 5: Predicting Problems: Predictive Analytics in Safety and Quality
A question to start with
A dairy plant tests every batch of milk for microbes. The tests take time, and the milk may be packed before the result arrives. One day the quality manager looks at the records differently: she takes the batches that failed in the past year and asks what they had in common before they failed. The failed batches tended to arrive warmer, to wait longer before cooling, and to come from a small set of collection routes. None of those facts alone proves a batch will fail. Together, they give a warning score: a batch that arrives warm, waits long, and comes from a flagged route is far more likely to fail than a batch that arrives cold and is cooled at once. The plant starts holding the high-score batches for a fast test before packing, and the number of failed packs reaching the shelf falls. No new test was invented. The past was used to warn about the future.
That is predictive analytics: using past records to estimate what is likely to happen next, so the team can act before the problem, not after it. This unit explains how predictions are built, how they are tested, where they work well in food safety and quality, and where they must not be trusted. The rule that runs through the unit is plain: a prediction is a probability, not a promise, and it must be checked on new data before it is used to make decisions.
Questions to think about
- A model says a batch has a high chance of failing. The batch then passes its test. Was the model wrong?
- A prediction works well on last year's data but badly on this year's data. What may have changed?
- If a model can predict spoilage from temperature history, should the plant stop testing the milk in the laboratory?
Think about the dairy story before you read on
A high chance of failing is not a promise of failing. If the model says eight in ten such batches fail, then two in ten pass, and a passing batch is expected, not a proof that the model is wrong. A model that works on last year's data and fails this year may face a new season, a new supplier, a new recipe, or a sensor that drifted, so the past no longer describes the present. A good prediction does not replace laboratory testing: it tells the team which batches to test first, which to hold, and which to watch, while the laboratory result still gives the evidence for the final decision.
What a prediction is, and what it is not
A prediction is an estimate of what is likely to happen, based on patterns in past records. It answers questions such as: how likely is this batch to fail, how many days of shelf life remain, which line is most likely to stop tomorrow, which supplier is most likely to deliver out-of-specification material next month? Each answer comes with a level of confidence: a chance, a range, or a score. A prediction without a confidence level is incomplete, because the team cannot tell whether it is a near certainty or a weak guess.
A prediction is not a measurement. A thermometer measures the temperature now. A model estimates the temperature tomorrow, or the chance that a batch will fail. A prediction is not a cause either: a model may find that warm arrival predicts failure, but the cause may be slow cooling, dirty equipment, or a long journey, and the fix depends on which cause is real. A prediction is also not a decision: it informs the person who decides, and the person still weighs the cost of acting against the cost of waiting.
The most useful predictions in food work are modest ones. They do not claim to foresee everything. They claim that batches with certain past patterns fail more often, that a rising temperature trend usually ends in a breach, or that a drift in a measurement usually precedes a quality fault. Modest predictions, used with care, save time and prevent harm. Grand predictions, trusted blindly, cause both false alarms and missed failures.
Quick self-check
Question: A model gives a batch a failure score of 8 out of 10. What should the team do?
Treat the batch as high risk: hold it, test it first, or check its history in person. The score is a warning, not a verdict. The final decision still waits for the test result and the checks.
Question: Why must every prediction carry a confidence level?
Because a prediction with a high chance and a prediction with a low chance need different responses. A near-certain warning justifies stopping a line. A weak warning justifies a closer look. Without the confidence level, the team cannot tell the two apart.
The inputs: features that carry the signal
A predictive model is built from inputs, also called features: the facts about each batch that the model is allowed to use. Good features are the facts that carry a signal about the outcome. For milk, the features might be the arrival temperature, the waiting time before cooling, the collection route, the season, the age of the equipment, and the last cleaning date. For biscuits, the features might be the flour moisture, the oven temperature profile, the mixing time, and the dough temperature. For a cold store, the features might be the temperature over the last day, the door openings, and the outside temperature.
Choosing features is a food science job before it is a computing job. The team lists the facts that could plausibly affect the outcome, checks that each one is recorded reliably, and leaves out facts that are not recorded, not reliable, or not relevant. A feature that is recorded badly will teach the model a wrong lesson. A feature that is a copy of the outcome, such as a laboratory result that is only available after the failure is known, will make the model look perfect on past data and useless in real use, because the feature will not be available when the prediction is needed.
Features should also be available in time. A prediction that needs the final laboratory result cannot warn before the laboratory result arrives. A prediction that needs only the arrival temperature and the waiting time can warn at the moment of arrival, which is when the warning is useful. The test for any feature is: will we know this fact at the moment we need the prediction?
Quick self-check
Question: A model predicts batch failure using the final laboratory result as an input. What is wrong with that?
The laboratory result arrives after the decision point, so the model cannot warn in time. It also copies the outcome it is trying to predict, so it will look accurate on past data and be useless in practice. The model should use only facts known before the decision.
Question: Who should choose the features for a food prediction model?
The food science and quality team, working with the data team. The food team knows which facts can plausibly affect safety and quality. The data team knows which facts are recorded reliably and how to handle them. Neither side can choose well alone.
Training and testing: the split that keeps models honest
A model is built, or trained, on one set of past records, and it must be tested on a different set that it has never seen. The usual way is to split the past records into two parts: a training set, used to build the model, and a test set, kept aside and used only to check it. If the model does well on the training set but badly on the test set, it has memorised the training records instead of learning the pattern, which is called overfitting. A model that overfits is like a student who memorised last year's exam paper and cannot answer a new question.
The test set must resemble the future the model will face: the same products, the same lines, the same seasons, or the model will be tested on a world it was never built for. A stronger check is to test the model on the most recent period, after training it on earlier periods, because that mimics real use: the past predicts the near future. An even stronger check is to run the model live for a few weeks without acting on it, and to compare its warnings with what actually happened. That live check, called a silent run, shows how the model behaves on truly new data.
The team should also check the two kinds of mistakes a model makes. A false alarm says a good batch will fail, and the cost is wasted time, extra testing, and held stock. A missed failure says a bad batch will pass, and the cost is a faulty product reaching the shelf. In food safety, a missed failure is usually the more serious mistake, so the team may set the warning level to accept more false alarms in exchange for fewer missed failures. That trade-off should be decided openly, not hidden inside the model.
The diagram shows the honest building steps in order. The steps point to the diagram: a model tested only on the data it was trained on has not been tested at all, and a silent run is the last check before the model is allowed to influence decisions.
Quick self-check
Question: A model is right on nine out of ten training batches and right on five out of ten test batches. What has happened?
It has overfitted: it memorised the training records instead of learning the pattern, so it fails on new records. The team should simplify the model, use fewer features, or gather more training records.
Question: Why is a missed failure usually worse than a false alarm in food safety?
Because a false alarm costs time and testing, while a missed failure lets a faulty product reach people who eat it. The team may accept more false alarms to catch more real failures, and that choice should be made openly.
Simple models first: scores, rules, and trends
The best first model is often a simple one. A score adds up warning signs: a point for warm arrival, a point for a long wait, a point for a flagged route, and a batch with a high total is held for testing. A rule says: if the temperature has risen for three hours in a row, send an alert. A trend model says: if the moisture of flour deliveries has risen for five deliveries in a row, warn the bakery to adjust the mix. These models are easy to explain, easy to check, and easy to fix when they are wrong.
More complex models, such as decision trees and regression models, can find patterns a simple score misses. A decision tree splits batches into groups by asking a series of yes or no questions: was the arrival warm, was the wait long, was the route flagged, and each path ends in a group with a known failure rate. A regression model draws a line or a curve that links a measurement, such as storage temperature, to an outcome, such as remaining shelf life. These models are still explainable if they are kept small: the team can read the tree or the line and see why the model warns.
The rule is to start simple and add complexity only when the simple model is proved too weak on the test set. A complex model that nobody can explain is a black box, and a black box that warns about food safety will not be trusted by the people who must act on it. Trust needs an explanation: which facts made the model warn, and how strong was each one?
Quick self-check
Question: Why start with a simple score instead of a complex model?
Because a simple score is easy to explain, easy to check, and easy to fix. If it works well enough, there is no need for complexity. If it fails on the test set, the team has a baseline to beat, and it knows exactly what the complex model must improve.
Question: A decision tree asks a series of yes or no questions about a batch. What does each path show?
Each path leads to a group of past batches that shared those answers, and the failure rate of that group. A batch that follows a path to a high-failure group earns a warning, and the team can read the path to see why.
Shelf life and spoilage: predicting time, not only failure
One of the most useful predictions in food is not pass or fail, but time: how many days of shelf life remain. Microbes grow faster when food is warmer, and slower when it is colder. If the team knows the temperature history of a batch, it can estimate how much of its keeping life has been used up. A batch that spent a day warm has used more of its life than a batch kept cold throughout, even if both were made on the same day. That estimate lets the team decide which batches to sell first, which to move to a colder store, and which to check first.
Spoilage prediction also uses the chemistry of the food. As fruit respires, it uses oxygen and gives out CO₂ and H₂O vapour. As fats break down, acidity rises and flavour changes. As milk ages, its acidity rises. Models that link temperature history, gas readings, and acidity to the remaining life give a forward view that a single test on one day cannot give. The prediction must still be checked against real shelf life tests: batches kept under the predicted conditions and tested until they fail, so the model learns the true end of life, not only a guess.
A prediction of remaining life is also a planning tool. If the model says a set of batches will reach the end of their life before they can be sold, the team can act early: discount, redirect, or repack, instead of discovering the problem on the day the date mark passes. The prediction turns a surprise into a plan.
Quick self-check
Question: Two batches of milk were made on the same day. One spent a day warm, the other stayed cold. Which has more shelf life left?
The cold one. Warmth speeds the growth of microbes and the chemical changes that end shelf life, so the warm batch has used more of its life even though both share the same production date.
Question: Why must a shelf life model be checked against real shelf life tests?
Because the model estimates the end of life from temperature and chemistry, and only a real test, keeping batches until they actually fail, shows whether the estimate is true. Without that check, the model may promise more life than the food really has.
Predicting equipment and process faults
Food safety and quality also depend on equipment that works. A pasteuriser that runs cool, a sealer that seals weakly, a freezer that cycles warm: each one can harm a batch before anyone notices. Predictive models can watch the equipment itself. A motor that vibrates more than usual, a pump that draws more power than usual, or a temperature that drifts slowly upward are early signs that a machine is heading for a fault. The model learns the normal pattern of each machine from past records and warns when the pattern changes.
Process faults can be predicted the same way. If the dough moisture, the mixing time, and the oven profile together predict a high breakage rate, the team can adjust the mix before the batch is baked, instead of counting broken biscuits after. If the acidity of a fermenting yoghurt is rising more slowly than usual, the team can check the culture and the temperature before the batch is lost. The prediction moves the response from after the fault to before it, which is where it is cheapest and safest.
Equipment prediction needs the same honesty as batch prediction: train on past records, test on unseen records, run silently first, and keep the explanation visible. A warning that says "the motor vibration has risen above its usual band for three days" is actionable. A warning that says only "fault likely" is not, because nobody knows what to check.
Quick self-check
Question: A freezer temperature drifts slowly upward over two weeks, never crossing the limit. Should the team act?
Yes. A slow drift is an early sign of a coming fault, such as a failing seal or a blocked coil. Acting now, by checking and servicing the freezer, is cheaper and safer than waiting for the limit to be crossed and the food inside to be harmed.
Question: A model warns that a machine will fail, but gives no reason. Should the team stop the line?
Not on that warning alone. The team should check the machine in person and look at the readings behind the warning. A warning without an explanation cannot be checked, so it should trigger an inspection, not an automatic stop.
When predictions go wrong
Predictions go wrong in a few known ways, and the team should watch for each one. The world changes: a new supplier, a new recipe, a new season, or a new line makes the past a poor guide to the future, and the model must be retrained on the new world. The data changes: a sensor is replaced, a form is redesigned, a unit is changed, and the model reads the new data as if it were the old data. The model is used for a question it was never built for: a model built to predict breakage is used to predict safety, and it fails because the two outcomes have different causes.
Feedback can also corrupt a prediction. If the team always holds the batches the model flags, those batches are tested and their failures are found, while the unflagged batches are never tested, so the team never learns whether the model missed failures among them. The model's own success hides its mistakes. The fix is to keep testing a sample of unflagged batches, so the missed failures can still be seen and the model can be corrected.
The final safeguard is a person. No prediction should automatically release, hold, or discard a batch without a human check, where safety is at stake. The model advises, the person decides, and the laboratory result, the taste, the smell, and the walk around the line still have their place. A model that is never questioned will one day be wrong with confidence, and confidence without questioning is the most dangerous kind of error.
Quick self-check
Question: A model was trained on winter data and is now used in summer. What is the risk?
The past no longer describes the present: temperatures, raw materials, and storage conditions have changed, so the patterns the model learned may not hold. The team should retrain the model on data that covers the new season, or check it carefully on summer records before trusting it.
Question: Why keep testing a sample of batches the model says are safe?
Because otherwise the team never learns whether the model misses failures among the batches it clears. Testing a sample of cleared batches reveals the missed failures and lets the model be corrected.
Try it yourself
Make a small table of thirty past batches for one product, with columns for arrival temperature (warm or cold), waiting time (long or short), route (flagged or clear), and outcome (pass or fail). Build in a pattern yourself: for example, make warm and long-wait batches fail more often. Now split the table into a training set of twenty batches and a test set of ten. Build a simple score from the training set: give a point for each warning sign, and work out the failure rate for each score. Then apply the score to the test set and see how well it predicts. Did the high-score batches fail more often in the test set too? Write down the two kinds of mistakes your score made: false alarms and missed failures. Which kind would worry you more for a food safety decision, and why?
Key ideas to carry forward
- Predictive analytics uses past records to estimate what is likely to happen next, so the team can act before a problem, not after it.
- A prediction is a probability, not a promise, and it must carry a confidence level.
- Features are the facts the model is allowed to use, and they must be known at the moment the prediction is needed.
- Train on one set of records and test on a different set that the model has never seen: a model that does well only on its training set has memorised, not learned.
- A silent run on new data is the last check before a model is allowed to influence decisions.
- A missed failure is usually more serious than a false alarm in food safety, and the trade-off between them should be decided openly.
- Start with simple models: scores, rules, trends, and small decision trees that can be explained, and add complexity only when the simple model is proved too weak.
- Temperature history can predict remaining shelf life, because warmth uses up keeping life faster than cold.
- Equipment and process faults can be predicted from drift and change in machine readings, and the warning should name what to check.
- Predictions go wrong when the world changes, the data changes, or the model is used for a question it was never built for, so models need retraining, retesting, and a human decision at the end.
Glossary
Decision tree
A model that splits batches into groups by asking a series of yes or no questions, with each path ending in a group with a known outcome rate. Food tech example: a tree asks whether milk arrived warm and whether the wait was long, and the path ends in a group with a known failure rate.
False alarm
A prediction that a good batch will fail, when the batch in fact passes. Food tech example: a model flags a cold, quickly cooled milk batch as high risk, and the batch passes its test.
Feature
One input fact that a model is allowed to use when making a prediction. Food tech example: arrival temperature is a feature used to predict whether a milk batch will fail.
Missed failure
A prediction that a bad batch will pass, when the batch in fact fails. Food tech example: a model clears a warm, long-wait milk batch, and the batch fails its test.
Model
A set of rules or calculations, built from past records, that turns inputs into a prediction. Food tech example: a score that adds warning signs for a milk batch is a simple model.
Overfitting
When a model memorises its training records instead of learning the general pattern, so it fails on new records. Food tech example: a model that is right on its training batches but wrong on new batches has overfitted.
Predictive analytics
Analysis that uses past records to estimate what is likely to happen next. Food tech example: using past failures to estimate the chance that a new milk batch will fail is predictive analytics.
Probability
A number that says how likely an event is, from zero for impossible to one for certain. Food tech example: a failure chance of eight in ten is a probability that a batch will fail.
Regression
A model that draws a line or a curve linking a measurement to an outcome. Food tech example: a regression links storage temperature to remaining shelf life for a juice.
Score
A simple prediction made by adding points for warning signs. Food tech example: a milk batch gets a point for warm arrival and a point for a long wait, and a high total means high risk.
Shelf life
The period during which a food keeps its expected safety and quality under stated storage conditions. Food tech example: a yoghurt keeps its flavour and safety until its shelf life ends, if it is kept cold.
Silent run
Running a model on new data without acting on its warnings, to compare its predictions with what actually happens. Food tech example: a failure model runs silently for three weeks so the team can see how often it is right before using it.
Test set
Past records kept aside and used only to check a model, never to build it. Food tech example: ten batches kept aside to test a failure score built from twenty other batches.
Training set
Past records used to build a model. Food tech example: twenty batches used to work out which warning signs predict failure.
Trend
A steady direction in data over time, such as a rising or falling line. Food tech example: a rising trend in flour moisture over five deliveries warns the bakery to adjust the mix.
Warning score
A number that summarises how many warning signs a batch carries. Food tech example: a batch with a high warning score is held for a fast test before packing.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 6: Dashboards, Alerts and Decisions
Note
Unit 6: Dashboards, Alerts and Decisions
A question to start with
A factory has three screens on the quality office wall. The first shows every number the factory collects: hundreds of readings, all flashing, none of them easy to read. The second shows only three numbers: the cold store temperatures, the batches waiting for test results, and the open alerts. The third shows the same three numbers, but the cold store temperature is two days old. The team uses the second screen every hour. The first screen is ignored because it shows too much. The third screen is dangerous because it looks current and is not. The difference between the three screens is not the data behind them. It is the design: what is shown, what is left out, and whether the viewer can tell if the numbers are fresh.
This unit is about the last stage of the data system: showing the results so people can decide and act. It covers dashboards, alerts, reports, and the decisions they support. The theme is restraint: a good display shows less, but shows it well, and it always shows whether the numbers can be trusted and how fresh they are.
Questions to think about
- A dashboard shows fifty numbers on one screen. Why might a team with fifty numbers decide worse than a team with five?
- An alert sounds every hour for a small temperature wobble. What will the team do after a week of hourly alerts, and what is the risk?
- A report says a batch failed, but the report is dated last week. What should the reader check before acting on it?
Think about the three screens before you read on
Fifty numbers overload the viewer: the eye cannot rank them, so the important one hides among the rest. Five well-chosen numbers can be read at a glance and acted on. An alert that sounds every hour for a small wobble will soon be ignored, and then a real alert will be ignored with it, which is called alert fatigue. A report dated last week may describe a batch that has already been handled, so the reader should check the date, the batch number, and whether a newer report exists before acting.
What a dashboard is for
A dashboard is a screen that shows the most important numbers and charts together, updated often, so a team can see the state of a process at a glance. Its job is not to show everything. Its job is to answer the questions the team asks every day: is anything out of limits right now, what needs attention first, and is the process steady or drifting? A dashboard that answers those three questions is doing its job. A dashboard that shows every measurement ever taken is a data dump, not a dashboard.
Different people need different dashboards. The person on the line needs the here and now: the current temperatures, the current line speed, the open alerts, and the batches running now. The quality manager needs the week: the failure rate, the supplier comparison, the overdue tests, and the open investigations. The senior manager needs the month: the trends, the costs of waste and rework, and the progress on fixes. One screen cannot serve all three, so a good system has a few dashboards, each built for one audience and one set of questions.
A dashboard also needs a clear owner. Someone must check that the numbers are fresh, that the links work, and that the charts still match the process. A dashboard with no owner slowly goes stale: a sensor is moved, a product is renamed, a limit is changed, and the screen keeps showing the old picture. The owner checks the dashboard against reality on a regular schedule and fixes it when the two drift apart.
Quick self-check
Question: Why should the line operator and the senior manager have different dashboards?
Because they ask different questions. The operator needs the current state to act now. The senior manager needs the trend over a month to decide where to invest and what to fix. One screen cannot answer both sets of questions well.
Question: A dashboard shows a temperature from two days ago as if it were current. What is the danger?
The team may believe the store is cold when it is in fact warm, and take no action. A stale number that looks fresh is worse than no number, because it gives false confidence. Every dashboard must show how fresh its numbers are.
Designing a dashboard people will actually use
Good dashboard design follows a few plain rules. First, put the most important thing at the top, in the largest type: the number that decides the next action. Second, show a small number of items: five to seven is a good range for one screen, because the eye can hold that many at once. Third, use the same place for the same thing every time, so the team learns where to look without thinking. Fourth, show the limit beside the number, so the reader can see at once whether the value is inside or outside its band. Fifth, show the time: when the number was last updated, and what period it covers.
Colour should be used sparingly and consistently: one colour for normal, one for warning, and one for a breach, used the same way on every screen. Too many colours turn the screen into noise. Text should be plain: short labels, the unit beside every number, and no abbreviations that only one person understands. A chart should be the simplest one that answers the question: a line for a trend, a bar for a comparison, a single number for a count.
The dashboard should also be tested with its real users. Watch a person use it for five minutes: can they find the open alerts, can they tell which store is warm, can they say when the numbers were last updated? If they cannot, the design needs changing, however good it looks to the person who built it. A dashboard is finished when its users can answer their daily questions from it without help.
Quick self-check
Question: Why put the limit beside the number on a dashboard?
So the reader can judge the number at once, without remembering the limit or looking it up elsewhere. A temperature of 6 degrees means little on its own. A temperature of 6 degrees beside a limit of 5 degrees says at once that the store is too warm.
Question: A dashboard uses red for normal and green for a breach on one screen, and the opposite on another. What is wrong?
The colours are inconsistent, so the reader must relearn the meaning on each screen, and will one day read a breach as normal. Colours must mean the same thing on every screen in the system.
Alerts: when to shout, when to whisper, when to stay silent
An alert is a message that something needs attention now. Good alerts have four parts: what happened, where it happened, how serious it is, and what to do first. "Cold store 2 is at 7 degrees, above its limit of 5 degrees, check the door and the cooling unit now" is a good alert. "Temperature problem" is not, because it says nothing about where, how bad, or what to do.
Alerts should be graded by seriousness. A breach that can harm food needs an immediate alert to the person on duty, by sound and by message. A drift that is moving towards a limit needs a warning that can wait until the next check, shown on the dashboard but not sounded aloud. A small wobble inside the normal band needs no alert at all: it is normal variation, and alerting on it trains the team to ignore alerts. The grading should be written down and agreed with the people who receive the alerts, so everyone knows what each level means.
Every alert should also be tracked: when it fired, who received it, what they did, and how long it took to fix. That record turns alerts into data of their own: which alerts fire most often, which are ignored, which take longest to fix, and which never needed to fire at all. A monthly review of the alert record is how a noisy system becomes a quiet, trusted one.
The diagram shows the three alert levels in order of seriousness. The steps point to the diagram: alerting on normal variation is the fastest way to make a team ignore real alerts, so the third level, staying silent, is as important as the first.
Quick self-check
Question: A sensor wobbles inside its normal band and an alert sounds every hour. What should the team change?
The alert level: a wobble inside the band is normal variation and should not alert at all. The team should raise the alert point to the real limit, or to a drift warning, and let the chart show the wobble without sounding.
Question: What should a good alert message contain?
What happened, where it happened, how serious it is, and what to do first. A message with those four parts lets the receiver act at once, without searching for the missing facts.
Alert fatigue and how to prevent it
Alert fatigue is what happens when alerts fire so often that people stop hearing them. It is one of the most serious risks in a data system, because a team that has learned to ignore alerts will ignore the one alert that matters. The causes are familiar: limits set too tight, so normal variation alerts; sensors that drift, so false readings alert; alerts that repeat while a problem is already being fixed; and alerts sent to everyone, so nobody feels responsible.
Prevention starts with the limits. A limit should be set from the food science and the process: the temperature that keeps the food safe, with a warning band before it, not a tight band around the usual value that normal variation will cross. Prevention continues with the sensors: calibrated, placed well, and checked, so false readings are rare. It continues with the logic: an alert should fire once for a problem, then stay quiet while the problem is being fixed, with a reminder only if the problem is still open after a set time. And it continues with the routing: each alert goes to the person on duty for that area, not to everyone in the business.
The test for alert fatigue is simple: ask the team how many alerts they received last week, how many they acted on, and how many they ignored. If the ignored share is large, the system is too noisy and must be quietened, even if that means accepting that a few small warnings will be missed. A quiet system that is trusted beats a noisy system that is ignored.
Quick self-check
Question: An alert fires every ten minutes while a cold store is being repaired. What should the system do instead?
Fire once, then stay quiet while the repair is in progress, with a single reminder if the problem is still open after a set time. Repeated alerts for a known, open problem add noise and teach the team to ignore the sound.
Question: Why should an alert go to one named person on duty, not to everyone?
Because when an alert goes to everyone, each person assumes someone else will act, and often nobody does. A named person on duty knows the alert is theirs to answer.
Reports: the record that outlives the screen
Dashboards and alerts serve the present. Reports serve the record: they say what happened over a week or a month, what was done about it, and what is still open. A good report is short, dated, and honest about its limits. It names the period, the products, and the number of records behind each figure. It shows the trends, the top faults, the overdue actions, and the decisions taken. It says what is missing: which records were incomplete, which tests are still waiting, and which questions the data could not answer.
A report should be written for a reader who was not in the room. That means the words must be plain, the numbers must carry their units, and the charts must carry their titles and their counts. A report that says "failures fell" without saying from what to what, over which period, and on how many batches, says nothing that can be checked. A report that says "the failure rate fell from one in ten batches to one in twenty, over four weeks, on two hundred batches" can be checked, argued with, and acted on.
Reports also need a home and a habit. A report that is written and never read is wasted effort. A report that is read in a regular meeting, where actions are assigned and dates are set, becomes the engine of improvement. The meeting should ask three questions of every report: what changed, why did it change, and what will we do about it before the next report?
Quick self-check
Question: A report says "quality improved". Why is that not good enough?
Because it gives no number, no period, and no count. The reader cannot tell how much it improved, over what time, or on how many batches. A good report gives the figures, the period, and the count, so the claim can be checked.
Question: Why should a report say what is missing?
Because missing records limit what the report can claim. If half the dates are missing, the report cannot show a trend over time, and saying so stops the reader from drawing a conclusion the data cannot support.
From numbers to decisions: a simple decision routine
Data supports decisions, but it does not make them. A simple routine helps the team turn a dashboard or a report into an action. First, name the decision: what must be decided, and by when? Second, name the options: hold the batch or release it, stop the line or keep running, call the supplier or wait, fix the seal now or at the next shutdown. Third, name the evidence: what do the numbers say, how fresh are they, and what is missing? Fourth, name the costs: what does each option cost if it is right, and what does it cost if it is wrong? Fifth, decide, act, and write down who decided, what was decided, and when it will be reviewed.
That routine protects the team in two ways. It stops a decision being made on one number alone, because the evidence step asks what else is known. And it stops a decision from being forgotten, because the review step asks whether the decision worked. A decision that is never reviewed is a decision that is never learned from. A batch held for testing should be followed: did the test confirm the worry, or was the hold unnecessary? A line stopped for a fault should be followed: did the stop prevent more failures, or did it cost a day for nothing? The answers feed the next decision, and the system improves.
Decisions also need to respect the food safety plan and the law of the land in general terms: where a rule sets a limit or a required action, the data supports the action, it does not replace the rule. A dashboard cannot overrule a safety limit. A model cannot release a batch that the plan says must be held. The data system works inside the safety system, not above it.
Quick self-check
Question: A dashboard shows a cold store at 7 degrees. What is the first step in the decision routine?
Name the decision: what must be decided now, such as whether to move the food, check the store, or hold the stock. Then gather the evidence: is the reading fresh, does a second sensor agree, is the door shut? Only then choose among the options.
Question: Why write down who decided and when it will be reviewed?
So the decision can be checked later and learned from. If the decision worked, the team knows to do the same again. If it did not, the team knows what to change. Without the note, the lesson is lost.
Freshness, honesty, and the small print on every screen
Every dashboard, alert, and report should carry its small print: when the data was last updated, what period it covers, how many records it is based on, and what is missing. That small print is what makes the display honest. A number without a time is a number without a context. A chart without a count can be based on three records or three thousand. A report without a note on missing data can claim more than the data supports.
The team should also check the displays against reality on a regular schedule. Walk to the cold store and compare the dashboard temperature with a hand-held thermometer. Count the open alerts on the screen and compare them with the problems the team knows about. Check that the batch numbers on the dashboard match the batch numbers on the line. These reality checks catch the silent failures: the sensor that stopped sending, the feed that stopped updating, the dashboard that kept showing yesterday's numbers as if they were today's.
A final habit is to keep the displays simple over time. Each new request for "one more number on the screen" should be tested: which question does it answer, who will act on it, and what will be removed to make room? A dashboard that grows without limit becomes the noisy screen nobody uses. A dashboard that is pruned regularly stays useful.
Quick self-check
Question: A dashboard number has no time beside it. Why is that a problem?
Because the reader cannot tell whether the number is from now, this morning, or last week. A stale number can look current and lead to a wrong decision, so every number needs its time shown.
Question: Why walk to the cold store and check the dashboard against a hand-held thermometer?
Because the dashboard can fail silently: a sensor stops sending, a feed stops updating, and the screen keeps showing old numbers. A physical check catches the failure that the screen itself cannot show.
Try it yourself
Design a dashboard for one product on one page of paper. Choose five numbers or charts only: the ones that answer the questions the team asks every day. For each one, write its label, its unit, its limit, and how often it updates. Then write two alert messages for that product: one breach alert and one drift warning, each with what happened, where, how serious it is, and what to do first. Show your page to a classmate and ask them to answer three questions from it without your help: is anything out of limits, what needs attention first, and when were the numbers last updated? If they cannot answer, redesign the page until they can.
Key ideas to carry forward
- A dashboard shows the most important numbers and charts together, updated often, so a team can see the state of a process at a glance.
- Different people need different dashboards: the line needs the here and now, the quality manager needs the week, the senior manager needs the month.
- Good design puts the most important thing at the top, shows five to seven items, uses the same place for the same thing, shows the limit beside the number, and shows the time.
- A good alert says what happened, where it happened, how serious it is, and what to do first.
- Alerts should be graded: breach, drift, and normal variation, and normal variation should stay silent.
- Alert fatigue happens when alerts fire too often, and a team that has learned to ignore alerts will ignore the one that matters.
- Reports serve the record: they are short, dated, honest about what is missing, and read in a regular meeting where actions are assigned.
- A simple decision routine names the decision, the options, the evidence, the costs, and the review date, and it keeps the decision inside the food safety plan.
- Every display must carry its small print: when it was updated, what period it covers, how many records it is based on, and what is missing.
- Displays must be checked against reality on a regular schedule, and pruned so they stay simple and useful.
Glossary
Alert
A message that something needs attention now. Food tech example: an alert says cold store 2 is above its limit and names what to check first.
Alert fatigue
The state where alerts fire so often that people stop hearing them, including the ones that matter. Food tech example: hourly alerts for a small wobble inside the normal band make the team ignore a real breach alert.
Breach
A value outside its set limit. Food tech example: a cold store temperature above its limit is a breach that needs immediate action.
Dashboard
A screen that shows the most important numbers and charts together, updated often, so a team can see the state of a process at a glance. Food tech example: a dashboard shows cold store temperatures, open alerts, and today's test results on one screen.
Decision routine
A set order for making a decision: name the decision, the options, the evidence, the costs, then decide, act, and review. Food tech example: the team uses the routine to decide whether to hold a warm milk batch.
Drift
A slow, steady movement of a measurement towards or away from its usual level. Food tech example: a freezer temperature drifting slowly upward over two weeks is a drift that warns of a coming fault.
Freshness
How recently data was updated, shown beside the number so the reader knows whether it is current. Food tech example: a dashboard shows that a temperature reading is five minutes old.
Limit
The boundary a measurement must stay inside. Food tech example: the limit for a cold store is the highest temperature the food in it may reach.
Normal variation
The small, expected ups and downs of a process running well. Food tech example: a cold store temperature wobbling a little inside its band is normal variation.
Report
A written summary of what happened over a period, what was done, and what is still open. Food tech example: a weekly quality report lists the failure rate, the top faults, and the overdue actions.
Review
Looking back at a decision or a period to see whether it worked and what to change. Food tech example: the team reviews last week's alerts to see which ones were ignored and why.
Routing
Sending each alert to the person on duty for that area. Food tech example: a cold store alert goes to the person on duty for storage, not to everyone in the business.
Warning band
A range before a limit that warns the team a breach is coming. Food tech example: a cold store warning band starts below the limit, so the team can act before the limit is crossed.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.
Unit 7: A Worked Case: Tracing a Quality Problem with Data
Note
Unit 7: A Worked Case: Tracing a Quality Problem with Data
A question to start with
A biscuit factory receives a cluster of complaints in one week. Customers say the biscuits taste stale and break easily. The complaints name the same product, and most of them give a batch number from the pack. The quality officer has a choice: she can guess, or she can trace. She chooses to trace, and this unit follows her step by step, using every idea from Units 1 to 6. The case is a teaching case: the numbers are simple, the pattern is clear, and the steps are the ones a real team would follow. By the end, the officer knows which batches are affected, what caused the fault, what to fix, and how to stop it from returning.
The product is a plain biscuit, made on two lines, from flour delivered weekly, baked, cooled, packed, stored, and sold through shops. The complaint says two things: stale taste and easy breakage. Those two faults may share one cause, or they may have two causes. The trace must keep both possibilities open until the data says otherwise.
Questions to think about
- The complaints name a batch number. What is the first thing you would do with those batch numbers?
- The laboratory tests for the complained batches pass. Does that close the case?
- You find that the faulty batches share one flour delivery. What would you check next, before blaming the flour?
Think about the case before you read on
The first step is to list the complained batch numbers and pull every record for those batches: production, laboratory, storage, and transport, joined by batch number. Passing laboratory tests do not close the case: the laboratory may test for the wrong fault, the sample may not represent the complained packs, or the fault may have grown during storage after the test. A shared flour delivery is a lead, not a verdict: the next step is to check whether other batches from the same delivery are fine, whether the delivery was stored well, and whether another factor, such as the baking or the storage time, lines up with the same batches.
Step 1: Gather the complaints and make them data
The officer starts with the complaints themselves. There are a dozen of them, arriving over a week, by phone, email, and a shop comment card. Each one is read and turned into a row in a table: the date of the complaint, the product name, the batch number from the pack, the shop, the purchase date where known, the fault described in the customer's words, and how the biscuit was stored at home where the customer says. Two complaints have no batch number, and one has a batch number that does not match any production record. Those three are set aside with a note, not thrown away: they may be misread numbers, or they may belong to a batch the factory cannot trace, and either way the gap is recorded.
The first picture is simple: a count of complaints by batch. Three batches carry most of the complaints. The other batches carry one each, which may be the normal background of complaints that any product receives. The officer notes the dates too: the complaints arrive over a week, but the purchase dates cluster earlier, which suggests the fault was in the biscuits when they were sold, not something that happened in the customers' homes over a long time.
Quick self-check
Question: Why turn each complaint into a row in a table instead of reading them as a pile of messages?
Because a table can be counted, sorted, and joined to other records. A pile of messages can only be read one by one. The table shows at once which batches carry most complaints, which is the first lead in the trace.
Question: A complaint has a batch number that matches no production record. What should you do with it?
Set it aside with a note, and check whether the number was misread or mistyped. Do not force it to match a real batch by guessing, and do not delete it silently: the gap may matter if more such complaints arrive.
Step 2: Pull every record for the suspect batches
With the suspect batch numbers in hand, the officer pulls every record for those batches from each table: the production table gives the line, the date, the mixing time, the oven temperature profile, and the cooling time; the laboratory table gives the moisture, the fat, the acidity, and the breakage test results; the delivery table gives the flour supplier, the delivery date, and the arrival moisture; the storage table gives the store temperature and the days in store; the transport table gives the delivery route and the time on the road; the sales table gives which shops received each batch and when.
The pull is a join on the batch number, the skill from Unit 3. The officer counts the matches: every suspect batch finds its production and laboratory rows, but two batches find no storage row, because the store log for one week was kept on paper and never typed in. That gap is noted, and the paper log is fetched and typed in before the analysis goes on, because storage time is one of the possible causes of staleness.
Quick self-check
Question: Why fetch the paper store log and type it in, instead of analysing without it?
Because storage time and temperature are possible causes of staleness, and without those records the team cannot test that cause. Analysing without them would leave a leading suspect unexamined.
Question: What does the join count tell the officer?
How many batches found a match in each table, and which ones did not. A batch with no laboratory row may never have been tested, and a batch with no storage row has a gap in its story. Both need explaining before the analysis is trusted.
Step 3: Describe what happened
The officer now describes the suspect batches against the rest. The descriptive step, from Unit 4, is a set of counts and charts. The failure rate for the three suspect batches is high: most of their packs drew complaints, while the other batches from the same month drew almost none. The moisture results for the suspect batches sit at the high end of the range, while the other batches cluster lower. The breakage test results for the suspect batches are worse than the rest, with a wider spread.
A bar chart by line shows no clear difference: both lines made suspect batches. A bar chart by shift shows no clear difference either. A bar chart by flour delivery shows a sharp difference: the suspect batches all used flour from one delivery, and the other batches from the same delivery, baked on different days, also show higher moisture and more breakage, though they drew fewer complaints. The picture narrows: the flour delivery is now the leading suspect, and the line and the shift are ruled out.
Quick self-check
Question: The chart by line shows no difference. What does that tell the officer?
That the line is probably not the cause. A cause should show up in the split that matches it, and the line split shows nothing, so the officer can set the line aside and follow the flour lead.
Question: Why does it matter that other batches from the same flour delivery also show higher moisture?
Because it supports the flour lead with more than the complained batches alone. If only the complained batches showed the pattern, the cause could be something unique to them. When the whole delivery shows it, the delivery itself is implicated.
Step 4: Clean, check, and test the lead
Before acting on the flour lead, the officer cleans and checks the records behind it, using Unit 3. The moisture results for the suspect delivery are checked against the original laboratory sheets: the values are real, not typing errors, and the method and the basis are the same as for other deliveries. The delivery note is checked: the flour arrived on a rainy day, the load was uncovered for part of the unloading, and the arrival moisture was already at the high end. The storage log is checked: the flour from that delivery was stored near a door that lets in damp air, while other deliveries were stored deeper in the store.
The officer also tests the rival causes. Baking: the oven profiles for the suspect batches are normal, and the oven charts show no dip. Cooling: the cooling times are normal. Storage of the finished biscuits: the store temperatures are normal, and the days in store for the suspect batches are no longer than for other batches. Transport: the routes and times are normal. Each rival cause is checked against the data and set aside, not by opinion but by record. The flour delivery, wet on arrival and stored in a damp place, remains the best fit for the evidence.
A scatter plot supports the story: across all batches, higher flour moisture goes with higher biscuit moisture and more breakage. The dots form a rising band, not a cloud. The correlation is not proof on its own, but together with the delivery note, the storage log, and the ruling out of the rivals, it makes a strong case.
The diagram shows the trace in order. The steps point to the diagram: skipping the cleaning and the rival-cause checks would let a plausible lead become a hasty verdict, and a hasty verdict fixes the wrong thing.
Quick self-check
Question: Why check the original laboratory sheets before acting on the moisture lead?
Because a typing error or a different method or basis could make the moisture look high when it is not. Checking the original sheets proves the values are real and comparable before the team acts on them.
Question: The scatter plot shows higher flour moisture going with more breakage. Does that prove the flour caused the breakage?
Not alone. It is a correlation, and a third factor could cause both. It becomes a strong case when it is joined with the delivery note, the storage log, and the ruling out of the oven, cooling, storage, and transport as causes.
Step 5: Decide, act, and follow up
The officer writes the decision in the routine from Unit 6. The decision is: hold the remaining stock of the suspect batches, tell the shops that received them, and do not use the remaining flour from that delivery until it is tested and dried or set aside. The evidence is the complaint table, the moisture results, the delivery note, and the storage log. The options considered are to hold only the complained batches, to hold all batches from the delivery, or to wait for more tests; the wider hold is chosen because other batches from the same delivery also show higher moisture. The cost of the wider hold is more stock held, and the cost of the narrower hold is more faulty biscuits reaching customers.
The fix has two parts. The short fix is for this delivery: cover loads in rain, test arrival moisture before unloading, and store flour away from damp doors. The long fix is for the system: add a moisture check on arrival to the delivery checklist, add a dashboard alert for arrival moisture above the usual band, and add a rule that flour stored near the door is used first or moved. Each fix gets an owner and a date, and the next month of batches is watched on the dashboard: flour moisture, biscuit moisture, breakage, and complaints, all by delivery.
The follow-up is what turns a case into a lesson. After a month, the officer checks whether the new checks are being done, whether the alerts fire only when they should, and whether the complaint rate for new batches has fallen. The case is written up as a short report: what happened, what the data showed, what was decided, what was fixed, and what is still open. The report goes to the production team, the laboratory team, and the managers, so the next person who sees a cluster of stale-biscuit complaints starts from the trace, not from a guess.
Quick self-check
Question: Why hold all batches from the suspect delivery, not only the batches customers complained about?
Because other batches from the same delivery also show higher moisture and more breakage, even though they drew fewer complaints. Holding only the complained batches would leave faulty biscuits from the same delivery on the shelf.
Question: What is the difference between the short fix and the long fix?
The short fix handles this delivery: cover loads, test on arrival, move the flour. The long fix changes the system: a new checklist step, a dashboard alert, and a storage rule, so the same fault cannot return quietly next time.
What the case teaches about the whole subject
The case uses every unit in this subject, and it is worth naming the links. Unit 1 gave the idea: the story was in the joined records, not in any single one. Unit 2 gave the sources: complaints, production logs, laboratory results, delivery notes, storage logs, transport records, and sales records, each from its own place. Unit 3 gave the cleaning and the joining: the paper store log typed in, the moisture values checked against the original sheets, the batch number as the key. Unit 4 gave the description and the diagnosis: counts, rates, charts, splits by line and by delivery, the scatter plot, and the ruling out of rival causes. Unit 5 gives the forward look the team now adds: a simple score for new flour deliveries, so a wet delivery earns a warning at arrival, before it is baked. Unit 6 gives the display and the decision: a dashboard that shows flour moisture by delivery, an alert for a high arrival reading, and a written decision with an owner and a review date.
The case also shows the limits. The trace depended on batch numbers on the packs: the two complaints without a batch number could not be traced, and the team should make the batch number easier to find and to report. The trace depended on the paper store log being typed in: a record kept only on paper is a record that can be missed. The trace depended on people: the driver who noted the rain, the store keeper who knew the damp door, the officer who asked why. Data did not replace those people. It gave them a shared picture to work from.
Quick self-check
Question: Which unit gave the team the batch number as the link between records?
Units 2 and 3: Unit 2 named the batch number as the link that joins records from different places, and Unit 3 showed how to join tables on that key and how to check the join.
Question: How does Unit 5 change what happens after this case?
It adds a forward look: a simple score for new flour deliveries, so a wet delivery earns a warning at arrival, before it is baked into biscuits. The team moves from tracing after complaints to warning before the fault.
A second, shorter trace: the warm cold store
A second trace shows the same steps on a smaller problem. A cold store temperature chart shows a warm spell each afternoon. The officer pulls the sensor readings, the door log, and the delivery schedule. The warm spell lines up with the delivery hour, when the door stands open while pallets are unloaded. The sensor by the door shows the warmest readings, while the sensor in the middle shows a smaller rise. The fix is simple: a door closer, a curtain, and a rule to unload in smaller loads with the door shut between them. The dashboard gains a drift warning for the afternoon, and the next month's chart shows the warm spell gone. The case is small, but the steps are the same: gather, join, describe, split, check, decide, act, and follow up.
The two cases together show the range of the subject. The biscuit case needed many tables and many checks. The cold store case needed three records and one chart. Both used the same discipline: start from the question, join the records, describe before diagnosing, check before acting, and write down what was decided. That discipline is the subject, more than any single tool or chart.
Quick self-check
Question: In the cold store case, why did the sensor by the door matter more than the sensor in the middle?
Because the warm air entered by the door, so the door sensor measured the place at risk. The middle sensor showed a smaller rise and would have understated the problem for the food near the door.
Try it yourself
Write your own worked case in five steps, using the diagram above. Choose a product and a fault: stale bread, sour milk, soft biscuits, or faded juice colour. Invent a small set of records on paper: six complaints with batch numbers, a production table for ten batches, a laboratory table, a delivery note, and a storage log. Build in one cause, such as a warm delivery, a long storage time, or a missed cleaning step, and build in one rival cause that looks suspicious but is innocent. Then trace the case as the officer did: gather, join, describe, split, check the rivals, decide, and write the fix with an owner and a review date. Swap your case with a classmate and trace theirs: can you find the cause from their records alone, and did they give you the records you needed?
Key ideas to carry forward
- A trace starts with the complaints as data: each complaint becomes a row with a batch number, a date, a shop, and the fault in the customer's words.
- Pull every record for the suspect batches and join them on the batch number, counting the matches and fetching any paper records that are missing.
- Describe before diagnosing: count, chart, and split by line, shift, supplier, and delivery, and let the split that shows the difference point to the lead.
- Clean and check the records behind the lead against the original sheets, and test each rival cause against the data before acting.
- Correlation supports a case when it is joined with delivery notes, storage logs, and the ruling out of rivals, but it does not prove a cause on its own.
- Decide with the routine: name the decision, the options, the evidence, the costs, and the review date, and hold the wider set of affected batches where the evidence points that way.
- Fix twice: a short fix for the batch in hand, and a long fix for the system, with an owner and a date for each.
- Follow up: watch the next batches on the dashboard, check whether the new checks are being done, and write the case up so the next person starts from the trace, not from a guess.
- The whole subject works as one loop: joined records, clean tables, description, diagnosis, prediction, display, decision, and back to collection.
Glossary
Background complaints
The small number of complaints any product receives in normal times, against which a cluster stands out. Food tech example: one complaint a week about biscuits is background, while ten in a week is a cluster.
Cluster
A group of similar complaints or failures arriving close together in time or from the same batches. Food tech example: a cluster of stale-taste complaints about three biscuit batches in one week.
Follow-up
Checking after a fix whether it worked, whether the new checks are being done, and whether the problem has returned. Food tech example: watching the next month of flour moisture and complaints after a new arrival check is follow-up.
Hold
Keeping a batch back from sale until it is checked and a decision is made. Food tech example: the remaining stock of suspect biscuit batches is held while the flour delivery is investigated.
Lead
A possible cause that the data points to and that needs testing. Food tech example: one flour delivery is the lead when all suspect batches used flour from it.
Rival cause
Another possible cause that must be checked and ruled out before a lead is accepted. Food tech example: the oven is a rival cause for broken biscuits until its temperature charts prove it ran normally.
Short fix
A fix that handles the batch or delivery in hand at once. Food tech example: covering flour loads in rain and testing arrival moisture is a short fix.
Long fix
A fix that changes the system so the same fault cannot return quietly. Food tech example: adding a moisture check to the delivery checklist and a dashboard alert is a long fix.
Suspect batch
A batch named in complaints or flagged by data as possibly faulty. Food tech example: the three biscuit batches that carry most of the complaints are the suspect batches.
Trace
Following a batch or a problem back through its records, from the complaint to the cause. Food tech example: tracing a stale biscuit back through production, laboratory, delivery, and storage records.
Traceability
The ability to follow a food, a batch, or a result back through every step it passed through. Food tech example: a batch number lets a biscuit maker trace a pack back to its flour delivery.
Cluster investigation
A structured look at a cluster of complaints or failures to find the shared cause. Food tech example: the biscuit investigation lists the complained batches, pulls their records, and splits them by delivery.
Evidence
Records that support a claim about what happened. Food tech example: the delivery note and the storage log are evidence that a flour delivery arrived wet and was stored in a damp place.
Verdict
The conclusion of an investigation: what the cause was and what will be done. Food tech example: the verdict of the biscuit case is that wet flour, stored damp, caused the stale taste and breakage.
References
- Singh, R., Computer Applications in Food Technology, 1996.
- Irudayaraj, J., Food Processing Operations Modeling, Design, and Analysis, 2002.
- AOAC International, Official Methods of Analysis of AOAC International, 17th edition, 2000.
- Nielsen, S. S., Introduction to the Chemical Analysis of Foods, 1997.