Data Collection and Analysis Techniques
A root-cause investigation stalls because three records disagree: production logged a cook temperature at one time, the chart recorder shows another curve, and the laboratory received a sample labelled with a lot code that does not match either. Collection technique, not analysis skill, caused the failure. Good analysis begins long before a chart is drawn, in how data is defined, captured, cleaned, and stored.
Food plants collect data from people, instruments, and machines, each with different error patterns. Knowing those patterns, and the basic techniques for handling them, protects every later decision — release, recall, supplier action, or process change — from resting on numbers that never deserved trust.
Planning collection before collecting
Start from the decision the data must support and work backwards. Define each variable precisely: what is measured, where, with what instrument, in what unit, how often, and linked to which lot, line, and shift. A field called “temperature” is useless until you know whether it means air, product surface, or product core, and at which step.
Sampling design matters as much as instrument choice. Decide how many units, from which points, at which times, and by what selection method. Convenience sampling — testing whatever is nearest — systematically misses the edges of a process: first-off product, end-of-run product, and output after changeovers, where problems concentrate. Stratified sampling, which deliberately covers those strata, gives a truer picture for the same testing cost.
Collection methods and their error patterns
Automatic capture
Sensors, equipment logs, checkweighers, and laboratory instruments feeding systems directly remove transcription errors and give exact timestamps. Risks shift to calibration, sensor placement, clock synchronisation, and silent gaps when a connection drops. Build gap alarms and calibration checks into the method, not as afterthoughts.
Manual and observational collection
Human checks remain essential for hygiene observation, sensory assessment, and conditions no sensor reads. Error patterns include rounding to expected values, completing forms in batches from memory, and observer differences in judging a pass or fail. Counter them with clear criteria and reference samples, point-of-work entry on devices, range checks that reject impossible values, and periodic paired observations to measure agreement between observers.
Laboratory and test data
Record method, analyst, equipment, sample condition, and chain of custody alongside each result. A result without its method cannot be compared across laboratories or over time, because methods differ in sensitivity and in what they detect.
Cleaning and preparing data
Raw data almost always needs work before analysis: duplicate records from re-sent files, mixed units, inconsistent codes for the same supplier or defect, missing values, and obvious entry errors such as a humidity of 900 percent. Clean systematically and keep a record of every rule applied and every record changed or excluded. Silent cleaning destroys defensibility; documented cleaning strengthens it, because another analyst can reproduce your result.
Handle missing data by cause, not by habit. A missing sanitation result because a line did not run is information. A missing result because a test failed and was repeated until it passed is a serious integrity problem. Deleting awkward values or filling gaps with averages without flagging them hides exactly the signals quality data exists to reveal.
Analysis techniques you will use most
- Summaries with variation: averages alongside ranges or standard deviations, so a stable mean cannot hide wild swings.
- Trend and control charts: values over time against expected variation, separating drift and special causes from noise.
- Pareto analysis: ranking defect or complaint types by frequency or cost so effort lands on the few categories that dominate.
- Stratification and comparison: splitting results by line, shift, supplier, or season to locate where a problem actually lives.
- Correlation checks: testing whether two measures move together, treated as a lead for investigation, not proof of cause.
Present results so a busy manager can act: the chart, the sample size, the period, the data faults you know about, and the decision the result supports. State uncertainty plainly. A conclusion that survives honest caveats is worth more than a dramatic one that collapses under the first question in an audit.
Storage, access, and reuse
Collected data outlives the decision it first supported. Store it with its definitions, units, methods, and cleaning history, in formats you can still read when software changes. Name files and fields consistently, keep raw data unchanged, and work on copies for analysis, so any result can be traced back to original observations.
Control access in line with sensitivity: production data may be widely visible inside a plant, while complaint details and personnel-linked observations need tighter handling. Retention should follow your regulatory and customer requirements, with a deliberate decision for each dataset. Data kept forever without organisation becomes a liability; data deleted too early cannot defend a decision or reveal a long trend.
Revisit collection plans at least annually, and whenever a process, product, or supplier changes materially. A sampling point chosen for an old line layout may no longer represent current risk, and a field added for a forgotten project may still consume staff time daily. Pruning obsolete collection is as valuable as adding new measures, because attention is a limited resource.
Key points
- Collection quality decides analysis quality; define each variable, unit, location, and lot link before collecting.
- Stratified sampling covers first-off, end-of-run, and post-changeover product where problems concentrate.
- Automatic capture removes transcription error but adds calibration, placement, and data-gap risks.
- Manual data needs clear criteria, point-of-work entry, and observer agreement checks to stay reliable.
- Clean data with documented rules, keep an audit trail of changes, and treat missing data by its cause.
- Report variation, sample size, and uncertainty with every result so decisions rest on honest evidence.
