detect anomalies in pills during live image acquisition -凯发k8网页登录

download pretrained network

this example uses a pretrained network that is trained to detect pill anomalies. download the pretrained version of the pill anomaly detector by running the following commands.

trainedpillanomalydetectornet_url = ...
    "https://ssd.mathworks.com/supportfiles/vision/data/trainedfcddpillanomalydetectorspkg.zip";
downloadtrainednetwork(trainedpillanomalydetectornet_url,pwd); 
net = load(fullfile(pwd,"folderforsupportfilesinceptionmodel","trainedpillfcddnet.mat"));
detector = net.detector;

prepare data set for calibration and testing

this example uses the downloaded pill anomaly detector since the images used to train it are similar to the images captured and classified in this example. rather than retraining the existing model from scratch, this example shows how to capture images for calibration and testing. if your workflow requires a visual inspection model for a different application, refer to (computer vision toolbox) for how to train the model from scratch.

set up image acquisition hardware

connect to your camera using the videoinput function and specify its properties. this example uses the os generic video interface to acquire images from a logitech® webcam with the yuv2_352x288 video format.

vid = videoinput("winvideo","1","yuy2_352x288");
vid.returnedcolorspace = 'rgb';
vid.framespertrigger = 70;

get the source properties of the videoinput object. specify device-specific properties to adjust for optimal image data collection. the values you specify for these properties depend on the camera hardware and the surrounding environment that images are being captured in.

src = getselectedsource(vid);

acquire data for calibration and testing

after setting up and configuring your image acquisition hardware, you can collect images to calibrate and test the anomaly detection model. collect data in immediate acquisition mode. for more information about trigger types, see .

start(vid);
wait(vid);

after you are finished capturing images, save the acquired image frames to matlab workspace and save the frames as tif files using imwrite.

img = getdata(vid,70);
for f=1:vid.framesacquired
     imwrite(img(:,:,:,f), "image" f ".tif");
end
delete(vid);

in the current working directory create a folder titled imagedata. within this folder create two new folders titled normal and dirt. move the normal and dirt image files into their respective folders.

the data set used in this example contains 40 normal images and 30 dirt images.

these two images show an example image from each class. a normal pill with no defects is on the left and a pill contaminated with dirt is on the right. the properties of these images align closely with the properties of the training data set from the (computer vision toolbox) example. this example demonstrates how pre-trained models can be used after re-calibration and testing with a slightly modified data set.

load and preprocess data

the imagedatastore object manages the collection of image files from both normal and dirt image classes.

imagedir = fullfile(pwd,"imagedata");
imds = imagedatastore(imagedir,includesubfolders=true,labelsource="foldernames");

the images for calibration are used to determine a threshold value for the classifier. the classifier labels images with anomaly scores above the threshold as anomalous. the remaining images are for testing and are used to evaluate the classifier.

normaltrainratio  = 0;
anomalytrainratio = 0;
normalcalratio  = 0.70;
anomalycalratio = 0.60;
normaltestratio  = 1 - (normaltrainratio   normalcalratio);
anomalytestratio = 1 - (anomalytrainratio   anomalycalratio);
rng(0);
[imdstrain,imdscal,imdstest] = splitanomalydata(imds,["dirt"],...
    normallabelsratio=[normaltrainratio normalcalratio normaltestratio],...
    anomalylabelsratio=[anomalytrainratio anomalycalratio anomalytestratio]);

splitting anomaly dataset
-------------------------
* finalizing... done.
* number of files and proportions per class in all the datasets:
                          input                      train                   validation                    test       
              numfiles          ratio          numfiles    ratio    numfiles          ratio          numfiles    ratio
              _____________________________    _________________    _____________________________    _________________
    dirt         30       0.428571428571429       0          0         18       0.391304347826087       12        0.5 
    normal       40       0.571428571428571       0          0         28       0.608695652173913       12        0.5 

add binary labels to normal and dirt images in both calibration and test data sets by using the transform function with the operations specified by the addlabeldata helper function.

dscal = transform(imdscal,@addlabeldata,includeinfo=true);
dstest = transform(imdstest,@addlabeldata,includeinfo=true);

visualize a sample of calibration data that contains both normal and anomaly images.

numobs = length(imdscal.labels);
idx = randperm(numobs,9);
montage(imdscal,indices=idx)

calibrate and test classification model

select anomaly threshold

use the calibration data set to select an anomaly score threshold for the anomaly detector. the detector classifies images based on whether their scores are above or below the threshold value. the calibration data set uses both normal and anomalous images to determine the threshold.

obtain the mean anomaly score and ground truth label for each image in the calibration set and plot a histogram scores.

scores = predict(detector,dscal);
labels = imdscal.labels ~= "normal";
numbins = 20;
[~,edges] = histcounts(scores,numbins);
figure
hold on
hnormal = histogram(scores(labels==0),edges);
hanomaly = histogram(scores(labels==1),edges);
hold off
legend([hnormal,hanomaly],"normal","anomaly")
xlabel("mean anomaly score")
ylabel("counts")

calculate the optimal anomaly threshold using the (computer vision toolbox) function. specify the first two input arguments as the ground truth labels, labels, and predicted anomaly scores, scores, for the calibration data set. specify the third input argument as true because true positive anomaly images have a labels value of true.

thresh = anomalythreshold(labels,scores,true);

set the threshold property of the anomaly detector to the optimal value.

detector.threshold = thresh;

evaluate classification model

at this point, we are interested in evaluating the accuracy of the detector with the test images.

classify each image in the test set as either normal or anomalous using the anomaly detector, detector.

testsetoutputlabels = classify(detector,dstest);

get the ground truth labels of each test image.

testsettargetlabels = dstest.underlyingdatastores{1}.labels;

evaluate the anomaly detector by calculating performance metrics by using the (computer vision toolbox) function. the function calculates several metrics that evaluate the accuracy, precision, sensitivity, and specificity of the detector for the test data set.

metrics = evaluateanomalydetection(testsetoutputlabels,testsettargetlabels,"dirt");  

evaluating anomaly detection results
------------------------------------
* finalizing... done.
* data set metrics:
    globalaccuracy    meanaccuracy    precision    recall    specificity    f1score    falsepositiverate    falsenegativerate
    ______________    ____________    _________    ______    ___________    _______    _________________    _________________
          1                1              1          1            1            1               0                    0        

the confusionmatrix property of metrics contains the confusion matrix for the test set. extract the confusion matrix and display a confusion plot. the recalibrated classification model with the anomaly threshold selected is very accurate and predicts a small percentage of false positives and false negatives.

m = metrics.confusionmatrix{:,:};
confusionchart(m,["normal","anomaly"])
acc = sum(diag(m)) / sum(m,"all");
title("accuracy: " acc)

for more information about explainable classification decisions, see the (computer vision toolbox) example.

save the detector to a mat file for later use in matlab or in deployment workflows.

save('livepilldetector.mat','detector') 

classify live images of pills

you can use a custom video preview from a camera to display the results of the live classification of pills as normal or anomalous. this section demonstrates how to create a custom preview update callback function to process live images, identify individual pills, classify each pill, and show classification results overlayed with the camera preview video. you can find the code from this section as well as the update preview callback function attached to this example as supporting files.

acquire images using a webcam with the videoinput interface at a resolution of 1280x720. then, use a custom preview update callback function to identify each pill using the imfindcircles function, since the pills are circular. crop the section of the camera image around each pill and pass it to the detector for classification. based on the classification decision, display a label on each pill indicating whether it is a normal or anomalous pill. normal pills are labeled as good and anomalous pills are labeled as bad in the preview window.

load the anomaly detector saved previously.

load('livepilldetector.mat');

create a connection to the webcam and specify appropriate values for the device-specific properties. the following device-specific property values are optimized for this hardware setup. your setup might require a different configuration.

vid = videoinput('winvideo',1,'yuy2_1280x720');
vid.returnedcolorspace = "rgb";
nbands = vid.numberofbands;
vidres = vid.roiposition;
imwidth = vidres(3);
imheight = vidres(4);
src = getselectedsource(vid);
src.exposuremode = "manual";
src.exposure = -11;
src.focusmode = "manual";
src.focus = 11;
src.whitebalancemode = "manual";
src.whitebalance = 2800;
src.framerate = "20.0000";

create a figure window and axes in the figure. then create the image object in which you want to display the video preview data. the size of the image object must match the dimensions of the video frames. add a scroll panel to the image.

hfig = uifigure('toolbar','none',...
       'menubar', 'none',...
       'numbertitle','off',...
       'autoresizechildren','off',...
       'name','live pill classification');
haxis = uiaxes(hfig);
disabledefaultinteractivity(haxis);
himage = image(zeros(imheight, imwidth, nbands),'parent',haxis);
imscrollpanel(hfig,himage);

configure the update preview window callback function and the detector as application-defined data on the image object. to define the mypreview_fcn callback function, refer to the perform custom processing of preview data section.

setappdata(himage,'updatepreviewwindowfcn',@mypreview_fcn);
setappdata(himage,'detector',detector);

preview the video data in the custom figure window.

preview(vid,himage);

perform custom processing of preview data

create a callback function mypreview_fcn that processes each frame and identifies whether pills are normal or dirt. this callback function is called for each acquired frame when you call the preview function.

the mypreview_fcn callback function performs the following:

refer to the comments in the code for additional context.

function mypreview_fcn(obj,event,himage)
    persistent detector;
    persistent imgsize;
  
    if isempty(detector) && isempty(imgsize)
        % anomaly detector 
        detector = getappdata(himage,'detector');
        % width and height of images used during calibration
        imgsize = [352 288];
    end
  
    % find circles (pills) in the current image frame
    img = event.data;
    radiusrange = [30 80];
    circles = imfindcircles(img,radiusrange);
    % use classifier if circles are found
    if ~isempty(circles)
        % loop through all circles
        for i = 1:size(circles,1)
            % crop image around the circle (pill)
            % [pos1, pos2] - top-left corner pixel position
            % imgsize - width and height from [pos1, pos2]
            
            pos1 = max(circles(i,1)-imgsize(1)/2,0);
            pos2 = max(circles(i,2)-imgsize(2)/2,0);
            
            croppedimage = imcrop(img,[pos1, pos2,imgsize(1),imgsize(2)]);
            
            if ~isempty(croppedimage)
                % classify cropped image and assign a text label in the
                % center of the circle
                decision = classify(detector, croppedimage);
                if decision
                    img = inserttext(img,circles(i,:),"bad", textcolor="red",fontsize=14);
                else
                    img = inserttext(img,circles(i,:),"good", textcolor="green",fontsize=14);
                end
            end
        end
    end
    himage.cdata = img;
end

this code is expected to classify images correctly if there is one pill in the camera's field of view. if there are multiple pills in the field of view, the distance between the pills must be large enough so that the cropped images do not contain more than one pill.

deploy live image classification as standalone application

you can deploy the live image classification of pills as a standalone application using the application compiler (matlab compiler) app. you must have a license for matlab compiler. once you have created the standalone application, you can deploy it to multiple machines without additional matlab licenses.

you must do the following in the application compiler app to create the standalone application.