Real-time Sensor Fusion for Loss Detection Reference Implementation

Published: 05/04/2020  

Last Updated: 06/08/2021

Overview

Learn how to deploy sensor fusion technology with this reference implementation (RI), which simulates a self-checkout retail use case. The RI assembles sensor data with a microserivce architecture and provides a building block for the integration of various hardware devices, additional services, and computer vision (CV) features.

Select Configure & Download to download the reference implementation and the software listed below.  

Configure & Download


Time to Complete

Programming
Language

Available Software

Approximately 30-45 minutes

Google Go* programming language 1.12+


EdgeX Foundry*
VAS
Docker*
Docker Compose


Other Requirements 
Some software packages require root permissions.


Target System Requirements 

  • Ubuntu* 18.04.3 LTS (64 bit)
  • 6th (and above) Generation Intel® Core™ processors with Iris® Pro Graphics or Intel® HD Graphics
  • Optional (Sensors):
    • Point-of-sale (POS)
    • Barcode scanner
    • Scale under bagging area
    • Four USB cameras
    • RFID mat

Learning Objectives

Use the RI to: 

  • Create an expandable building block that integrates device sensors and services.  
  • Learn how to combine multiple sensors using a microservice architecture in a point-of-sale RI. 
  • Gain insight into EdgeX Foundry and its data flow. 

Expand the RI base components to create your own custom solutions.  

How It Works

The real-time sensor fusion at checkout (RTSF at checkout) RI combines sensor fusion and data assemblage using microservices to illustrate how to recognize a purchase item. Multiple sources of data, such as data from a variety of sensors, can help increase the accuracy of item recognition and, potentially, loss detection. 

The RI uses:

  1. Example Data from Multiple Sensors: Sensor devices trigger events, such as RFID tag recognition, food type detection, or a product entering and exiting a region of interest (ROI). The RI simulates both the events and data in software. 

  1. EdgeX Foundry: The EdgeX bus sends these events back to the main application as an event message. Through the microservices, it assembles the data to determine if the purchase object is a detected product, a reconcilable object. 

  1. Data Reconciliation:The data is then reconciled to make sure that it matches what is being purchased.

Figure 1 illustrates how the data of the RI flows from the physical layer, the sensors devices, to the services and then to EdgeX Foundry. In a production environment, the data would then be reconciled by a comparison to data in a product database, such as an Enterprise Resource Planning (ERP) database.  

Real Time Sensor Fusion at Checkout Architecture Diagram
Figure 1: RTSF at Checkout Architecture Diagram

The extensible framework of EdgeX Foundry enables add-on services and sensors. The software can potentially send POS, CV and RFID ROI, and weight scale simulated events. However, the prototype does not supply sample data for all those devices.

Production Suggestions 

The RTSF at checkout presents an example of a retail use case to demonstrate sensor fusion. To modify it, consider replacing or enhancing parts of the architecture with production-ready services or custom hardware. 

  • Improve the Object Detection Model: The included object detection model, based on the Intel® Distribution of OpenVINO™ toolkit and used by the Video Analytics Service, is an example of an object detection model that was trained with 15 example objects. Upgrade the object detection model for better accuracy by retraining the provided model or replacing it with a model relevant to your use case. 
  • Replace the Enterprise Resource Planning (ERP) System: The provided service is a basic implementation of a Product Information Management Lookup Service. It uses a JSON file as a database for the product information. Replace it with an ERP System, shown in Figure 1. 
  • Integrate Other Devices: Scale Events can be sent to either the EdgeX REST (REpresentational State Transfer) or MQTT (Message Queuing Telemetry Transport) Device Services.
  • Enhance or Replace Services: The RI provides reconciler, detector, product lookup, and RFID services. Consider improving the prototype by replacing these services with more extensive or robust versions of services. 

Create your own custom solution by modifying the base components.  

Get Started

Step 1: Install the Reference Implementation

Select Configure & Download to download the reference implementation and then follow the steps below to install it.

Configure & Download

 

NOTE: If the host system already has Docker images and containers, errors may occur while building the RI packages. Refer to the Troubleshooting: Installation Failure section at the end of this document before starting the RI installation. The Troubleshooting section details how to stop images and containers.

1. Open a new terminal and navigate to the download folder to unzip the RI package:

unzip real_time_sensor_fusion_for_loss_prevention.zip

2. Navigate to the real_time_sensor_fusion_for_loss_prevention/ directory:

cd real_time_sensor_fusion_for_loss_prevention/

3. Change permission of the executable edgesoftware file:

chmod 755 edgesoftware

4. Run the command below to install the RI:

./edgesoftware install

5. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.

NOTE: The key is also displayed (top left) on the download page after selecting the Download button. 

Figure 2: Prompt for product key

6. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module.

Figure 3: Successful installation.

NOTE: If the installation produces errors or fails, please refer to the Troubleshooting: Installation Failure section at the end of this document. Installation failure logs will be available at path - /var/log/esb-cli/Real_Time_Sensor_Fusion_for_Loss_Detection_//install.log.

Step 2: Build the Application

1. Navigate to the working directory:

cd Real_Time_Sensor_Fusion_for_Loss_Detection_<version>/Real_Time_Sensor_Fusion_for_Loss_Detection/rtsf-at-checkout-reference-design

 

NOTE: In the instructions above,  is the Intel® Distribution of OpenVINO™ toolkit version downloaded.  

2. Run the below command to build the real time sensor fusions Docker images:  

USE OF SUDO: The sudo command enables the user to run applications or commands as the superuser with temporarily elevated privileges. Add sudo to build commands and all Docker and Docker-compose commands if the user group is not added in Docker.  

sudo make docker

NOTE: This command may take a while to run depending on your internet connection and machine specifications. 

===Check for Success===

Check for the built Docker images by running the command below: 

sudo docker images

Inspect the list:

rtsf-at-checkout/event-reconciler
rtsf-at-checkout/device-scale
rtsf-at-checkout/product-lookup
rtsf-at-checkout/loss-detector
rtsf-at-checkout/rsp-controller-event-handler
rtsf-at-checkout/cv-region-of-interest

If all the above Docker image files to do not appear: 

1. Look through the console output for errors as sometimes dependencies fail to resolve and must be run again. 

2. Address any obvious issues. 

3, Check the Docker images again by running the following command: 

sudo docker images

4. Return to the beginning of Step1: Build the Application to try again. 

Step 3: Start the Application

1. Run the command below to start the EdgeX Foundry device services and then the RI suite:

sudo make run-base
Figure 4: Successful run of make run-base

 

NOTE: If you encounter any issues, please refer to the Troubleshooting: Run the Application Failure section at the end of this document. 

2. List the Docker images that are built successfully:

sudo docker ps --format 'table{{.Image}}\t{{.Status}}'

==Check for Success===

Check for the following docker images and Up status: 

Figure 5: Docker images and status

NOTE: Only edgexfoundry/docker-core-config-seed-go:1.1.0 should display a status of Exited.

Status times for images will vary. If any of the above services display a status of Exited, use the following steps to fix the proxy config issue:

  1. Stop Docker Containers:
    sudo make down
  2. Remove Docker proxy config file:
    mv ~/.docker/config.json ~/.docker/config.json.backup
  3. Return to the beginning of Step 3: Start the Application.

Step 4: Install Postman

Postman will be used to test EdgeX Foundry setup and send the Rest API request for the simulated data. It is the driver for the prototype’s simulated data. 

  1. Go to Postman Downloads
  2. Choose the Download the App button. 
  3. Choose the correct version, 32 or 64-bit, and download the tar.gz installation file. 
  4. Extract the installation file in a desired location.  
  5. After extraction, double-click the application and sign in with your existing account or create a new account to sign in. 

After the RI core components are installed and running, use Postman to simulate data and events. 

Run the Application

This section contains three main subsections: 

  • Checkout Events:  Provides a high-level explanation of the POS events used in Simulate a Reconcilable Object.  
  • Simulate a Reconcilable Object: Lists the individual steps to invoke POS events individually in the Postman app.
  • Use Postman Collections: Expands the use case examples, building on the Simulate a Reconcilable Object, with additional events and simulated data for other devices. It includes use cases with suspect items. 

Checkout Events 

The following are the details for the RTSF at Checkout event types used in Simulate a Reconcilable Object. 

POS Events 

POS events drive the RI. They cannot be omitted. There are five different POS events required for this reference design: 

POS Event Description of Occurrence
Basket-open  A session has started at the self-checkout. 
Scanned-item  An item has been scanned at the self-checkout. 
Payment-start  The payment has started at the self-checkout. 
Payment-success  The payment has successfully completed at the self-checkout. 
Basket-close  The session has ended at the self-checkout. 

Example Data for Events

Basket Open 

{ "lane_id" : "1", "basket_id": "abc-012345-def", "customer_id": "joe5", "employee_id": "mary1", "event_time" : 15736013010000 } 

Scanned Item 

{ "lane_id" : "1", "basket_id" : "abc-012345-def", "product_id" : "00000000324588", "product_id_type" : "UPC", "product_name" : "Red Apples", "quantity" : 3.0, "quantity_unit" : "EA", "unit_price" : 0.99, "customer_id" : "joe5", "employee_id" : "mary1", "event_time" : 15736013170000 } 

Payment Start 

{ "lane_id" : "1", "basket_id" : "abc-012345-def", "customer_id" : "joe5", "employee_id" : "mary1", "event_time" : 15736013660000 } 

Payment Success 

{ "lane_id" : "1", "basket_id" : "abc-012345-def", "customer_id" : "joe5", "employee_id" : "mary1", "event_time" : 15736013780000 } 

Basket Close 

{ "lane_id" : "1", "basket_id" : "abc-012345-def", "customer_id" : "joe5", "employee_id" : "mary1", "event_time" : 15736013940000 } 

 

Simulate a Reconcilable Object

All the core components of RTSF at Checkout should be installed and running before running the steps below.

1. Open Logs: Open the Docker logs in a terminal window to make sure checkout events are processed correctly. To open the Docker logs, run: 

sudo docker logs -f event-reconciler 

===Check for Success=== 

A message of Starting HTTP Web Server on port :48095 indicates success.

Figure 6: Starting the HTTP Web Server on Port: 48095

 

If any connection error occurs, stop the services by running the following command: 

sudo make down 

Return to Step 3: Start the Application. Do not continue until the logs display successfully.  

2. Test EdgeX Foundry: Open Postman and send an HTTP GET request with this URL:

http://localhost:49986/api/v1/ping

Use this HTTP GET request to test EdgeX Foundry REST operation, to ensure the service is online.  

a. Choose File -> New and then Request.  

b. Add Request name and save to a folder or collection.  

c. Add the URL above to the Enter request URL textbox. 

d. Choose Send. 

===Check for Success=== 

If the test was successful, a message of 1.1.1, the version number of the RI, and a status of 200 OK are returned as in Figure 7.  

Figure 7: Successful test of EdgeX Foundry

 

NOTE: If the request returns Response Status Code 404 or no response, make sure the reference implementation was built correctly and the application was started properly according to the instructions in Step 2: Build the Application. Do not continue until the request produces a Status: 200 OK (bottom left of Figure 7).

3. Initiate a transaction: For basket-open, send a POST request wih this URL:

http://localhost:49986/api/v1/resource/device-pos-rest/basket-open

Select Body, then Raw. Enter the text below, and press Send

{  
"lane_id":"1",  
"basket_id": "abc-012345-def",  
"customer_id": "joe5",  
"employee_id": "mary1",  
"event_time":15736013930000  
} 

===Check for Success===  

The request returns 1 in the return body and a status of 200 OK. If it does not, refer to the Troubleshooting: Device REST Error at the end of this document.  

NOTE: Instructions three through seven, involve POST requests that result in the return of 1 and a status of 200 OK. See Pretty at the bottom left and Status: 200 OK at the bottom right of Figure 13.  

4. Scan an item: For scanned-item, send a POST request with this URL:

http://localhost:49986/api/v1/resource/device-pos-rest/scanned-item

Select Body, then Raw. Enter the text below, and press Send

{  
"lane_id":"1",  
"basket_id": "abc-012345-def",  
"product_id": "00000000571111",  
"product_id_type": "UPC",  
"product_name": "Trail Mix",  
"quantity": 1,  
"quantity_unit": "EA",  
"unit_price": 5.99,  
"customer_id": "joe5",  
"employee_id": "mary1",  
"event_time":15736013940000 
 } 

===Check for Success=== 

The request returns 1 in the return body and a status of 200 OK. If it does not, refer to the Troubleshooting: Device REST Error at the end of this document. 

5. Prepare for payment. For payment-start, send a POST request with this URL:

http://localhost:49986/api/v1/resource/device-pos-rest/payment-start

Select Body, then Raw. Enter the text below, and press Send

{  
"lane_id":"1", 
 "basket_id": "abc-012345-def", 
 "customer_id": "joe5", "employee_id": "mary1", 
 "event_time":15736013950000  
} 

===Check for Success===  

The request returns 1 in the return body and a status of 200 OK. If it does not, refer to the Troubleshooting: Device REST Error at the end of this document. 

6. Payment succeeded. For payment-success, send a POST request with this URL:

http://localhost:49986/api/v1/resource/device-pos-rest/payment-success

Select Body, then Raw. Enter the text below, and press Send

{  
"lane_id":"1",  
"basket_id": "abc-012345-def",  
"customer_id": "joe5",  
"employee_id": "mary1",  
"event_time":15736013960000  
} 

===Check for Success===  

The request returns 1 in the return body and a status of 200 OK. If it does not, refer to the Troubleshooting: Device REST Error at the end of this document. 

7. Transaction is closed. For basket-close, send a POST request with this URL:

http://localhost:49986/api/v1/resource/device-pos-rest/basket-close

Select Body, then Raw. Enter the text below, and press Send

{  
"lane_id":"1",  
"basket_id": "abc-012345-def", 
 "customer_id": "joe5",  
"employee_id": "mary1",  
"event_time":15736013970000 
 } 

===Check for Success===  

The request returns 1 in the return body and a status of 200 OK. If it does not, refer to the Troubleshooting: Device REST Error at the end of this document. 

Use Postman Collections

To eliminate making separate HTTP calls for each event, RTSF at Checkout provides several full scenarios as Postman Collections. These collections test different sensor combinations, various products, and customer behaviors. Include and build on the Postman Collections to: 

  1. Understand the reference solution. 
  2. Generate simulated data to test or improve the reconciliation algorithm. 
  3. Simulate one or more device(s), with your own device running alongside simulated devices. 
  4. Perform testing and validation. 
  5. Discern the difference between reconcilable object and irreconcilable objects.  

The collection builds in complexity, demonstrating simulated data and events and additional sensor devices.  

POS Collection Use Cases Description
POS and Scale (No Suspect items)  Scan and weigh three reconcilable items. Transact payment. 
POS and Scale (Suspect scale item)  Scan and weigh three reconcilable items and one irreconcilable item. Transact payment 
POS, Scale, CV (No Suspect items)  Scan and weigh three reconcilable items, with items moving through ROI. Transact payment.
POS, Scale, CV (Suspect CV and Scale items)  Scan and weigh three reconcilable items and one irreconcilable object, with items moving through ROI. Transact payment.
POS, Scale, RFID (No Suspect items)  Scan and weigh three reconcilable items with RFID staging. Transact payment  
 
POS, Scale, RFID (Suspect RFID items)  Scan and weigh three reconcilable items and one irreconcilable with RFID staging. Transact payment.

 

WARNING: The instructions describe how to use the Postman user interface to import a collection. Import instructions depend on the version of Postman used and may not correspond exactly with the instructions below.  

To use the Postman Collections: 

  1. Open Postman and navigate to a workspace. 
  2. Choose File, then Import. 
  3. Select the Folder tab and choose the files to upload, the included collections in /loss-detection-app/postman-collections.  
  4. Select the included collections. 
  5. After the import completes, navigate to the desired collection, and choose the “...” menu.  
  6. Select Run the Collection
  7. Set the desired Number of Iterations and Delay
  8. Choose the button for the collection, for example Run POS and Scale (No suspect Items). 
  9. Check the log files with the following command: 
sudo docker logs –f event-reconciler 

===Check for Success=== 

Each event should finish with a status of 200 OK.  

Figure 8: Successful Run of Collection POS an Scale (No suspect items)

 

Summary and Next Steps

You have successfully created a simulated scenario with the POS. Next, try the section, Components to build a solution that is based on this reference design, in the Real-time Sensor Fusion at Checkout Get Started Guide.

Learn More

To modify and enhance this prototype for your own use case, see the Production Suggestions which outlines several ideas for replacing RI components. 

To continue learning, see the following guides and software resources: 

Troubleshooting 

Installation Failure

If host system already has Docker images and its containers running, you will have issues during the RI installation. You must stop/force stop existing containers and Images.

  • To remove all stopped containers, dangling images, and unused networks:

sudo docker system prune --volumes
  • To stop Docker containers:

sudo docker stop $(sudo docker ps -aq)
  • To remove Docker containers:

sudo docker rm $(sudo docker ps -aq)
  • To remove all Docker images:
sudo docker rmi -f $(sudo docker images -aq)

 

Run the Application Failure

During Step 3: Start the Application, 'make run' may fail with error with the message 'address already in use' due to the same container running in different instances.

Figure 9: Address already in use

To resolve this, get the PID of the processes using the required addresses and force stop them using the commands below:

sudo netstat -tulpn | grep <address in use>
sudo kill -9 <PID of the process>

MQTT Address Error

If you get MQTT address ‘already in use’ error, get the PID of the processes using the required addresses and for them to stop them using the commands below

 

sudo lsof -i -P -n | grep <port number>
sudo kill -9 <process id>

Address any obvious issues, for example network issues or Docker proxy issues.

Device-Rest Error

If you get a Response Status Code 404 or get no response, make sure you correctly built and ran device-rest. 

sudo netstat -tulpn | grep <address in use>
sudo kill -9 <PID of the process>
  • Stop the services by running the following command: $ make down

  • Return to Step 3: Start the Application. Do not continue until you successfully receive a message. 

Docker Image Build Failure

If Docker image build on corporate network fails, follow the steps below.

1. Get DNS server using the command:

nmcli dev show | grep 'IP4.DNS'

2. Configure Docker to use the server. Paste the line below in  /etc/docker/daemon.json file:

{
    "dns": ["<dns-server-from-above-command>"]
}

3. Restart Docker:

sudo systemctl daemon-reload && sudo systemctl restart docker

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.