Global Network

Machine-vision software deciphers handwritten characters to deliver OCR for warehouse management

Winn Hardin, Contributing Editor, and Lutz Kreutzer, MVTec Software

WITT Weiden is a mail-order retailer that sells goods from 400 different suppliers from around the world, each with its own language and method of writing. Many of these suppliers are too small to have Data Matrix printers and other warehouse management automation tools, so labels are handwritten on the side of shipping boxes.

In an attempt to automate these handwritten labels, WITT developed a standard shipping label with clearly marked areas for each handwritten numeral. The remaining challenge was to develop an automated tracking and sorting system that could machine-read the handwritten labels with better than 96% accuracy. WITT turned to automation provider ECKELMANN to develop a flexible optical character recognition (OCR) system based on an IDS Imaging Development Systems uEye camera, MVTec Software Halcon 8.0 image-processing software, and dual support-vector machine (SVM) and neural-network (NN) analytic approach to OCR (see Fig. 1).

FIGURE 1. At the WITT Weiden mail-order warehouse, a customized OCR system developed by system integrator ECKELMANN images and tracks cartons from 400 worldwide suppliers.Click here to enlarge image

Center of the world

In addition to WITT Weiden’s mail-order business, the company services 75 outlet stores located throughout Germany. WITT boasts more than 2200 employees and generates €600 million in revenue per year. One of the centers of the WITT empire is its warehouse in Weiden, Germany. Trucks deliver the goods from more than 400 suppliers.

To manage the chaos of having suppliers from so many different parts of the world while still providing cost-effective distribution services, WITT aggressively pursued low-cost standardization-an approach that greatly assisted ECKELMANN in the design of the new warehouse tracking system.

Goods arrive by truck, packed in one of three sizes of standard WITT boxes. WITT requires its suppliers to use both the standard-size boxes and new standardized labels. Many of the suppliers are small manufacturers located in Asia, for whom printing of barcode labels would not be possible. Therefore, WITT has only two requirements: Use the label and relay all relevant data (item number, quantity, and size) in Arabic numerals-either printed or handwritten.

WITT had previously installed a machine-vision system to read shipping labels, but the proprietary system proved too expensive to maintain. During a recent warehouse improvement initiative, WITT executives decided to upgrade the vision system. The new system would have to recognize a minimum of 96% of the handwritten characters on the shipping labels at a frequency of 1.5 s/label. It needed to use standard PC and hardware components, could not impact operations during installation, and could not require an additional support system that would burden the warehouse infrastructure budget.

Communicating with WMS

Each box eventually arrives at the OCR vision system. The system is designed to read the printed and handwritten labels, measure the size of the cartons-whether one carton or two stacked cartons are present-and combine that information with the carton’s weight, and then print all the information on an interim label used by WITT’s enterprise resource planning (ERP) and warehouse management system (WMS; see Fig. 2).

FIGURE 2. Packages from many different cultural areas deliver different handwritten characters. An in-house label with item number and storage position is glued to the supplier’s printed or hand-written label.Click here to enlarge image

As WITT demanded, ECKELMANN designed the system using a standard PC with a Gigabit Ethernet (GigE) port running MVTec’s Halcon 8.0 image-processing software and Windows XP. The GigE port is connected to a standard commercial GigE switch, which connects to the IDS uEye digital monochrome camera and WMS system via the LAN. To provide additional redundancy for the PC host, the hard drive, including all inspection routines and Halcon programming, is mirrored on the plant’s network, allowing maintenance technicians to replace a failed hard drive and get the vision system up and running within an hour.

An industry-standard, energy-saving ERSO 150-W lamp (ERSO Indulux HIT 150W) is placed on the side of the conveyor with the label. Dockworkers make sure to place each carton on the conveyor so that the label is always presented to the machine-vision side when it reaches the OCR station.

NNs + SVMs

As each carton enters the machine-vision OCR inspection zone, a photoeye sensor connected to a Wiesemann & Theis (W&T) Web-IO interface communicates back to the PC via TCP over LAN. A conveyor scale weighs the carton and sends that information back to the PC through a second Web-IO interface. “The scale has a COM port and uses a very simple ASCII communication protocol. Because standard PCs in the future may not have COM ports, the communication to the scale is also realized via TCP over LAN using a W&T COM server,” explains Arno Dewald, project engineer at ECKELMANN. “The intention was to realize all communication to external components such as the camera and scale via Ethernet so we don’t need special hardware in the PC” (see Fig. 3).

FIGURE 3. As each carton enters the inspection zone, a photoeye sensor connected to a Web-IO interface communicates back to the PC. A conveyor scale weighs the carton and sends that information back to the PC through a second interface. The weight, size, and OCR results are sent to the WMS across the LAN. The uEye GigE camera acquires an image of the label, and the Halcon software begins a dual-path approach to OCR.Click here to enlarge image

The weight, size, and OCR results are sent to the WMS across the LAN. The WMS allocates a storage location depending on size and weight of the carton. This storage location, along with all determined information (size, weight, and OCR result), is printed on the carton by the label printer in another location, which prints an interim label used by the warehouse staff. To support the proprietary software environment of WITT Weiden, exactly specified ASCII strings were created to communicate with the plant. The uEye GigE camera acquires an image of the label, and the Halcon software begins a dual-path approach to OCR (see Fig. 4).

FIGURE 4. Lighting for the vision system consists of a standard 150-W halogen lamp and image acquisition is performed by a uEye camera. Images are captured in front of a black background.Click here to enlarge image

First, the carton in the image is segmented from a black background placed opposite the camera. The size of the side of the carton is analyzed via blob analysis to determine the size of the carton and whether one carton is present or two cartons are stacked one atop the other. If two cartons are identified, an alarm is sounded to call an attendant.

Once the carton is segmented, Halcon software uses a geometric pattern search to locate the label on the carton and the specific squares that hold the handwritten or printed characters (see Fig. 5). These characters are segmented from the label and checked against standard OCR font libraries. When labels have printed characters, this fast OCR check will quickly identify the label information and send it to both the WMS and printer for data storage and a new label.

FIGURE 5. PC user interface displays different identification results. Package size identification and handwritten character recognition are based on Halcon 8.0.Click here to enlarge image

If the first classification fails, the characters are analyzed against various handwritten character sets stored as vector sets. ECKELMANN designed the handwritten OCR analysis routine to use both an NN classifier and an SVM classifier, where results from both analyses can be used to check each other and improve the accuracy of the overall read.

The system first uses an NN algorithm to identify the characters. The result is checked against stored “plausible” values, such as current supplier identification numbers and common numbers of cartons shipped. “If the result is plausible, the OCR ends here,” explains Johannes Stelter at ECKELMANN. “If not, the section is passed to a second NN or SVM and so on. We are using NN and SVM in a competing way because both technologies have advantages and combining them increases the rate of successful reads. The verification is done using somea priori knowledge--that is, the range of allowed dress sizes.”

If the NN result is determined to be unlikely, ECKELMANN turns to the SVMs built into the Halcon 8.0. SVMs are an alternative training method for polynomial, radial basis function, and multilayer perceptron classifiers saved by a quadratic programming problem with linear constraints, rather than non-convex, unconstrained minimization problems used with standard NN training.

“The implementation of the image processing as we planned it was only possible by several unique Halcon 8.0 attributes,” says ECKELMANN’s Dewald. Initial results showed that the handwritten OCR vision system correctly read 100% of undamaged labels during the first month of operation.

“We are very contented regarding the project settlement and the performance of the application,” says Sabine Schaumberger, project manager at WITT Weiden. “Even during the fast-paced Christmas business, the launch of the application by the ongoing factory was no problem.”