Yoshihiro Kawai (Leader), Vision and Manipulation Research Group, the Intelligent Systems Research Institute (Director: Hirohisa Hirukawa) of the National Institute of Advanced Industrial Science and Technology (AIST; president: Tamotsu Nomakuchi), and Fumiaki Tomita (Invited Senior Researcher), Kansai Collaboration Center (Manager: Takahisa Taguchi) of AIST, have developed a three-dimensional whole shape measurement system for automatic processing of marine and agricultural products in collaboration with Nikko Co., Ltd. (Nikko; Representative Director: Atsushi Sato).

This system is capable of measuring the three-dimensional whole shape of products of various forms, as carried on belt conveyors. Based on the measured shape data, the system can generate automatic processing data for cutting each product into equal-sized pieces. It can also handle various kinds of marine and agricultural products including defrosted and hence softened or frozen/distorted fish fillets with scales and pork ribs, which used to be difficult to measure with sufficient accuracy. It is expected that this system will contribute to the improvement of yield and productivity for processing marine and agricultural products.

A prototype of this system will be exhibited by Nikko at the International Food Machinery & Technology Exhibition (FOOMA JAPAN 2011), to be held from June 7 to 10, 2011, at Tokyo Big Sight, Koto, Tokyo.

Three-dimensional whole shape measurement system for marine and agricultural product and examples of measurement and cut pieces

In the food processing industry, due to recent steep cost increases, demand is increasing for technology that can improve the yield of product processing. Meanwhile, as skilled workers are getting older, an automatic processing system that enables stable processing using machinery is required. However, since marine and agricultural products, even products of the same kind, vary in their conditions and/or shapes, their weight and shapes have been very difficult to measure accurately. No automatic processing system with a high processing yield has been realized for marine and agricultural products thus far.

Nikko has been manufacturing and selling various processing systems for agricultural and marine products. Their automatic fish fillet-processing machine experienced the problem of being unable to process accurately, because it cannot stably measure the shapes of defrosted/softened or frozen/distorted fish fillets and can measure only the top surface. To expand their product lineup further, Nikko has been searching for technology that enables the accurate measurement of the shapes of various marine and agricultural products.

In the meantime, AIST has developed the “Advanced 3-D Vision System VVV,” a type of measurement and recognition technology for various objects, and has been working on the application of the technology for the manufacturing industry and for service robotics in order to give the benefits of technological progress back to society.

This time, AIST and Nikko found that their needs and ideas match each other, and they have thus agreed to work together toward solving the problems regarding existing products by applying three-dimensional measurement technology using stereo cameras, which resulted in the development of this system.

This research and development project was supported by the “Demonstration Research Program for the Procurement of Inspection and Measurement Equipment by Small, Medium-size, and Venture Enterprises (FY2007-2009),” which is part of the Small and Medium-size Enterprise Industrial Engineering Research and Development Program of the Ministry of Economy, Trade and Industry.

The shape of a fish fillet is curved at the top surface and even at the bottom surface, where internal organs are removed. Therefore, to obtain a complete three-dimensional shape, measurements of both the top and bottom surfaces need to be performed simultaneously. For the measurement of the bottom surface, the researchers first tried to apply the light-section method, which has been used for conventional top surface measurement, where a laser light is projected to the object and its image is captured by a camera. In this case, two belt conveyors need to be set up and the measurement is carried out through the gap between them and the plane of laser light needs to be aligned with the gap. If the camera is located too close to the plane of laser light, the accuracy is, in principle, degraded. Thus, as shown in Fig. 1 (a), the camera needs to be located away from the light source in the conveyor’s moving direction. However, when the gap is narrow, the conveyor’s thickness blocks the laser light and the measurement fails. On the contrary, when the gap is wide, the fish fillet either falls off or gets jammed in the gap and slips.

Consequently, the researchers adopted the correlation stereo method using stereo cameras as the measuring instruments, and they have developed a measurement system that provides a stable result even if the gap between the belt conveyors is set narrow. This method allows the cameras to be aligned parallel to the gap, as shown in Fig. 1 (b), and thus, the belt conveyors can be arranged with a narrow gap to achieve a stable movement of fish fillets. Eight cameras (four sets of stereo cameras) are used to perform measurements from four directions, such that cross-sectional data of each fish fillet while being conveyed are obtained. By combining all data from the head to the tail, the three-dimensional whole shape data of the fillet is obtained (Fig. 2).

(a) Light-section method (conventional method)	(b) Correlation stereo method (newly developed method)
Figure 1: Schematic drawings of three-dimensional shape measurement techniques

Figure 2: Results of the three-dimensional whole shape measurement of a fish fillet (top and bottom surfaces)

Fish fillets may become shiny and cause specular reflection, depending on frozen or wet conditions, whereas black scale areas on the backside do not reflect light well. Such surface conditions may disturb the three-dimensional shape measurement. Therefore, the light source positions were devised to make shades darker and reduce the influence of specular reflection. The researchers also utilized the brightness distribution information and assumed the continuity of the object’s surface. They thus developed an algorithm for stereo image processing, which enables stable measurement even for those black areas with few clues.

As described above, the researchers have developed a “three-dimensional whole shape measurement system” that makes it possible to measure the three-dimensional whole shape of a fish fillet being carried on belt conveyors (Fig. 3). This system is capable of performing three-dimensional whole shape measurement at a conveyer speed of up to 8.3 m/min., with an accuracy of ±1 mm in the direction of height. For example, in the case of a salmon fillet with a total length of 440 mm, its three-dimensional whole shape data can be obtained in less than six seconds (including conveyor time). By providing additional information, such as the weight of one cut piece and the cutting method used, processing information can be calculated based on the obtained shape data, as shown in Fig. 4. With an automatic processing machine attached to the backend of this measurement system, the fillet is cut into slices that have the same weights, as shown in Fig. 5. Even for a defrosted fish fillet, which conventional machines were unable to measure stably, this system can measure the three-dimensional whole shape, which enables sufficient processing accuracy. By modifying the camera and other settings, this system can measure the three-dimensional shape of various marine and agricultural products, which in turn allows for various types of processing, such as cutting pork ribs into portions of the same weight.

Figure 3: Three-dimensional whole shape measurement system

Figure 4: Processing information calculated based on the measurement result

Figure 5: Salmon slices of equal weight cut by a processing machine

Aiming at commercialization of the three-dimensional whole shape measurement system, Nikko will continue to strive for improved robustness, higher speed, and lower system cost.