Nobuyuki Otsu (AIST Fellow), and Katsuhiko Sakaue (Principal Research Scientist), Kenji Iwata (Post-Doctoral Research Scientist), Takumi Kobayashi (Research Scientist) and Yutaka Sato (Research Scientist), the Information Technology Research Institute (Director: Koichi Hashida) of the National Institute of Advanced Industrial Science and Technology (AIST) (President: Hiroyuki Yoshikawa), have developed a software that automatically detects unusual motions in a camera video in real time using an AIST-developed image recognition technology.

Developing a monitoring camera that automatically detects unusual motions is difficult.  However, the detection technology is in great demand today due to rapid increase of monitoring cameras.  AIST has developed a calculation method called cubic higher order local autocorrelation (CHLAC).  Using this method, AIST developed a basic technology to identify a person and detect the unusual motions in a camera video, and confirmed its excellent performance so far.  (AIST Press Release: May 24, 2005)

It has been desired to increase the speed of CHLAC to enable real-time operation of CHLAC in a compact processing system, because higher speed allows CHLAC to be used for compact and inexpensive monitoring cameras.  A newly developed parallel processing algorithm has increased its speed by about 10 times and has enabled its real time operation in the processing system like a notebook PC.  A platform software “Lavatube” was also developed.  With operations like combining icons, Lavatube supports an easy construction of a processing system of camera videos such as CHLAC.

The high-speed CHLAC and Lavatube will be presented and exhibited in the Hokuriku Technology Fair to be held at Fukui Prefectural Industry Hall on 18th and 19th of October, and the Industry-Academic-Government Exchange Fair to be held at Tokyo Big Sight from 28th to 30th of November.

Real-time detection of unusual motion by CHLAC implemented on Lavatube. Usual and unusual motions are expressed numerically by using CHLAC.

Demonstration : unusual motion of a pedestrian (Windows Media content) Play : 45"

> Demonstration : prying a locker open (Windows Media content) Play : 45"

> Demonstration : antomatic announcement to a wheelchaired person (Windows Media content) Play : 24"

With growing concern regarding security, a large number of monitoring cameras are being installed in various places lately.  However, because it is expensive to have humans continuously monitor all the images, videos are recorded and used for a follow-up when an accident or crime occurs.  Observers can monitor many cameras using a central management system in large facilities.  It is, however, difficult to simultaneously monitor a large number of camera videos without oversight as long as humans conduct the monitoring; thus, a system that recognizes the videos of a monitoring camera has been strongly desired.  Because automatic recognition system can detect unusual motions in real time, monitoring cameras can be used not only for follow-ups but also for a further effective security system in many places.

Nobuyuki Otsu (AIST Fellow) et al. developed a higher order local autocorrelation (HLAC), which has the learning ability, as a recognition method of still images; HLAC was selected as one of the remarkable inventions named in the 50th convention by the Science and Technology Agency.  They expanded HLAC to make it applicable to motion image processing, and invented a calculation method called CHLAC.  Based on CHLAC, they developed a basic technology to identify humans in a camera video and detect unusual motions, and confirmed its excellent performance.  (AIST Press Release: May 24, 2005)

CHLAC require smaller amount of calculation compared with the other methods for the same purpose.  However, high speed CHLAC, that can process more than 30 frames per second in a smaller processing system, has been desired.  We have developed a new algorithm that makes CHLAC about 10 times faster than before by parallel computation.  Using the algorithm, a small processing system at the notebook PC level can process multiple numbers of camera videos in real time, without relying on a special hardware.

To provide a framework that supports the construction of a system using CHLAC, we developed a platform software “Lavatube” that helps construct a system to process camera videos through the combination of icons.  In the past, constructing an image processing system required expertise of image processing and programming.  Lavatube, however, enables a developer to easily construct an image processing system by connecting icons.  At the same time, he can efficiently develop the system because he can adjust parameters while displaying processing results in real time.

The processing system using CHLAC detects unusual motions in the video by analyzing the feature vectors of camera videos that CHLAC obtains by principal component analysis.  Here, CHLAC can detect an unusual motion by defining it as “a motion outside the distribution of usual motions,” without creating a model of an unusual motion that varies with the scene and cannot be defined beforehand.  The system learned the sample distribution (subspace) of usual motions beforehand, and the system can detects unusual motions, those are different from the usual motions.  In Figure 1, for example, CHLAC memorizes the scenes of opening and closing the locker as usual motions, and precisely detects any other motions (here, prying a locker open).

Fig.1 An example of unusual motion detection. (Prying a locker open)

In addition, CHLAC exhibits an excellent performance even in the open air where the influence of various disturbances such as change in sunlight and shaking of trees is unavoidable.  It is possible to eliminate these disturbances, when the system learned such disturbances as parts of usual motions.  As shown in Figure 2, the system with CHLAC can detect an unusual motion (overleaping the fence in this case) even in the scene where a pedestrian is found walking among trees swinging in the wind.

Fig.2 An example of unusual motion detection. (Outdoor environment)

Although CHLAC exhibits such an excellent performance, it requires too many calculations to process in real time.  To settle this problem, we developed a parallel processing algorithm that allows for real time processing even in processing systems at a notebook PC level.  It employs single instruction/multiple data (SIMD) to increase the speed.  The SIMD, which is incorporated in the majority of current CPUs, allows a process to be performed on elements in a data set in parallel by using a single instruction.  As shown in Figure 3, the speed remarkably increased when an order MMX/SSE2/SSE3 mounted in a Intel x86 processor is employed.  With the help of high-speed processing, it has become possible to detect and notify unusual motion at the moment it occurs, instead of after it occurred.

Fig.3 Benchmark result

“Lavatube” is a software developed to assist the development of a time-series image processing system using CHLAC etc., and it enables a developer to easily construct a image processing system by connecting icons on a graphical user interface (GUI), as shown in Figure 4.  Because he can adjust parameters on GUI, he can concurrently confirm the parameters and the processing results to efficiently construct the system.  In addition, inexpensive USB cameras and image files of avi or mpeg formats are supported as standard, and he can use them just by arranging their icons.

An example of image recognition system implemented on Lavatube.

Description of processing flow by GUI

Fig.4

As described above, the automation of monitoring cameras has been realized with the help of the high-speed CHLAC that has the ability to detect unusual motions in videos and identify humans, and Lavatube that supports the construction of a image processing system.  Though it is an issue that has long been supposed to be difficult to realize despite the strong demand.

CHLAC opens up applications such as security and disaster prevention (monitoring camera system, security system, and disaster prevention surveillance system), automatic indexing of video (automatic construction of video’s index), medical welfare and sports (rehabilitation, correction of motion, and training system), dialog system, and robot’s vision.  Practical application is expected to accelerate in various fields through synergy with the Lavatube that strongly assists development.