Two senior researchers of the National Institute of Advanced Industrial Science and Technology, an independent administrative institution (AIST): Dr. Osamu Morikawa, Institute for Human Science and Biomedical Engineering (IHSBE) and Dr. Kenji Toda, Information Technology Research Institute (ITRI) with their colleagues, manufactured on a trial basis an innovative display system designated “Virtual Projectors and a Screen” (VPS), where multiple images of different categories could be displayed at once on a single screen. With the new display system, you can view images of varied formalities, such as television, digital photo, personal computer, etc. on a single screen simultaneously and under independent control for respective sizes and positions.

So far, simultaneous display of plural images on a single screen required either synthesizing input video signals on a computer or a system for exclusive use, or resorting to split-screen or picture-in- picture feature on the part of display system, where the number of input images and/or input-output format has to be restricted. While the synthesis with a computer or a specialized system functionally holds greater degree of freedom, it needs an advanced skill and is not manageable for a beginner. For the functions on the display system side, the degree of freedom is constrained by a number of conditions such as requirement of video signal of one and same category, limited number of images and prohibition of overlay.

The newly developed video display system is, as it were, an electronic emulation for projecting multiple images on to a screen by using projectors. Hence, it is designated as “Virtual Projectors and a Screen” (VPS). The VPS display system consists of three components: multiple virtual projectors, a virtual screen involving layer structures as many as the number of projectors, and a display unit to present contents of the virtual screen.

Virtual projectors as many as the number of input video signals are provided, and each of images can be changed independently in respect to size and position without mutual interaction. Overlaid area of each image can be displayed according to the user-specified priority and colored or cleared under specific conditions. The system accepts input video signals of different formalities and resolution: for instance, still photo from a digital camera, high definition TV signals and computer output. In the system for trial manufacture, two types of input video signals: NTSC (National Television System Committee) signal, DVI (Digital Video Interface) signal are adopted, and video signals resulted from the arithmetic operation of images are fed into a commercially available liquid crystal display. It is readily possible to increase the number of input video signals, to accept input video signals of different formality and to drive liquid crystal panel and light emitting device directly.

The VPS display is characterized by easy operation; with a video unit and a digital camera first connected, images are displayed, and the position and size of displays can be changed as you like it by simply operating a remote controller. The input device may be disconnected or switched off whenever unwanted, to delete the corresponding image. You can add or remove any image at will without expertise nor skill. On the basis of successful demonstration of VPS display function, the future R&D efforts will be directed to the commercialization, including the augmented diversity of input video signals and the circuit optimization for reducing cost.

Fig. 1. Simultaneous display of images of varied formalities with overlay at selected positions on a screen. (an example of VPS display)

“Biomass” refers to biological resources available on a massive scale, such as trees, grasses, seaweeds, agricultural and forestry wastes. Since the quantity of carbon dioxide emitted on burning the biomass is equivalent to that released through their death and bacterial decomposition, it constitutes so-called “carbon-neutral” renewable energy which is not to be counted in CO2 emission responsible to global warming, and its effective utilization is expected to contribute to the alleviation of the greenhouse effect. As is well known, woods have been used as fuel since earlier. Besides, in addition to the fuel use, the biomass can be transformed through the energy conversion technology into transportable energy, such as ethanol, methanol, bio-diesel fuel and so on. It is focal issues of the latest R&D tasks how to produce these renewable fuels efficiently and economically. Since it is important to exploit the biomass resources effectively along a consistent flow of work steps, such as harvesting, transfer, conversion and practical utilization, close collaboration is indispensable among scientists and engineers in related fields of disciplined concerned.

The IHSBE-AIST is dedicated to studies on how to turn data and products brought out through the information technology (IT) revolution beneficial for every user, aiming at design and creation of understandable web pages, hyper-mirror remote interaction system to form a dialog link using synthetic images, and 3D simulation system for training endoscopic surgery. In the course of these studies, it was noted that simultaneous display of multiple images might be very helpful in many cases.

Let us take an example of simultaneous display of multiple images from a scene of studies at the IHSBE-AIST. If an endoscopic view and an auxiliary video data helpful for the operation are displayed on two different screens, the operating surgeon has to switch his sight line alternately from one screen to the other [Fig. 2-A]. This may disturb the surgeon’s concentration. When the NTSC signals from the endoscope is combined with the auxiliary analog RGB signals (resolution: 800 x 600 pixels) from the computer on a single screen, the surgeon will be able to view both info simultaneously with minimum shift of sight line [Fig. 2-B]. Moreover, if the tilt of the endoscope is overlaid [a yellow line in the top right of Fig. 3], the surgeon may perceive the level line of the endoscope intuitively [bottom of Fig. 3]. In this way, simultaneous display of multiple images on a single screen facilitates the image recognition effectively.

Fig. 2. Simultaneous display of an endoscope image and an auxiliary info image, separately or jointly.
A: Using two screens	B: Combined on single screen

Fig. 3. Example of a combined display of an endoscopic surgery scene with an auxiliary info image (top) and a combined display

To display multiple images on a single screen, two methods have been available: (1) the endoscopic image (NTSC signals) is fed into a computer, and synthesized into input video signal---- computer-based image synthesis. (2) the auxiliary video signal from the computer (analog RGB signals of resolution 800 x 600 pixels is converted to NTSC signal by use of a scan converter, and two signals are combined by an image synthesizer to be presented on a standard display----- image combination by a synthesizer [Fig. 4 top]. The IHSBE-AIST has been implementing the combined display by using an image synthesizer, which is large-sized, high-priced and requires complicated processes for installation, wiring and adjustment. The image synthesis by a computer needs special-purpose software and the system installation and operation are complicated and time-consuming. The two methods are, therefore, difficult to execute for unskilled persons.

It may be necessary to think the matter from a different angle: in place of “feeding input video signal combined by a computer or an image synthesizer into a display”, an innovative way of display is proposed, “accepting multiple input video signals, as they are, and processing these signals internally to bring forth simultaneous display [Figs. 1 and 4, bottom].

The R&D works for the new display scheme are being carried out in collaboration with the ITRI-AIST. The latter institute has been engaged in studies on high performance integration system, and acceleration of operational speed, and promotion of parallel and distributed processing with FPGA (field programmable gate array) devices, that is, LSI chips with rewritable logic circuits. These innovative technologies have been utilized in the present trial manufacture.

Fig. 4. Means for implementing an integrated screen Current method using image synthesizer (top) and New display method

1. Description of VPS Display

The newly developed display scheme is, as it were, an electronic emulation of projecting multiple images on to a screen by using plural projectors, and designated as “Virtual Projectors and a Screen” (VPS) system [Fig. 5].

Fig. 5. Operational principle of VPS An example of simultaneous display with three projectors. An additional input video signal can be provided by installing a corresponding projector. This process is emulated electronically.

A display unit based on VPS display consists of three components: multiple virtual projectors, a virtual screen with layer structures corresponding to virtual projectors, and a display device to present contents of virtual screens [Fig. 6].Multiple virtual projectors and the display device operate asynchronously with each other.

The substantial entity of the virtual screen consists of computer memory, and the virtual projectors are writing circuits for the memory. The display unit includes a simple image processing circuit and a drive circuit, reading data from each layer of the virtual screen, computing for each of pixels how to work data in particular layer, and driving a liquid crystal panel or a light emitting device.

Fig. 6. Construction of a VPS display unit of 4-input configuration

Multiple input video signals are projected through a corresponding virtual projector on to the virtual screen at a selected position. The displayed image is derived from a layer corresponding to the virtual projector concerned. As each of virtual projectors independently corresponds to each input video signal, the conventional constraints for video signals are not applicable. So long as a virtual projector is available, the input video signal may not conform the NTSC signal format 30 frames/sec, and a frame is not to be rectangular. The geometry of video signal may be specified freely in such a way as “changed area in a move”.

Each of input video signals is projected by a corresponding virtual projector on to the virtual screen where a uniform 2D image is formed. The display unit reads the content of the virtual screen and decides how to combine contents of different layers. The user has only to indicate the projection position on the screen and the priority of layer display sequence.

The VPS display is characterized by easy operation. First, you connect a video device or a digital camera to view the image, and change its position and size as you like it by using a remote controller. The input device may be disconnected or switched off whenever you feel unnecessary. In this way, ant image can be added or deleted freely by an unskilled person without expertise.

Another merit of the VPS system is lack of discordance between digital camera signal (NTSC) and computer image signal (analog RGB signal of resolution 800 x 600 pixel image) in the image synthesis. Characters written by a computer are clearly legible so long as using computer image signal (RGB analog of 800 x 600 pixel resolution), but are degraded when converted to camera image signal (NTSC). With the VPS display, the computer image signal is displayed as they are without causing such an inconvenience.

In the IHSBE-ITRI collaboration, a VPS display system with three units of virtual projector was manufactured on the trial basis. The input video signal for the virtual projector was either NTSC or DVI format, and the virtual screen size was set to 1280 x 1024 pixels. As the conventional display for NTSC or DVI signal is formatted to rectangular geometry, and circular image of endoscope cannot be displayed directly. In case of VPS display, the endoscope image can be displayed as it is, only if a circular image is defined.

In the actual endoscope, so far, the NTSC format was provided by filling the peripheral space between circular field and rectangular area with black color. In the VPS system, a special “clear color” was provided so that an image of any geometry might be entered as a rectangular NTSC signal by painting the periphery with clear color.

The trial-manufactured VPS system contains four units of fast FPGA (LSI chips with rewritable logical circuits) for the sake of facilitating circuit design, with redundant processing capability [Fig. 7]. In Fig. 7, each of four areas surrounded by dotted lines represents a unit of FPGA device. For commercializing the system, it will be needed to develop special purpose LSI chips for reducing cost.

Fig. 7. A block diagram of the trial manufactured VPS display system.