Research Activities

(as of 2004)

Recognition of human actions and related topics

Recognition of human actions is an important research topic which has many application areas. One of our research goals is to establish a library of computational procedures for the recognition of a wide variety of human actions . Examples of such actions are the "walking" and "running" actions, which are entire body actions, manipulative actions such as "grasping a cup", or the "reading action" which can only be judged by jointly evaluating several sub-activities over an extended period of time.

Our projects can be divided into two categories according to the sensing techniques being used:

1. One of the projects is based on solely monocular color image sequences as input data. The research objective here is to develop techniques that are able to recover enough 3D information from an image sequence to be able to estimate the 3D trajectories and velocities of moving persons and their major body parts, such as hands and legs.

Example of "walking" action:

Left image: Comprehensive view of the action scene; Center image: Walker trajectory in 3D space, estimated with our method; Right image: Walker velocity, estimated with our method.

2. In another project, monocular color image sequences in conjunction with (synchronously taken) thermal image sequences (from an infrared camera) and 3D object surface point measurements from a stereo camera are used as input data. The objective here is to develop first the most appropriate image acquisition system and then develop reliable data fusion techniques which work both at the signal level and decision-making level in order to (hopefully) obtain robust action recognition results.

Another topic involving the automatic sensing and recognition of human actions we have been persuing in our laboratory is the recognition of objects which are being pointed at by a person in front of the camera system. First, the system detects a finger pointing action in real-time, then tracks the hands of the person and determines whether the hand motion represents a finger pointing action. If it does, the system estimates the direction of the line-of-sight of the person and intersects it with the stored environment model of the room. As a result of this intersection, objects pointed to by the person can be identified. The following photo shows such a finger pointing scene.

Human face recognition

In this area we proceed along three different lines of research which we are planning to eventually merge into one coherent method for general--purpose, unconstrained face recognition .

(1) Identification of faces by applying Neural Networks and Support Vector Machine learning and recognition methods to 2-dimensional face image sets which are obtained with image sensors working in the visual wavelengths and infrared wavelengths spectra.

(2) We investigate the possibility of using 3-dimensional face surface point sets obtained with a 3D laser scanner for face identification.

(3) We develop methods for the on-line acquisition of human face images of persons moving in a room; for this purpose we use the combination of a laser scanner with a line scan characteristic and a color CCD camera, both mounted on the same platform and pointed in the same direction. The following images show the system and the laser scanner.

Development of color and texture similarity measures

Color plays an important role in our daily lifes, and likewise it is an important source of information in the fields of image processing and computer vision. Perhaps the most fundamental operation involving colors is their judgment as to whether two colors are the same, or similar, or completely different. If they are similar, we would like to know how similar they are. For this purpose we need color similarity measures; these are mathematical formulae which assign to two test colors a real number in the range between 0 and 1: 1 for equal and 0 for completely different, and all numbers in between give the degree of similarity. In our laboratory, we evaluate existing color similarity measures and develop new ones according to various criteria that make these measures useful for various applications. For example, it is sometimes necessary to judge by computer whether a given color similar to the color of human skin (within the same ethnic group), and it is desirable to have the best color similarity measure available that can be used for this task. The following image shows several color similarity images which were computed with several different color similarity measures. Note the differences in their performance.

Development of wearable machine vision systems