Project Description

Parallelization of a Polarization Image Processing System

FraunhoferIIScamera2

Download the White Paper about the use of the ARGO toolchain by Fraunhofer IIS:
Scilab Fraunhofer IIS White Paper ARGO – Parallelization of Polarization Image Processing Systems

Fraunhofer Use Case

This application is a specialized image processing system for image data originating from a novel polarization image sensor (POLKA) developed at Fraunhofer IIS [7] . This camera is used in industrial inspection, for example in in – line glass [8] and car- bon fiber [9] quality monitoring. Polarization image data is significantly different from ‘traditional’ (i.e. color) image data and requires widely different – and significantly more computation intensive – processing operations as shown in the following workflow:

IIS-workflow

Challenges

A gain/offset correction is performed on each pixel to equalize sensitivity and linearity inhomogeneity. For this purpose, additional calibration data is required (G/O input in the workflow above).

Since each pixel only provides a part of the polarization in- formation of the incoming photons, the unavailable information is interpolated from the surrounding pixels (similar to Bayer pattern interpolation on color image sen sors). From the interpolated pixel values we now compute the Stokes vector, which is a vector that provides the complete polarization information of each pixel. By appropriate transformations, the Stokes vectors are converted into the degree of linear polarization (DOLP) and angle of linear polarization (AOLP). These parameters are usually the starting point for any further application dependent processing (which is not shown here). For demonstration purposes, we convert AOLP and DOLP into a RGB color image that can be used for visualizing polarization effect.

Polarization image processing is currently used in industrial inspection. For example, inline glass inspection is depicted in the two pictures. Glass products are transported at up to 10 items per second and images are captured. Typically, a single inspection PC will handle multiple cameras and requires at least 20 fps processing capabilities. Currently, for one camera, this rate can be achieved, but in case of multiple camera outputs processed by one PC or in case of different use – cases where the number of output measurement frames increases, it can drop to 6 – 10 fps. This obligates for each use-case to reconsider/investi- gate further optimization possibilities. Our aim is to achieve a minimum of 25 fps as a hard constraint independent of use – case and processing elements in the algorithm chain. This is a hard constraint knowing that without any optimizations and parallelization, we can only achieve around 6 fps.

FraunhoferIIScamera-inaction

FraunhoferIIS-COTS

Solution

The picture on the left shows the POLKA Polarization Camera with glass measurements performed in a single shot per item. Since this is a measurement device, the precision of the measured data is of uttermost importance. Therefore, the standard algorithm is further adapted for each sensor and polarization data is further processed for different use cases. Especially trigonometric computation leads to a large computation overhead.

An alternative based on a number of COTS cam eras is shown in the picture on the right. This system complements the POLKA capabilities with increased spatial resolution and lower system cost. This construction, however, requires a dditional image fusion. The required registration and alignment further increase the compu- tational complexity of the measurement operation [10] .

In both cases, their underlying algorithms need to be adapted to each use case, starting from the Scilab high-level algorithmic description, all the way down to the embedded C / VHDL implementation.

FraunhoferIISuseCase

Results

With the complete toolchain starting from the Scilab code and ending up with parallelized C code on heterogeneous embedded systems, the results are:

  • Uniform computation over the consecutive steps of the workflow
  • Speedup ~3
  • No changes in Scilab necessary
  • High data locality: data can stay on a single core most of the time

schedulingIIS

parallelizationIIS

Last article

Download the paper presented at the conference ERTS 2018:
Interactive Parallelization of Embedded Real-Time Applications Starting from Open-Source Scilab & Xcos

References

 [7]  S. Junger, W. Tschekali nkij, N. Verwaal and N. Weber, “On – chip nanostructures for polarization imaging and multispectral sensing using dedicated layers of modified CMOS processes,” in Proceedings of the SPIE conference on Photonic and Phononic Properties of Engineered Nanostruct ures , 2011.
[8] A. Nowak, “In – line residual stress measurement,” Glass Worldwide, pp. 72 – 73, 2015.
[9] M. Schöberl, K. Kasnakli and A. Nowak, “Measuring Strand Orientation in Carbon Fiber Reinforced Plas – tics (CFRP) with Polarization,” in 19th World Conference on Non – Destructive Testing , Munich, 2016.

The ARGO Project

The goal of the ARGO (WCET-Aware Parallelization of Model-Based Applications for Heterogeneous Parallel Systems) research project is to develop a toolchain which translates  model-based Scilab/Xcos applications into multi-core optimized C code with guaranteed real-time constraints.

http://www.argo-project.eu/

 

Keywords:
Worst case execution time, embedded system, safety critical, model-based design

 

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688131

H2020