人类视网膜的研究与在图像处理的应用 [OpenCV]

目标

这篇文章主要呈现了一个人类视网膜模型，用于展示一些有趣图像处理和增强的特性。在这篇文章中你将学到：

• 从你的视网膜中发掘两个主通道
• 视网膜模型的基本使用
• 视网膜处理的一些参数调整

总体概述

该模型源于 Jeanny Herault 在 Gipsa 的研究，这是一个关于使用  Listic (code maintainer) 进行图像处理的实验室。这并非完整的模型，但是已经可以呈现一些有趣的事情，这些可以用于增强图像的处理体验。该模型可以使用人类的视网膜信息：

Note : 示例图像能够从 OpenEXR网站 下载。示例中，在视网膜操作之前输入图像线性变换到0-255范围但每个通道的数值均为浮点型数。直方图的后5%被截断（去除了大部分错误HDR像素）。相似操作请查看示例 opencv/samples/cpp/OpenEXRimages_HDR_Retina_toneMapping.cpp。 下面说明值考虑在常用8位/通道图像

retina模型输出通道

retina模型给出了得益于上述操作的两个输出。

Retina用例

该模型基本上可以用于时空视频效果，但也可用于以下目的：

• 执行具有增强的信噪比的纹理分析以及针对输入图像亮度范围的鲁棒性增强的细节（检查出Parvocellular retina 通道输出）
• 利用先前引用的属性执行运动分析。

有关详细信息，请参阅以下文件：

• Benoit A., Caplier A., Durette B., Herault, J., “使用人类视觉系统建模的仿生低级图像处理”, Elsevier, 计算机视觉与图像理解 114 (2010), pp. 758-773. DOI < http://dx.doi.org/10.1016/j.cviu.2010.01.011 >
• 请看看Jeanny Herault的参考作品，你可以在他的书中阅读：

视觉：图像、信号和神经网络：视觉感知的神经处理模型 (神经处理研究进展),，作者: Jeanny Herault, ISBN: 9814273686. WAPI (Tower ID): 113266891.

这个retina过滤器代码包括 phd/research 同事的研究贡献，其中代码已由作者重写：

• 查看 retinacolor.hpp 模块以发现 Brice Chaix de Lavarene phD所写的关于色彩拼接/马赛克和他的参考文献： B. Chaix de Lavarene, D. Alleysson, B. Durette, J. Herault (2007). “通过递归过滤高效去马赛克”，IEEE国际图像处理会议ICIP 2007
• 查看 imagelogpolprojection.hpp 以发现 retina 空间日志采样，源自 Barthelemy Durette phd with Jeanny Herault. 视网膜/ V1皮层投影也被提出并且源于 Jeanny 的讨论. ====> 更多的信息在上面引用的Jeanny Heraults的书中。

代码教程

请参考文件中的原始教程源码

opencv_folder/samples/cpp/tutorial_code/contrib/retina_tutorial.cpp .

假设你已经正确安装OpenCV，使用如下命令进行编译。需要如下库进行编译：opencv_core（cv::Mat 和友元对象管理），opencv_highgui（图像/视频的读取和显示）以及opencv_contrib(Retina描述子）。

// compile
gcc retina_tutorial.cpp -o Retina_tuto -lopencv_core -lopencv_highgui -lopencv_contrib

// Run commands : add 'log' as a last parameter to apply a spatial log sampling (simulates retina sampling)
// run on webcam
./Retina_tuto -video
// run on video file
./Retina_tuto -video myVideo.avi
// run on an image
./Retina_tuto -image myPicture.jpg
// run on an image with log sampling
./Retina_tuto -image myPicture.jpg log

下面是代码的解释：

视网膜定义出现在 contrib 包中，并包含一个简单的使用示例：

#include "opencv2/opencv.hpp"

通过一个 help 函数为用户提供一些运行的提示：

// help程序
static void help(std::string errorMessage)
{
 std::cout<<"Program init error : "<<errorMessage<<std::endl;
 std::cout<<"\nProgram call procedure : retinaDemo [processing mode] [Optional : media target] [Optional LAST parameter: \"log\" to activate retina log sampling]"<<std::endl;
 std::cout<<"\t[processing mode] :"<<std::endl;
 std::cout<<"\t -image : for still image processing"<<std::endl;
 std::cout<<"\t -video : for video stream processing"<<std::endl;
 std::cout<<"\t[Optional : media target] :"<<std::endl;
 std::cout<<"\t if processing an image or video file, then, specify the path and filename of the target to process"<<std::endl;
 std::cout<<"\t leave empty if processing video stream coming from a connected video device"<<std::endl;
 std::cout<<"\t[Optional : activate retina log sampling] : an optional last parameter can be specified for retina spatial log sampling"<<std::endl;
 std::cout<<"\t set \"log\" without quotes to activate this sampling, output frame size will be divided by 4"<<std::endl;
 std::cout<<"\nExamples:"<<std::endl;
 std::cout<<"\t-Image processing : ./retinaDemo -image lena.jpg"<<std::endl;
 std::cout<<"\t-Image processing with log sampling : ./retinaDemo -image lena.jpg log"<<std::endl;
 std::cout<<"\t-Video processing : ./retinaDemo -video myMovie.mp4"<<std::endl;
 std::cout<<"\t-Live video processing : ./retinaDemo -video"<<std::endl;
 std::cout<<"\nPlease start again with new parameters"<<std::endl;
 std::cout<<"****************************************************"<<std::endl;
 std::cout<<" NOTE : this program generates the default retina parameters file 'RetinaDefaultParameters.xml'"<<std::endl;
 std::cout<<" => you can use this to fine tune parameters and load them if you save to file 'RetinaSpecificParameters.xml'"<<std::endl;
}

然后，启动主程序并首先声明一个加载输入图像的 cv::Mat 矩阵。 分配一个 cv::VideoCapture 对象准备加载视频流（如有必要）

int main(int argc, char* argv[]) {
  // 声明retina输入缓冲区...不同的输入媒体，其缓冲区不同
  cv::Mat inputFrame;
  cv::VideoCapture videoCapture; // 如果使用视频媒体，则会在此处声明其管理器

在主程序中，程序处理前，首先检查输入命令参数。 这里它加载来自单次加载的图像（如果用户选择命令 - 图像）或从视频流（如果用户选择命令 - 视频）的第一输入图像。 此外，如果用户在其程序调用结束时添加了 log  命令，则由 retina 执行的空间对数图像采样由布尔标志 useLogSampling 决定。

一旦所有输入参数被处理，第一个图像应该被加载，如果没有加载，则显示错误并停止程序：

if (inputFrame.empty())
{
    help("Input media could not be loaded, aborting");
    return -1;
}

现在，一切都准备好可以运行retina 模型了。 我建议在这里分配一个retina 实例和管理最终的日志采样选项。 Retina构造函数至少需要一个cv::Size对象，该对象显示需要管理的输入数据的大小。 可以激活其他选项，例如颜色及其相关的颜色复用策略（这里选择使用枚举cv :: RETINA_COLOR_BAYER的Bayer多路复用）。 如果使用日志采样，可以调整图像缩小因子（使输出图像更小）并且日志采样强度可调。

// pointer to a retina object
cv::Ptr<cv::Retina> myRetina;

// if the last parameter is 'log', then activate log sampling (favour foveal vision and subsamples peripheral vision)
if (useLogSampling)
{
    myRetina = new cv::Retina(inputFrame.size(), true, cv::RETINA_COLOR_BAYER, true, 2.0, 10.0);
}
else// -> else allocate "classical" retina :
    myRetina = new cv::Retina(inputFrame.size());

一旦完成，上述代码就会生成一个默认 xml 文件，包含视网膜的默认参数。这个可以让你很方便的在你的配置中使用这个模板。这里所生成的模板 xml 文件名为 RetinaDefaultParameters.xml .

// 保存默认视网膜参数文件，以便阅读并修改然后重载
myRetina->write("RetinaDefaultParameters.xml");

紧接着，程序视图加载另外一个名为 RetinaSpecificParameters.xml   的 XML 文件文件。如果你已经创建了这个文件并引入，那么该文件会被加载，否则将使用默认的视网膜参数。

// 如果文件存在则加载
myRetina->setup("RetinaSpecificParameters.xml");

这个文件并不是必须的，只是提供了这样一个选择，你可以清空视网膜缓冲区来强制清除以往事件。

// 重置所有视网膜缓冲区 (想想你闭上眼睛很长时间)
myRetina->clearBuffers();

现在可以运行视网膜程序了，首先我创建了一个输出缓冲区用于接收两个视网膜频道的输出：

// 声明retina输出缓冲区
cv::Mat retinaOutput_parvo;
cv::Mat retinaOutput_magno;

然后在循环中运行视网膜，根据需要从视频序列中加载新的帧，并将输出写回专用缓冲区。

// 无限循环
while(true)
{
    // 如果使用视频流，则抓取新帧，否则输入保持不变
    if (videoCapture.isOpened())
        videoCapture>>inputFrame;

    //在加载的输入帧上运行retina滤波器
    myRetina->run(inputFrame);
    // 检索并显示retina输出
    myRetina->getParvo(retinaOutput_parvo);
    myRetina->getMagno(retinaOutput_magno);
    cv::imshow("retina input", inputFrame);
    cv::imshow("Retina Parvo", retinaOutput_parvo);
    cv::imshow("Retina Magno", retinaOutput_magno);
    cv::waitKey(10);
}

这样就完成了。但是如果你想确保系统安全，请小心管理异常。该程序在一些无效数据会发生异常，例如没有输入帧、错误的设置等等。接下来我建议使用 try catch 代码块来执行视网膜处理代码，如下所示：

try{
     // pointer to a retina object
     cv::Ptr<cv::Retina> myRetina;
     [---]
     // processing loop with no stop condition
     while(true)
     {
         [---]
     }

}catch(cv::Exception e)
{
    std::cerr<<"Error using Retina : "<<e.what()<<std::endl;
}

视网膜参数该如何处理？

首先，推荐如下参考论文：

• Benoit A., Caplier A., Durette B., Herault, J., “Using Human Visual System Modeling For Bio-Inspired Low Level Image Processing” , Elsevier, Computer Vision and Image Understanding 114 (2010), pp. 758-773. DOI < http://dx.doi.org/10.1016/j.cviu.2010.01.011 >

然后打开上述示例代码生成的配置文件 RetinaDefaultParameters.xml ，内容如下：

<?xml version="1.0"?>
<opencv_storage>
<OPLandIPLparvo>
  <colorMode>1</colorMode>
  <normaliseOutput>1</normaliseOutput>
  <photoreceptorsLocalAdaptationSensitivity>7.0e-01</photoreceptorsLocalAdaptationSensitivity>
  <photoreceptorsTemporalConstant>5.0e-01</photoreceptorsTemporalConstant>
  <photoreceptorsSpatialConstant>5.3e-01</photoreceptorsSpatialConstant>
  <horizontalCellsGain>0.</horizontalCellsGain>
  <hcellsTemporalConstant>1.</hcellsTemporalConstant>
  <hcellsSpatialConstant>7.</hcellsSpatialConstant>
  <ganglionCellsSensitivity>7.0e-01</ganglionCellsSensitivity></OPLandIPLparvo>
<IPLmagno>
  <normaliseOutput>1</normaliseOutput>
  <parasolCells_beta>0.</parasolCells_beta>
  <parasolCells_tau>0.</parasolCells_tau>
  <parasolCells_k>7.</parasolCells_k>
  <amacrinCellsTemporalCutFrequency>1.2e+00</amacrinCellsTemporalCutFrequency>
  <V0CompressionParameter>9.5e-01</V0CompressionParameter>
  <localAdaptintegration_tau>0.</localAdaptintegration_tau>
  <localAdaptintegration_k>7.</localAdaptintegration_k></IPLmagno>
</opencv_storage>

基本参数

最简单的参数如下：

• colorMode : 让retina 处理颜色信息（如果为1）或灰度图像（如果为0）。 在后一种情况下，将只处理输入的第一个通道。
• normaliseOutput : 每个通道都有此参数，如果值为1，则关注的通道输出在0和255之间重新缩放。在这种情况下，请注意Magnocellular的输出电平（运动/瞬态通道检测）。 残留噪声也将重新调整！

注意:使用颜色需要彩色通道多路复用/解复用，这需要更多的处理。 您可以使用灰度级获得更快的处理速度：对于所有视网膜处理，每像素大约获得30倍的速度，并且最近已针对多核架构进行了并行处理。

Photo-receptors parameters

The following parameters act on the entry point of the retina - photo-receptors - and impact all the following processes. These sensors are low pass spatio-temporal filters that smooth temporal and spatial data and also adjust there sensitivity to local luminance thus improving details extraction and high frequency noise canceling.

• photoreceptorsLocalAdaptationSensitivity between 0 and 1. Values close to 1 allow high luminance log compression effect at the photo-receptors level. Values closer to 0 give a more linear sensitivity. Increased alone, it can burn the Parvo (details channel) output image. If adjusted in collaboration with ganglionCellsSensitivity images can be very contrasted whatever the local luminance there is... at the price of a naturalness decrease.
• photoreceptorsTemporalConstant this setups the temporal constant of the low pass filter effect at the entry of the retina. High value lead to strong temporal smoothing effect : moving objects are blurred and can disappear while static object are favored. But when starting the retina processing, stable state is reached lately.
• photoreceptorsSpatialConstant specifies the spatial constant related to photo-receptors low pass filter effect. This parameters specify the minimum allowed spatial signal period allowed in the following. Typically, this filter should cut high frequency noise. Then a 0 value doesn’t cut anything noise while higher values start to cut high spatial frequencies and more and more lower frequencies... Then, do not go to high if you wanna see some details of the input images ! A good compromise for color images is 0.53 since this won’t affect too much the color spectrum. Higher values would lead to gray and blurred output images.

Horizontal cells parameters

This parameter set tunes the neural network connected to the photo-receptors, the horizontal cells. It modulates photo-receptors sensitivity and completes the processing for final spectral whitening (part of the spatial band pass effect thus favoring visual details enhancement).

• horizontalCellsGain here is a critical parameter ! If you are not interested by the mean luminance and focus on details enhancement, then, set to zero. But if you want to keep some environment luminance data, let some low spatial frequencies pass into the system and set a higher value (<1).
• hcellsTemporalConstant similar to photo-receptors, this acts on the temporal constant of a low pass temporal filter that smooths input data. Here, a high value generates a high retina after effect while a lower value makes the retina more reactive.
• hcellsSpatialConstant is the spatial constant of the low pass filter of these cells filter. It specifies the lowest spatial frequency allowed in the following. Visually, a high value leads to very low spatial frequencies processing and leads to salient halo effects. Lower values reduce this effect but the limit is : do not go lower than the value of photoreceptorsSpatialConstant . Those 2 parameters actually specify the spatial band-pass of the retina.

NOTEafter the processing managed by the previous parameters, input data is cleaned from noise and luminance in already partly enhanced. The following parameters act on the last processing stages of the two outing retina signals.

Parvo (details channel) dedicated parameter

• ganglionCellsSensitivity specifies the strength of the final local adaptation occurring at the output of this details dedicated channel. Parameter values remain between 0 and 1. Low value tend to give a linear response while higher values enforces the remaining low contrasted areas.

Note :this parameter can correct eventual burned images by favoring low energetic details of the visual scene, even in bright areas.

IPL Magno (motion/transient channel) parameters

Once image information is cleaned, this channel acts as a high pass temporal filter that only selects signals related to transient signals (events, motion, etc.). A low pass spatial filter smooths extracted transient data and a final logarithmic compression enhances low transient events thus enhancing event sensitivity.

• parasolCells_beta generally set to zero, can be considered as an amplifier gain at the entry point of this processing stage. Generally set to 0.
• parasolCells_tau the temporal smoothing effect that can be added
• parasolCells_k the spatial constant of the spatial filtering effect, set it at a high value to favor low spatial frequency signals that are lower subject to residual noise.
• amacrinCellsTemporalCutFrequency specifies the temporal constant of the high pass filter. High values let slow transient events to be selected.
• V0CompressionParameter specifies the strength of the log compression. Similar behaviors to previous description but here it enforces sensitivity of transient events.
• localAdaptintegration_tau generally set to 0, no real use here actually
• localAdaptintegration_k specifies the size of the area on which local adaptation is performed. Low values lead to short range local adaptation (higher sensitivity to noise), high values secure log compression.