KinectV2 - Extracting color object using depth camera and "DepthToColor" mapping

Hello !

First of all i'm sorry for the choppiness of my writing, still trying to improve my English writing ability.

I'm getting started with Processing v3 and I'd really appreciate some help on this. I've followed shiffman kinect tutorials, i'm currently using a kinectv2 and the KinectPV2 library. I'm trying to extract the closest color object from the color image by using the "MapDepthToColor" example of the library. I'm also trying to apply the shiffman threshold tutorials to extract objects located on a specific range of depth.

Here is the code :

import KinectPV2.*;
import g4p_controls.*;
KinectPV2 kinect;

//Threshold parameters - raw Data values from [0 - 4500]
float minThresh = 450;
float maxThresh = 650;

//RGB image
PImage imgC; //used to display depth image
PImage imgC2; //used to display color image

//Display window setup
GWindow window;

int [] depthZero;

//BUFFER ARRAY TO CLEAN DE PIXLES
PImage depthToColorImg;

void setup() {

  size(1024, 848, P3D);

  depthToColorImg = createImage(512, 424, PImage.RGB);
  // ARRAY OF 512*424 (217088);
  depthZero = new int[ KinectPV2.WIDTHDepth * KinectPV2.HEIGHTDepth];

  //SET THE ARRAY TO 0s
  for (int i = 0; i < KinectPV2.WIDTHDepth; i++) {
    for (int j = 0; j < KinectPV2.HEIGHTDepth; j++) {
      depthZero[424*i + j] = 0;
    }
  }

  //init kinect sensor
  kinect = new KinectPV2(this);
  kinect.enableDepthImg(true);
  kinect.enableColorImg(true);
  kinect.enablePointCloud(true); // to enable getMapDepthToColor()
  kinect.init();

  //create GUI
  createGUI();

  //get kinect depth image properties
  PImage imgD2 = kinect.getDepthImage();
  //create color image using depth frame width and height
  imgC = createImage(imgD2.width, imgD2.height, RGB);
}

void draw() {
  background(0);

  float [] mapDCT = kinect.getMapDepthToColor(); // length : 434,176

  //get the raw data from depth and color
  int [] colorRaw = kinect.getRawColor(); // Length : 2,073,600

  //clean de pixels
  PApplet.arrayCopy(depthZero, depthToColorImg.pixels);

  int count = 0;

  depthToColorImg.loadPixels();
  for (int i = 0; i < KinectPV2.WIDTHDepth; i++) {
    for (int j = 0; j < KinectPV2.HEIGHTDepth; j++) {

      //incoming pixels 512 x 424 with position in 1920 x 1080
      float valX = mapDCT[count * 2 + 0];
      float valY = mapDCT[count * 2 + 1];

      //maps the pixels to 512 x 424, not necessary but looks better
      int valXDepth = (int)((valX/1920.0) * 512.0);
      int valYDepth = (int)((valY/1080.0) * 424.0);

      int  valXColor = (int)(valX);
      int  valYColor = (int)(valY);

      if ( valXDepth >= 0 && valXDepth < 512 && valYDepth >= 0 && valYDepth < 424 &&
        valXColor >= 0 && valXColor < 1920 && valYColor >= 0 && valYColor < 1080) {
        color colorPixel = colorRaw[valYColor * 1920 + valXColor];
        //color colorPixel = depthRaw[valYDepth*512 + valXDepth];
        depthToColorImg.pixels[valYDepth * 512 + valXDepth] = colorPixel;
      }
      count++;
    }
  }

  // Depth image
  PImage imgD = kinect.getDepthImage();
  PImage imgC2 = kinect.getColorImage();
  // loadPixels allows to operate on the pixels on that image based on the raw depth
  imgC.loadPixels();
  imgC2.loadPixels();

  //used for threshold calibration
  //minThresh = map(mouseX, 0, width, 0 ,4500);
  //maxThresh = map(mouseX, 0, height, 0 ,4500);

  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepthData();

  for (int x = 0; x < imgD.width; x++) {
    for (int y = 0; y <imgD.height; y++) {

      // offset is an index that store an int value corresponding to a (x,y) pixel
      int offset = x + y * imgD.width;
      // d get the raw depth for the current offset value
      int d = depth[offset];

      // Thresholds for the object depth
      if (d > minThresh && d < maxThresh && x > 100 && y > 100) {
        imgC.pixels[offset] = color(255, 0, 150);
      } else {
        // By pressing the "extract background" button, the background pixels color values are changed to get a black background
        if (extractEventBtn.isEnabled() == false) {
          imgC.pixels[offset] = color(0);
          depthToColorImg.pixels[offset] = color(0);
        } else {
          imgC.pixels[offset] = imgD.pixels[offset];
          //depthToColorImg.pixels[offset] = imgD.pixels[offset];
        }
      }
    }
  }

  imgC.updatePixels();
  imgC2.updatePixels();
  depthToColorImg.updatePixels();
  //draw the image
  image(depthToColorImg, 0, 424);
  image(imgC, 0, 0);
  image(imgC2, 512, 0, 512, 424);

  fill(255);
  textSize(32);
  text(minThresh + " " + maxThresh, 10, 64);
  //ellipse(rx, ry, 32, 32);
}
![]()

Here is an example where i'm trying to extract only the color pixels of my arm:

Then i extract the background and i'm wondering about a good way to reduce the gap between the color and the depth image after the mapping so that we can see only the color pixels.

Thanks in advance for your help !