Simplifying Point Cloud Labeling with Contextual Images and Point Cloud Filtering

Annotating point clouds from multi-line 360° LiDAR is exceedingly difficult. Providing context in the form of camera frames and limiting the point cloud to the Field Of View (FOV) of the camera simplifies things. To achieve this, we first had to replace our old, and not so stable LiDAR mount, with a sturdier one capable of also holding a camera.

With the LiDAR and the camera closer together, the next step was to synchronize and store the data coming from both sensors. The collected data was huge, so a simple Python script was created to allow for the selection of sequences of interest. Once the sequences were visually selected, they were saved in a format that the annotation tool CVAT can understand.

It was noted that although now CVAT provided the camera frames as context to annotate the LiDAR point clouds, the point clouds were still too large (360° horizontally, 32° vertically). It was not easy to know which part of the cloud corresponded to the camera frame (visual context), and many objects were still hard to identify.

To solve the issue, a C++ program using the Point Cloud Library (PCL) was created. PCL’s FrustumCulling filter was used for this purpose. The filter lets the user define a vertical and horizontal Field Of View (FOV), as well as the near and far plane distance. After some testing, the best parameters were defined to approximate the FOV of the camera. The points of the input cloud that fall outside of the approximated FOV are filtered out, and the points of the output closely match what the camera sees. This greatly facilitates the annotation of objects in the point cloud. Watch the above video for more exciting details.

Demonstrating SEDRAD, The Self Driving Ads Robot at AppWorks.

On Saturday, May 14, 2022, we demonstrated SEDRAD at the AppWorks offices in Taipei, Taiwan. The goal was to get approval to use the robot during their upcoming Demo Day #24. The demonstration was a big success and SEDRAD is set to navigate autonomously while showing information about the participating startups in the event.

AppWorks is the leading startup accelerator in Taiwan. It helps early-stage startups with resources and advice and facilitates their access to industry experts and potential investors. AppWorks admits two batches of startups per year, and at the end of each period, they hold their awaited Demo Day. We are excited to showcase SEDRAD during the upcoming event. Stay tuned for more information.

Testing 3D Annotation Tools

Before you can train a supervised Deep Learning model, you must first label your data.

Today I am testing Intel OpenVINO’s CVAT and MATLAB’s Lidar Labeler annotation tools for 3D data. First impressions, CVAT makes it easier to navigate the point cloud, but a small bug makes it hard to place the initial cuboid, making things a little slower overall. Lidar Labeler’s navigation is a little more difficult, but because it has no bug when placing the cuboid, it overall becomes faster to use. CVAT being free, if they were to fix the bug, it would become the preferred tool for now.

#DeepLearning #MachineLearning #AI #SelfDrivingCars #LiDAR

RTK Heading + LiDAR (Temporary) Mount Ready

After a few days of playing with some Makeblock (blue) metal pieces, I finally created a temporary mount for my RTK Heading (Dual GNSS) + 32-beans LiDAR system. It should be enough to test the sensors while a more stable one is built. I also conducted a quick indoor test of the LiDAR, it has been raining for two weeks so no chance to go outdoors yet.

SEDRAD, The Self-Driving Ads Robot is Finally Here

I am pleased to announce that the first version of SEDRAD, The Self-Driving Ads Robot, is finally here. I have released it as part of the final submission of the OpenCV Spatial AI Competition #Oak2021. Watch the video to learn more about what SEDRAD is capable of doing, and if you have any questions, don’t hesitate to contact me.

Drivable Path Segmentation Test 1

A couple of weeks ago I was collecting and labeling driving images to teach #SEDRAD how to recognize the surface to drive on using semantic segmentation.

The first deeplabv3+ model running fully on Oak-D cameras is ready and we took it for a spin. It is not perfect but it is a first step towards improving the safety of our #SelfDriving #Ads #Robot.

#Oak2021 #OpenCV #robotics #AI #MachineLearning #SuperAnnotate #autonomousdriving

Our Face Analysis Now Powered by Oak-D

An old Soul Hackers Labs’ trick now powered by Oak-D. People tracking and face detection happens on edge device. The detected face is fed to SHL’s face analyzer (on host) to determine the age, gender, emotions, attention level, viewing time of the tracked person. Metadata is produced and stored to generate useful reports for advertisers.

Tracking and detecting faces was the most resource consuming part and now the host computer has been freed from this burden, thanks Oak-D!

#SEDRAD, the Self-Driving Ads Robot coming soon! #Oak2021 #OpenCV #AI #machinelearning #robotics #retailanalytics #digitalsignage #digitalsignagesolutions

Annotation Process Under Way

One of the steps of teaching the Self-Driving Ads Robot (SEDRAD) how to navigate the environment is to teach it which surfaces are good for it to drive on. We began the tedious task of collecting images of the surrounding, and creating a segmentation of the images to later “teach” SEDRAD to recognize the surface to follow and stay on. This process comprises image acquisition, image annotation (segmenting in our case), and then segmentation validation. It is very time consuming. We also ran into a problem. Due to technical issues, the real SEDRAD was not available over the weekend, when we collected the data. Instead, we mounted the cameras at the same height and separation from each other on a wagon. Annotation here we go!