CS 639 Project - Blemish Removal

Progress since the proposal.
In the last proposal we thought of enhancing the algorithm to make it more inclusive for various skin tones, however, we decided to change our focus slightly to make the algorithm detect blemishes automatically. This is because when we tested the code for blemish removal on different skin tones, we realized that it works equally well for both images. However, the code does not automatically detect blemishes; the user has to click on whatever spot they want to be edited, and then the program enhances that area. This will be a starting point for us to try and improve the existing solution we are using.

Currently, the Python implementation we are applying uses a Sobel filter along with some best patch detection. It uses functions from cv2 (OpenCV) and numpy. For examples of this solution in action, view the before and after images section. To try and improve upon this solution, we have found research papers with possible changes to implement. These papers are some of our initial findings and may or may not contribute to the final solution. For more information regarding these papers, view the research section.

Briefly explain what problem you are trying to solve.
The problem we are trying to solve is that when we take portraits or any close-up pictures, our blemishes or acne are seen clearly. This is something that many people want to smoothen out or remove completely from photos to make themselves look acne free. The difficulties of this problem come from the desire to have the blemishes removed and skin accurately replaced as if there was no blemish at all.

Why is this problem important?
This problem is relevant today because many people utilize photo-editing apps, so it would be interesting to see how we could apply concepts we have learned in the course so far to make our own feature to smooth out blemishes or enhance already existing implementations. It is also applicable as using funny or appearance altering filters become more prevalent on social media apps like Snapchat and TikTok. Since this feature is already prevalent today, we want to work towards improving the feature. We hope to do so by experimenting with various filtering techniques that we found while researching our project.

What is the current state-of-the-art?
The current state-of-the-art for beautification apps and filters are live effects that adjust to your face's dimensions and color. These apps/filters will automatically detect specific features and modify them. Examples being, making your nose smaller, lips brighter, eyebrows narrower, acne removed, etc. These are done with advanced detection algorithms using varying techniques.

Some of these methods involve Gaussian Mixture Models, convolutional neural networks, Fast Global Smoothing Filter, Mutually Guided Filter, Attribute-aware Dynamic Guided Filter, and many more⁽¹⁾. These filters do not have to be live changing though, and many apps will take a picture and apply thorough filters. Even most native camera apps on phones have basic filters that adjust exposure, saturation, etc.

Are you planning on re-implementing an existing solution, or propose your own new approach?
We are planning on re-implementing an existing solution using Sobel Edge Detection. Additionally, we plan on exploring other edge detection algorithms that are mentioned in the research papers we are studying as well as class lectures. We read through various research papers and learned about different filtering techniques such as Box Blurring, Gaussian Blurring, Differential Privacy Blurring, Gabor filter, Circle Detection, and Hough Transformation. We plan on understanding those algorithms and detecting which of these makes most sense for our project and experiment with those. We are particularly interested in experimenting with circle detection as well as the hough transformation to start off with.

If you are proposing your own approach, why do you think existing approaches cannot adequately solve this problem? Why do you think your solution will work better?
In the short term, we are not proposing a completely new approach. We are moreso looking to enhance the current algorithm by making it automatically detect blemishes rather than having to manually select each blemish. As of now, this existing approach is not the best because it does not have the option to just remove all blemishes at once, and it requires extra input from the user's end, which means it takes more time. In this way, we are hoping that once we add our improvements, the program will work better and more efficiently.

How will you evaluate the performance of your solution? What results and comparisons are you eventually planning to show? Include a time-line that you would like to follow.
We will evaluate the performance of the solution by testing it with images to see if the application is able to detect blemishes and then filter out those spots, instead of the current state where we are manually filtering out the pictures. We have not created a concrete metric for this yet, but will hopefully develop a more rigorous evaluation as the project progresses.

We plan on meeting once a week to discuss our project implementation as well as work on the code to improve the existing applications. The week before the project presentation date, we will meet more frequently to make sure we are prepared to submit the project and present.

1. "FabSoften: Face Beautification via Dynamic Skin Smoothing, Guided Feathering, and Texture Restoration" [Link]
   • This paper goes over the implementation of a smoothing filter method that is named attribute-aware dynamic smoothing. It is guided by facial attributes like the number of blemishes, skin texture, skin color, etc. It also details comparisons to other state-of-the-art techniques.
2. "Different Approaches to Blurring Digital Images and Their Effect on Facial Detection" [Link]
   • This source discusses the various types of image blurring techniques as well as their effect on facial detection. Specifically, it talks about how privacy is being compromised because attackers use facial detection techniques for potential identity detection. While this is not the premise of our project, this paper is still useful because it goes through various techniques which can be used for blurring faces, and that is applicable to our project.
   • This paper goes through different blurring techniques such as Box Blurring, Gaussian Blurring, and Differential Privacy Blurring. It compares these techniques using data sets and evaluates which is the best.
   • Low-Pass Filters and High-Pass Filters are also explained, and this allows us to understand edge detection as well as how the blurring techniques relate to these types of filters.
   • Lastly, the paper touches on how matrices are involved in each blurring technique, so it provides a mathematical background for us to study as well.
3. "Computer Vision with Machine Learning on Smartphones for Beauty Applications" [Link]
   • This paper gives a brief background on facial filtering and object detection through different applications such as makeup transfer and blemish removal. Chapter 5 of this paper discusses wrinkle and blemish removal through different filtering techniques. It specifically talks about the Gabor filter and explains the algorithm as well as illustrates some of the code implementations. The author discusses the background of the Gabor filter and how to estimate skin scores. They also show how the code can be implemented to give certain results. This chapter also mentions the shortcomings of the Gabor filter and what other methods are currently being researched for further improvements in the future.
   • This article provided us with foundational knowledge as it gave us more background on the topic and walked us through the algorithms. The current application that we are basing our project on uses the Sobel filter and this research paper gave us a new filtering method that we can test out. Our goal is to find a filtering technique that works best for blemish removal and this paper furthered our discussion on what other methods we can experiment with before picking one that works most effectively.
   • Low-Pass Filters and High-Pass Filters are also explained, and this allows us to understand edge detection as well as how the blurring techniques relate to these types of filters.
4. "A Fast Multi-Modal Approach to Facial Feature Detection" [Link]
   • This paper overviews a method to help identify eyes, nose, and the mouth on faces by using 2D information. This allows facial features to be identified quickly and precisely without compromising too much on robustness. The paper presents the algorithm the researchers used and how each aspect of the algorithm (2D and 3D) affected the results.

A major hurdle we had to overcome was figuring out how to identify the best patch given the area that it is in. After digging deeper through research papers and studies, we came across circle detection as a means to help us in this process

We also struggled to connect the detection piece with the actual filtering method. We were able to implement an application that would detect all the acne spots using circle detection but we struggled to get those blemishes removed using the sobel filter.

One of the major issues that we came across in our automatic detection implementation was that it has trouble detecting white heads. In the images where the acne is red with a white bump the filter does not do much. This is something that we would like our project to continue to improve towards for the future. Being able to detect discolorations of varying forms is a key aspect in taking this project to the next level.

Another future implementation for this project could be to make a live filter that works on input directly from a camera — similar to snapchat, instagram or tiktok filters