style-transfer

An image processing project where we implement style transfer, which is a type of content mixing of two images, via texture synthesis algorithms, we implemented Style Transfer based on Elad's et al. paper "Style-Transfer via Texture-Synthesis", which relies on classical methods. The original paper can be found at original, and our documentation is available at our doc.

The Proposed Algorithm

1. Segmentation
2. Building Gaussian Pyramids
3. Initialization
4. Color Transfer
5. Patch Matching
6. Robust Aggregation
7. Content Fusion
8. Denoise

Each step will be discussed in details.

1) Segmentation

Our exploration encompassed the implementation of different segmentation techniques. Initially, we employed GrabCut, as advocated in Elad's paper. However, GrabCut's reliance on user-defined foreground and background annotations often resulted in suboptimal performance, particularly when these annotations were absent, leading to degraded outcomes. Subsequently, an alternative edge-based segmentation technique was adopted. This technique involved a series of steps, starting with the application of the Canny edge detection algorithm, followed by dilation, contour extraction, and subsequent filling. Notably, this approach exhibited superior performance in multiple scenarios, showcasing improved outcomes compared to GrabCut in diverse contexts.

Segmentation mask using GrabCut algorithm	Segmentation mask using edge-Based Segmentation

2) Building Gaussian Pyramids

The process of constructing Gaussian pyramids involves downscaling the resolution of the content, style and segmentation mask images, creating multiple layers. Operating the style transfer algorithm on each of these layers is instrumental in enhancing the content image's capacity to assimilate broader stylistic features from the style image.

3) Initialization

Following the completion of pre-processing functions and pyramid construction, the algorithm initiates X, the primary image subjected to algorithmic operations. Initially set as the content image, X undergoes an additional procedural step wherein a significant amount of Gaussian noise is applied. The deliberate injection of Gaussian noise into the content image, serving as the basis for subsequent operations, fosters a more pronounced and authentic style transfer experience, minimizing repetitive patterns and enhancing the overall stylistic coherence within the transformed image.

Patch matching on uniform white image with no noise	Patch matching on uniform image with added noise

4) Color Transfer

The process of color transfer involves transposing the color palette from the style image onto the content image. Several methodologies were explored to accomplish this task, commencing with histogram matching, as advocated in Elad’s paper. While histogram matching showcased efficacy in many instances, it occasionally exhibited limitations, leading to instances of extreme color transfer. To mitigate these challenges and ensure a more nuanced and controlled color transformation, an alternative approach was adopted. Subsequently, color transfer in LAB channels was implemented as an alternative methodology.

Original image	Style

Color transfer using histogram matching	Color transfer using LAB channels.

5) Patch Matching

The crux of patch matching revolves around identifying the nearest patch in the style image that corresponds to the current patch in the content image. This fundamental process ensures that the changes introduced post-style transfer remain coherent with the original content. The minimization task aims to determine the most fitting patch in the style image, corresponding to each patch in the content image, using a Nearest Neighbor approach. However, the exhaustive nature of Nearest Neighbor for each patch incurs substantial computational overheads. To address the computational complexity while retaining fidelity in patch matching, Principal Component Analysis (PCA) was employed as an alternative.

Patch matching with patch sizes = [40, 30]	Patch matching with patch sizes = [20, 10]

6) Robust Aggregation

Robust Aggregation, a crucial procedure within the style transfer algorithm, involves the systematic adjustment of each patch in the content image. This adjustment aims to iteratively align each content patch closer to its matched counterpart in the style image. The methodology employed for this purpose, as elucidated in Kwatra's paper, relies on Iteratively Reweighted Least Squares (IRLS). This technique systematically refines the content patches by iteratively adjusting their attributes to converge towards the corresponding style patches.

7) Content Fusion

Content Fusion serves as the pivotal stage in the style transfer process, focusing on reapplying the content image in alignment with the segmentation outcome. This crucial step aims to ensure the preservation of significant content elements within the final stylized output.

8) Denoise

Denoising stands as a crucial step within the style transfer process, pivotal in guaranteeing seamless transitions within the resultant image. It plays a pivotal role in refining the overall visual quality by minimizing unwanted artifacts or noise. Throughout our experimentation, various denoising methodologies were rigorously assessed for their efficacy in achieving smooth transitions within the output image. Among the methodologies scrutinized, the Bilateral Filter emerged as the most proficient performer.

Experiment Results

Content	Style	Result

We also modified the algorithm to preserve the geometric shapes in the style images.

Content	Style	Result

©️Developers

Name	Email
Fares Atef	[email protected]
Ghaith Mohamed	[email protected]
Amr ElSheshtawy	[email protected]
Amr Magdy	[email protected]