Seam carving

Original image to be made narrower
Scaling is undesirable because the castle is distorted.
Cropping is undesirable because part of the castle is removed.
Seam carving

Seam carving (or liquid rescaling) is an algorithm for content-aware image resizing, developed by Shai Avidan, of Mitsubishi Electric Research Laboratories (MERL), and Ariel Shamir, of the Interdisciplinary Center and MERL. It functions by establishing a number of seams (paths of least importance) in an image and automatically removes seams to reduce image size or inserts seams to extend it. Seam carving also allows manually defining areas in which pixels may not be modified, and features the ability to remove whole objects from photographs.

The purpose of the algorithm is image retargeting, which is the problem of displaying images without distortion on media of various sizes (cell phones, projection screens) using document standards, like HTML, that already support dynamic changes in page layout and text but not images.[1]

Image Retargeting was invented by Vidya Setlur, Saeko Takage, Ramesh Raskar, Michael Gleicher and Bruce Gooch in 2005.[2] The work by Setlur et al. won the 10-year impact award in 2015.

Seams

Seams can be either vertical or horizontal. A vertical seam is a path of pixels connected from top to bottom in an image with one pixel in each row.[1] A horizontal seam is similar with the exception of the connection being from left to right. The importance/energy function values a pixel by measuring its contrast with its neighbor pixels.

Computing seams

Computing the seam consists of finding the path of minimum energy cost from one end of the image to another. This can be done via Dijkstra's algorithm, dynamic programming, greedy algorithm or graph cuts among others.[1]

Dynamic programming

Dynamic programming is a programming method that stores the results of sub-calculations in order to simplify calculating a more complex result. Dynamic programming is used in seam carving for computing seams.

If attempting to compute a vertical seam (path) of lowest energy, for each pixel in a row we compute the energy of the current pixel plus the energy of one of the three possible pixels above it.

This is better described by this image:

DynamicProgrammingLeastEnergyPathA

The first row has no rows above it, so the sum (black) is just the energy value of the current pixel (red).

The second row, if we look at the second pixel for example, we see its energy value is 2 (red). If we look above it, it has a choice of either 1, 4, or 3 (black). Since 1 is the minimum number of the three values, we ignore the other two and set the sum of the pixel to its energy value which is 2 (red) plus 1 (black).

After the above operation is carried out for every pixel in the second row, we go to the third row:

DynamicProgrammingLeastEnergyPathB

We repeat the process from row two in row three to end up with the final cumulative sums for the seams/paths. The lowest value or values are the seams with the lowest energy, which would be in this example the seams with '5' in the last row.

To trace the seam/path, work from the last row and follow the green arrows:

DynamicProgrammingLeastEnergyPathC

More specifically, find the pixel on the last row with the minimum cumulative energy and use this as the starting pixel. Then from the three pixels above it choose the one with the minimum cumulative energy and so on.

Algorithm

1) Starting with an image such as:

BroadwayTowerSeamCarvingA

2) The weight/density/energy of each pixel is then calculated. This can be done by various algorithms: gradient magnitude, entropy, visual saliency, eye-gaze movement.[1] In this example gradient magnitude gives 'satisfactory results.'

BroadwayTowerSeamCarvingB

3) After having the energy of the image, a list of seams is generated. Seams are ranked by energy, with low energy seams being of least importance to the content of the image. Seams can then be calculated via the dynamic programming approach.

Seams shown with the energy function:

BroadwayTowerSeamCarvingC

Seams shown with the original image:

BroadwayTowerSeamCarvingD

4) Then the seams are removed from the image, reducing the size of the image as a result:

BroadwayTowerSeamCarvingE

The recalculated energy function of the image would be:

BroadwayTowerSeamCarvingF

Issues

  1. The algorithm may need user-provided information to reduce errors. This can consist of painting the regions which are to be preserved. With human faces it is possible to use face detection.
  2. Sometimes the algorithm, by removing a low energy seam, may end up inadvertently creating a seam of higher energy. The solution to this is to simulate a removal of a seam, and then check the energy delta to see if the energy increases. If it does, prefer other seams instead.[3]

Implementations

Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages to compare the original image (top), its width rescaled to the percentage using seam-carving (middle), and rescaled to the same size using interpolation (bottom).
Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages as above. Note that the faces are affected less than their surroundings.

Adobe Systems acquired a non-exclusive license to seam carving technology from MERL,[4] and implemented it as a feature in Photoshop CS4, where it is called Content Aware Scaling.[5] As the license is non-exclusive, other popular computer graphics applications, among which are GIMP, digiKam, ImageMagick, as well as some stand-alone programs, among which are iResizer,[6] also have implementations of this technique, some of which are released as free and open source software.[7][8][9]

Improvements and extensions

  • Better energy function and application to video.[3]
  • Multi-operator: Combine with cropping and scaling.[10]
  • Much faster removal of multiple seams[11]
  • On-demand, server-side seam carving [12]

A 2010 review of eight image retargeting methods found that seam carving produced output that was ranked among the worst of the tested algorithms. It was, however, a part of one of the highest-ranking algorithms: the multi-operator extension mentioned above (combined with cropping and scaling).[13]

The content aware scale is frequently used for memey YouTube videos.[14][15][16]

See also

References

  1. 1 2 3 4 Seam Carving for Content-Aware Image Resizing. Shai Avidan, Ariel Shamir, 2007.
  2. ↑ http://dl.acm.org/citation.cfm?id=1149499
  3. 1 2 Improved Seam Carving for Video Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2008.
  4. ↑ Mitsubishi Electric press release, Business Wire, December 16, 2008.
  5. ↑ Adobe Photoshop CS4 new feature list.
  6. ↑ iResizer Content aware image resizing software by Teorex
  7. ↑ Liquid Rescale, seam carving plug-in for GIMP
  8. ↑ Announcement of inclusion in digiKam
  9. ↑ Seam carving capability included in ImageMagick
  10. ↑ Multi-operator Media Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2009.
  11. ↑ Real-time content-aware image resizing Science in China Series F: Information Sciences, 2009 SCIENCE IN CHINA PRESS. Archived July 7, 2011, at the Wayback Machine.
  12. ↑ On-demand, server-side seam carving based on CAIR
  13. ↑ Rubinstein, Michael; Gutierrez, Diego; Sorkine, Olga; Shamir, Ariel (2010). "A Comparative Study of Image Retargeting" (PDF). ACM Transactions on Graphics. 29 (5). See also the RetargetMe benchmark.
  14. ↑ punpun (2016-11-21), We Are Number One but the word 'one' makes the content aware scale stronger [English Subtitles], retrieved 2016-12-19
  15. ↑ punpun (2016-11-27), The bee movie trailer but every time they say bee the content aware scale gets stronger, retrieved 2016-12-19
  16. ↑ TeslaCRM (2016-11-27), The Mine Song but every "mine" increases content aware scale., retrieved 2016-12-19
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.