[widget] Understanding Limited Palette’s Color Control Settings
Contents
[widget] Understanding Limited Palette’s Color Control Settings¶
The widget below illustrates how images generated in “Limited Palette” mode are affected by changes to color control settings.
Press the “▷” icon to begin the animation.
The first run with any particular set of settings will probably show an empty image because the widget is janky and downloads only what it needs on the fly. What can I say: I’m an ML engineer, not a webdeveloper.
What is “Limited Palette” mode?¶
In “Unlimited Palette” mode, pytti directly optimizes pixel values to try to maximize the similarity between the generated image and the input prompts. Limited Palette mode uses this same process, but adds additional constraints on how the colors in the image (i.e. the pixel values) are selected.
We start by specifying a number of “palettes”. In this context, you can think of a palette as a container with a fixed number of slots, where each slot holds a single color. During optimization steps, colors which are all members of ths same “palette” container are optimized together. This has the effect that the “palette” objects become sort of “attached” to semantic objects in the image. Let’s say for example you have an init image of an ocean horizon, so half of the picture is water and half of it is the sky. If we set the number of palettes to 2, chances are one palette will primarily carry colors for painting the ocean and the other will carry colors for painting the sky. This is not a hard-and-fast rule, but you should anticipate that palette size settings will interact with the diversity of semantic content in the generated images.
For advice and additional insights about palette and color behaviors in pytti, we recommend the community document Way of the TTI Artist by oxysoft#6139 and collaborators.
Description of Settings in Widget¶
All settings except smoothing_weight are specific to Limited Palette mode.
palette_size: Number of colors in each palette.palettes: Total number of palettes. The image will have palette_size*palettes colors total.gamma: Relative gamma value. Higher values make the image darker and higher contrast, lower values make the image lighter and lower contrast.hdr_weight: How strongly the optimizer will maintain the gamma. Set to 0 to disable.palette_normalization_weight: How strongly the optimizer will maintain the palettes’ presence in the image. Prevents the image from losing palettes.smoothing_weight: Makes the image smoother using “total variation loss” (old-school image denoising). Can also be negative for that deep fried look.
Widget¶
import re
from pathlib import Path
import pandas as pd
import panel as pn
pn.extension()
outputs_root = Path('images_out')
folder_prefix = 'permutations_limited_palette_2D'
folders = list(outputs_root.glob(f'{folder_prefix}_*'))
def format_val(v):
try:
v = float(v)
if int(v) == v:
v = int(v)
except:
pass
return v
def parse_folder_name(folder):
metadata_string = folder.name[1+len(folder_prefix):]
pattern = r"_?([a-zA-Z_]+)-([0-9.]+)"
matches = re.findall(pattern, metadata_string)
d_ = {k:format_val(v) for k,v in matches}
d_['fpath'] = folder
d_['n_images'] = len(list(folder.glob('*.png')))
return d_
df_meta = pd.DataFrame([parse_folder_name(f) for f in folders])
variant_names = [v for v in df_meta.columns.tolist() if v not in ['fpath']]
variant_ranges = {v:df_meta[v].unique() for v in variant_names}
[v.sort() for v in variant_ranges.values()]
###########################
url_prefix = "https://raw.githubusercontent.com/dmarx/pytti-settings-test/main/images_out/"
image_paths = [str(p) for p in Path('images_out').glob('**/*.png')]
d_image_urls = {im_path:im_path.replace('images_out/', url_prefix) for im_path in image_paths}
###########################
n_imgs_per_group = 40
kargs = {k:pn.widgets.DiscreteSlider(name=k, options=list(v), value=v[0]) for k,v in variant_ranges.items()}
kargs['i'] = pn.widgets.Player(interval=100, name='step', start=1, end=n_imgs_per_group, step=1, value=1, loop_policy='reflect')
@pn.interact(
**kargs
)
def display_images(
palettes,
palette_size,
gamma,
hdr_weight,
smoothing_weight,
palette_normalization_weight,
i,
):
folder = df_meta[
(palettes == df_meta['palettes']) &
(palette_size == df_meta['palette_size']) &
(gamma == df_meta['gamma']) &
(hdr_weight == df_meta['hdr_weight']) &
(smoothing_weight == df_meta['smoothing_weight']) &
(palette_normalization_weight == df_meta['palette_normalization_weight'])
]['fpath'].values[0]
im_path = str(folder / f"{folder.name}_{i}.png")
im_url = d_image_urls[im_path]
return pn.pane.HTML(f'<img src="{im_url}" width="700">', width=700, height=350, sizing_mode='fixed')
pn.panel(display_images).embed(max_opts=n_imgs_per_group, max_states=999999999)