[Oiio-dev] Handling unassociated alpha

[Troy Sobotka]

You said in a post "It's wrong in so many ways. I can expand on this
idea if any of you don't know why."

I'd love to hear your official take on associated versus unassociated
alpha and why. Given the need to unassociate alpha for color
management and other linear operation, is there an optimal approach
you can envision?

Thanks for your time,
TJS

[Larry Gritz]

Post by Troy Sobotka
You said in a post "It's wrong in so many ways. I can expand on this
idea if any of you don't know why."
I'd love to hear your official take on associated versus unassociated
alpha and why.

Well, this is hardly "official", but here's my take:

If you think of image synthesis as akin to what a camera does, and the image like a photograph where the pixel values are proportional to how many photons hit the detector in each bin, then associated alpha is what you want -- the "color" values really are how bright each pixel should appear. (If you are using a linear color mapping of value to intensity, which I also recommend under all circumstances.) "Alpha" in this case is is an extra piece of information, that has no direct analog in photography, and tells you what the "coverage" is for each pixel, in case you want to do compositing of images. But if you strip the alpha off, the color still makes total sense as the brightness at each pixel, and you can display the image using only the information in the color channels (with proper compensation for the color response for your display).

Math is all linear and easy with associated alpha. The compositing operation "O = A over B" is very straightforward, and is identical in all channels:

Or = Ar + (1-Aa) * Br;
Og = Ag + (1-Aa) * Bg;
Ob = Ab + (1-Aa) * Bb;
Oa = Aa + (1-Aa) * Ba;

If you want to express your math as vectors/tuples,

O = A + (1-Aa) * B;

Nice!

And because the RGB values represent the amount of light hitting the detector, it makes perfect sense for it to be 0 (no light hit), regardless of how much of the background is covered, and it also makes sense to have values > 1 (there is no limit to how bright it can be) and for that matter, rgb > alpha. As a simple example, the color (.5, .5, 0, 0) might be used to represent a yellow glow that adds light without blocking any of the background at all.

With the other approach, the RGB channels do not mean much on their own, certainly not anything analogous to light and cameras. The "alpha" is a mask, so the color of each pixel is actually RGB*A. This has some strange consequences:

1. The over operation is a little funny now:

Or = Ar*Aa + (1-Aa) * Br;
Og = Ag*Aa + (1-Aa) * Bg;
Ob = Ab*Aa + (1-Aa) * Bb;
Oa = Aa + (1-Aa) * Ba;

It's both more math (who cares now, but it used to be helpful) but it's also less elegant -- you can't easily express it in vector notation as cleanly, and the math done to get the new alpha is different than what you do for the color channels. This is good circumstantial evidence that we're not working with as clean an abstraction.

2. Suddenly, it is no longer possible to express the "layer that adds yellow emissive light without blocking what's behind it." There is no unassociated (r,g,b,0) that can be yellow, since the amount of light is always the "color" multiplied by the "mask". If the mask is 0, so is the amount of light, by definition.

Post by Troy Sobotka
Given the need to unassociate alpha for color
management and other linear operation, is there an optimal approach
you can envision?

I'm not sure I know which operations you mean. Can you give some examples of operations where you'd need to operate on the "unassociated" values? And how you'd deal with the fact that the set of color/coverage tuples representable by associated values is a superset of what is representable by unassociated, what do these operations do with the values not representable (like my transparent but incandescent yellow)?


I am not completely without sympathy to apps that started out life with unassociated values. Color+mask is analogous to loading your airbrush with a certain color paint, and spraying it with a mask covering some areas. So for a painting app designed a long time ago (few bits per pixel, no floating point, and no experience with image synthesis), it was a representation that made some sense, and allows independent manipulation of the mask without losing precision in the color channels. But it doesn't make much sense these days, IMHO. Image synthesis used the cleaner abstraction, painting did not.

--
Larry Gritz
lg at larrygritz.com

[Brecht Van Lommel]

Background: there is some discussion going on now about picking an
internal alpha format convention for Blender, and the question about
the consequences of either format came up.

I'm not sure I know which operations you mean. ?Can you give some examples of operations where you'd need to operate on the "unassociated" values? ?And how you'd deal with the fact that the set of color/coverage tuples representable by associated values is a superset of what is representable by unassociated, what do these operations do with the values not representable (like my transparent but incandescent yellow)?

Color correction operations often work better on unassociated RGB
values, or at the least, more like artists would be used to them in
painting applications. Another example would be blurring just the
alpha channel to feather edges, if you decompose/compose RGB and A
channels, unassociated values make this work out right.

Color space conversion may also work better on unassociated values,
but it depends on the situation, the missing background color
information makes this ill defined. For typical png input/output,
doing the conversion on unassociated values seems to be the a safer
default (but that is hand-wavy).

Regarding values that are not representable, a second layer / render
pass for emission could be used. Users may find it easier to think of
images as color + mask and get confused by some of the 'side effects'
of associated alpha, and might rather have something that works more
like painting apps and do some operations with more steps / layers.

I wouldn't want to argue against associated alpha though, mainly tried
to explain cases where you'd need to use unassociated values.

Brecht.

[Brecht Van Lommel]

Background: there is some discussion going on now about picking an
internal alpha format convention for Blender, and the question about
the consequences of either format came up.

I'm not sure I know which operations you mean. ?Can you give some examples of operations where you'd need to operate on the "unassociated" values? ?And how you'd deal with the fact that the set of color/coverage tuples representable by associated values is a superset of what is representable by unassociated, what do these operations do with the values not representable (like my transparent but incandescent yellow)?

Color correction operations often work better on unassociated RGB
values, or at the least, more like artists would be used to them in
painting applications. Another example would be blurring just the
alpha channel to feather edges, if you decompose/compose RGB and A
channels, unassociated values make this work out right.

Color space conversion may also work better on unassociated values,
but it depends on the situation, the missing background color
information makes this ill defined. For typical png input/output,
doing the conversion on unassociated values seems to be the a safer
default (but that is hand-wavy).

Regarding values that are not representable, a second layer / render
pass for emission could be used. Users may find it easier to think of
images as color + mask and get confused by some of the 'side effects'
of associated alpha, and might rather have something that works more
like painting apps and do some operations with more steps / layers.

I wouldn't want to argue against associated alpha though, mainly tried
to explain cases where you'd need to use unassociated values.

Brecht.

[Troy Sobotka]

You said in a post "It's wrong in so many ways. I can expand on this
idea if any of you don't know why."

I'd love to hear your official take on associated versus unassociated
alpha and why. Given the need to unassociate alpha for color
management and other linear operation, is there an optimal approach
you can envision?

Thanks for your time,
TJS

[Larry Gritz]

Post by Troy Sobotka
You said in a post "It's wrong in so many ways. I can expand on this
idea if any of you don't know why."
I'd love to hear your official take on associated versus unassociated
alpha and why.

Well, this is hardly "official", but here's my take:

If you think of image synthesis as akin to what a camera does, and the image like a photograph where the pixel values are proportional to how many photons hit the detector in each bin, then associated alpha is what you want -- the "color" values really are how bright each pixel should appear. (If you are using a linear color mapping of value to intensity, which I also recommend under all circumstances.) "Alpha" in this case is is an extra piece of information, that has no direct analog in photography, and tells you what the "coverage" is for each pixel, in case you want to do compositing of images. But if you strip the alpha off, the color still makes total sense as the brightness at each pixel, and you can display the image using only the information in the color channels (with proper compensation for the color response for your display).

Math is all linear and easy with associated alpha. The compositing operation "O = A over B" is very straightforward, and is identical in all channels:

Or = Ar + (1-Aa) * Br;
Og = Ag + (1-Aa) * Bg;
Ob = Ab + (1-Aa) * Bb;
Oa = Aa + (1-Aa) * Ba;

If you want to express your math as vectors/tuples,

O = A + (1-Aa) * B;

Nice!

And because the RGB values represent the amount of light hitting the detector, it makes perfect sense for it to be 0 (no light hit), regardless of how much of the background is covered, and it also makes sense to have values > 1 (there is no limit to how bright it can be) and for that matter, rgb > alpha. As a simple example, the color (.5, .5, 0, 0) might be used to represent a yellow glow that adds light without blocking any of the background at all.

With the other approach, the RGB channels do not mean much on their own, certainly not anything analogous to light and cameras. The "alpha" is a mask, so the color of each pixel is actually RGB*A. This has some strange consequences:

1. The over operation is a little funny now:

Or = Ar*Aa + (1-Aa) * Br;
Og = Ag*Aa + (1-Aa) * Bg;
Ob = Ab*Aa + (1-Aa) * Bb;
Oa = Aa + (1-Aa) * Ba;

It's both more math (who cares now, but it used to be helpful) but it's also less elegant -- you can't easily express it in vector notation as cleanly, and the math done to get the new alpha is different than what you do for the color channels. This is good circumstantial evidence that we're not working with as clean an abstraction.

2. Suddenly, it is no longer possible to express the "layer that adds yellow emissive light without blocking what's behind it." There is no unassociated (r,g,b,0) that can be yellow, since the amount of light is always the "color" multiplied by the "mask". If the mask is 0, so is the amount of light, by definition.

Post by Troy Sobotka
Given the need to unassociate alpha for color
management and other linear operation, is there an optimal approach
you can envision?

I'm not sure I know which operations you mean. Can you give some examples of operations where you'd need to operate on the "unassociated" values? And how you'd deal with the fact that the set of color/coverage tuples representable by associated values is a superset of what is representable by unassociated, what do these operations do with the values not representable (like my transparent but incandescent yellow)?


I am not completely without sympathy to apps that started out life with unassociated values. Color+mask is analogous to loading your airbrush with a certain color paint, and spraying it with a mask covering some areas. So for a painting app designed a long time ago (few bits per pixel, no floating point, and no experience with image synthesis), it was a representation that made some sense, and allows independent manipulation of the mask without losing precision in the color channels. But it doesn't make much sense these days, IMHO. Image synthesis used the cleaner abstraction, painting did not.

--
Larry Gritz
lg at larrygritz.com