desktop-shell: How to enable really alpha blending of weston background?

Discussion:

YoungJun Jo

2016-02-22 12:36:41 UTC

Environment
OS : Linux kernel 3.10
Version : Weston 1.8.0
H/W spec : embedded soc based on ARM

Hello, All

I have a question about weston background.
I want enable alpha blending to background, so i have typed
'background-color=0x00ffffff' on weston.ini.
But that's not really alpha blending. Because alpha value of pixel is a
'ff'.
My chips have four H/W layers, top of the layer is UI layer and second
layer is video layer.

I reviewed source code of 'desktop-shell.c' but i can't find any hints.

Need some additional ideas to solve this problem. Any feedback is
appreciated.

Thanks.

Pekka Paalanen

2016-02-22 13:04:03 UTC

Permalink

On Mon, 22 Feb 2016 21:36:41 +0900

Post by YoungJun Jo
Environment
OS : Linux kernel 3.10
Version : Weston 1.8.0
H/W spec : embedded soc based on ARM
Hello, All
I have a question about weston background.
I want enable alpha blending to background, so i have typed
'background-color=0x00ffffff' on weston.ini.
But that's not really alpha blending. Because alpha value of pixel is a
'ff'.
My chips have four H/W layers, top of the layer is UI layer and second
layer is video layer.
I reviewed source code of 'desktop-shell.c' but i can't find any hints.
Need some additional ideas to solve this problem. Any feedback is
appreciated.

Hi,

Weston does not do transparent backgrounds. The backends instruct the
renderers to allocate an opaque framebuffer if possible, because there
is not supposed to be anything "behind" Weston. After all, Weston is in
charge of everything going to the display, so by definition there is
nothing to show through to.

IOW, the decision to use an opaque framebuffer is in each backend code.
I suppose you could change that if you wanted to: the Pixman format for
pixman-renderer, or opaque_attribs vs. alpha_attribs for the
gl-renderer.

What are you *actually* trying to do?

Thanks,
pq

YoungJun Jo

2016-02-23 01:11:39 UTC

Permalink

Hello,

Thanks for your reply.

What i want is alpha blending for my chips.
As i mentioned, My chips have four independent H/W overlays.

For example,
H/W Overlay 1 => Weston, Applications, etc..
H/W Overlay 2 => Video play.
H/W Overlay 3 = > Empty.
H/W Overlay 4 = > Empty.

I also know that weston don't know H/W Overlays above mentioned.

Scenario:
1. If i click play button on my video application, it will be call alpha
blending API.
2. When weston receives alpha blending api, weston background will be
transparent.
3. And i could see video playing from H/W Layer 2.(Still H/W Layer1 shows
GUI of video control buttons)

So i want really alpha blending of weston background.

I'm sorry to describe it difficult to understand.

Thanks.

Pekka Paalanen

2016-02-23 09:29:34 UTC

Permalink

On Tue, 23 Feb 2016 10:11:39 +0900

Post by YoungJun Jo
Hello,
Thanks for your reply.
What i want is alpha blending for my chips.
As i mentioned, My chips have four independent H/W overlays.
For example,
H/W Overlay 1 => Weston, Applications, etc..
H/W Overlay 2 => Video play.
H/W Overlay 3 = > Empty.
H/W Overlay 4 = > Empty.
I also know that weston don't know H/W Overlays above mentioned.

Hi,

ok. The current state of Weston is slightly sad, as the support for
hardware overlays is disabled by default, because enabling it causes
performance problems. All that will be fixed once the patch series to
support atomic/nuclear KMS in the DRM backend by Daniel Stone will
land. I think Daniel is polishing it for submission whenever he has
time for it.

I would expect the situation to improve for Weston 1.11, but that's no
promise.

Btw. if you are using an fbdev backed, you have already lost: it will
never support hardware overlays. Use the DRM-backend instead. It is
possible to activate overlay support already with the debug key binding
Mod+Shift+space, v.

Post by YoungJun Jo
1. If i click play button on my video application, it will be call alpha
blending API.
2. When weston receives alpha blending api, weston background will be
transparent.
3. And i could see video playing from H/W Layer 2.(Still H/W Layer1 shows
GUI of video control buttons)
So i want really alpha blending of weston background.

I suppose you could do it like that, assuming that:
- you will only ever support your one special video player
- your video player is not cooperating with the display server, but it
is bypassing the display server instead and holding the display
server as a hostage (otherwise the display server, e.g. weston, can
mess up your video view)
- your video player is likely specific to your hardware and cannot work
elsewhere
- you will have to use a modified Weston, as upstream will never accept
what you would need for this
- you will need to create a protocol extension for the "alpha blending
API" or whatever to tell weston to get out of the way
- you won't have any application window mapped while playing e.g.
fullscreen video
- you will waste power by always having to have a fullscreen completely
transparent overlay on top the video overlay, rather than having the
video on the top-most overlay

and probably many more limitations that make the approach very painful
to use in anything but a simple single-purpose device with a throw-away
software stack.

Essentially, you are using a display server (weston) to get a window
system, and then your aim is to remove the window system and bypass the
display server, telling weston to not mess with your video player.

Btw. I forgot a few things earlier. Not only you need to have the
backend tell the renderer use an alpha-capable framebuffer format, you
also need to hack clients/desktop-shell.c to not force the background
to be opaque. I'm not sure, but there might also be further things to
adjust in a backend to make sure the alpha bits actually get into the
hardware.

The proper solution using Wayland and Weston as they are intended would
involve the following:
- ensure your display hardware has proper DRM drivers supporting atomic
KMS
- wait for the atomic support to land in Weston, which...
- puts Weston in control of all hardware planes, so that...
- Weston will automatically use all hardware planes for everything it
can
- use any general purpose video player you want which supports
linux_dmabuf Wayland protocol extension...
- and that will get the video on the hw overlay automatically when it
is possible, and through hardware accelerated composition when the
overlay cannot be used.

When the video player uses EGL or linux_dmabuf to push the video frames
to Weston, then Weston has the possibility to show them on a hardware
overlay.

As Weston will be in charge of everything on the display, the
fullscreen video playback won't suffer from the same waste of power as
your approach.

The proper solution is likely to result in a near-optimal use of
hardware resources in all cases automatically, both where overlays can
and cannot be used. The proper solution is also generic in that it is
not tied to a particular hardware platform.

In summary, do what you must, but I would strongly recommend the proper
solution if at all possible for you.

Thanks,
pq

Post by YoungJun Jo

Jasper St. Pierre

2016-03-12 19:33:40 UTC

Permalink

I have to bring this up, because it's not necessarily true. There's
something you're missing. After working on embedded SoCs, I realize
that a lot of them put the YUV video plane *behind* the main RGB
plane. This allows them to do subtitles and OSD controls over the
video without stacking it RGB -> YUV -> RGB, as you might imagine.

Thankfully, some of these systems allow you to put the video on top of
RGB, but not all. Some even just have two YUV planes (for
picture-in-picture) and one RGB plane on top, not even a cursor plane.

On embedded systems where they have fullscreen control, they simply
render alpha-graphics into the RGB window, and then stack a YUV plane
behind it to blend against.

Wayland, in its current subsurface architecture, can't support this
system, unfortunately. The best thing I could think of here was to
have Wayland consider a punch-through when it sees a YUV plane stacked
under an RGB one from the same client, and it knows the scanout is
ARGB. But maybe that's too auto-magical.

I think that could work, but the implementation would need to be
tricky about all the special cases, if we cared enough to make those
work, e.g. surfaces under the punch-through need to be knocked out,
but surfaces above can't be, since they're stacked on top. And that
implies surfaces under the punch-through can't themselves use
overlays, etc.

Post by Pekka Paalanen
On Tue, 23 Feb 2016 10:11:39 +0900

Hi,
ok. The current state of Weston is slightly sad, as the support for
hardware overlays is disabled by default, because enabling it causes
performance problems. All that will be fixed once the patch series to
support atomic/nuclear KMS in the DRM backend by Daniel Stone will
land. I think Daniel is polishing it for submission whenever he has
time for it.
I would expect the situation to improve for Weston 1.11, but that's no
promise.
Btw. if you are using an fbdev backed, you have already lost: it will
never support hardware overlays. Use the DRM-backend instead. It is
possible to activate overlay support already with the debug key binding
Mod+Shift+space, v.

- you will only ever support your one special video player
- your video player is not cooperating with the display server, but it
is bypassing the display server instead and holding the display
server as a hostage (otherwise the display server, e.g. weston, can
mess up your video view)
- your video player is likely specific to your hardware and cannot work
elsewhere
- you will have to use a modified Weston, as upstream will never accept
what you would need for this
- you will need to create a protocol extension for the "alpha blending
API" or whatever to tell weston to get out of the way
- you won't have any application window mapped while playing e.g.
fullscreen video
- you will waste power by always having to have a fullscreen completely
transparent overlay on top the video overlay, rather than having the
video on the top-most overlay
and probably many more limitations that make the approach very painful
to use in anything but a simple single-purpose device with a throw-away
software stack.
Essentially, you are using a display server (weston) to get a window
system, and then your aim is to remove the window system and bypass the
display server, telling weston to not mess with your video player.
Btw. I forgot a few things earlier. Not only you need to have the
backend tell the renderer use an alpha-capable framebuffer format, you
also need to hack clients/desktop-shell.c to not force the background
to be opaque. I'm not sure, but there might also be further things to
adjust in a backend to make sure the alpha bits actually get into the
hardware.
The proper solution using Wayland and Weston as they are intended would
- ensure your display hardware has proper DRM drivers supporting atomic
KMS
- wait for the atomic support to land in Weston, which...
- puts Weston in control of all hardware planes, so that...
- Weston will automatically use all hardware planes for everything it
can
- use any general purpose video player you want which supports
linux_dmabuf Wayland protocol extension...
- and that will get the video on the hw overlay automatically when it
is possible, and through hardware accelerated composition when the
overlay cannot be used.
When the video player uses EGL or linux_dmabuf to push the video frames
to Weston, then Weston has the possibility to show them on a hardware
overlay.
As Weston will be in charge of everything on the display, the
fullscreen video playback won't suffer from the same waste of power as
your approach.
The proper solution is likely to result in a near-optimal use of
hardware resources in all cases automatically, both where overlays can
and cannot be used. The proper solution is also generic in that it is
not tied to a particular hardware platform.
In summary, do what you must, but I would strongly recommend the proper
solution if at all possible for you.
Thanks,
pq

Post by YoungJun Jo

_______________________________________________
wayland-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/wayland-devel

--
Jasper

Pekka Paalanen

2016-03-14 12:44:34 UTC

Permalink

On Sat, 12 Mar 2016 11:33:40 -0800

Post by Jasper St. Pierre
I have to bring this up, because it's not necessarily true. There's
something you're missing. After working on embedded SoCs, I realize
that a lot of them put the YUV video plane *behind* the main RGB
plane. This allows them to do subtitles and OSD controls over the
video without stacking it RGB -> YUV -> RGB, as you might imagine.

Oh my.

Post by Jasper St. Pierre
Thankfully, some of these systems allow you to put the video on top of
RGB, but not all. Some even just have two YUV planes (for
picture-in-picture) and one RGB plane on top, not even a cursor plane.
On embedded systems where they have fullscreen control, they simply
render alpha-graphics into the RGB window, and then stack a YUV plane
behind it to blend against.
Wayland, in its current subsurface architecture, can't support this
system, unfortunately. The best thing I could think of here was to
have Wayland consider a punch-through when it sees a YUV plane stacked
under an RGB one from the same client, and it knows the scanout is
ARGB. But maybe that's too auto-magical.
I think that could work, but the implementation would need to be
tricky about all the special cases, if we cared enough to make those
work, e.g. surfaces under the punch-through need to be knocked out,
but surfaces above can't be, since they're stacked on top. And that
implies surfaces under the punch-through can't themselves use
overlays, etc.

I don't think it should be *that* hard, but of course I'm only talking
from Weston implementation point of view.

Let's say the scenegraph has a bunch of arbitrary surfaces below and
above a particular YUV surface that would be nice to put in a separate
hw plane. This YUV-surface is also opaque, which is a requirement to
even consider underlay.

Weston goes through the surface stack from top to bottom, looking for
surfaces that could be put on overlays. It comes to the YUV-surface,
but there are already things on top of it, so it cannot be promoted to
an overlay. Seeing that using underlay is possible from DRM
perspective, it decides to put it on an underlay. Putting a surface in
an underlay requires adding an extra surface filled with RGBA (0,0,0,0)
color for the renderer to create a punch-through (or just set a flag in
the weston_view that this is a punch-through surface).

Weston continues going through the surface stack. If it finds another
YUV surface, the accumulated occlusions prevent it from going to
another underlay, perhaps. So the surface to plane assignment should
work.

Then the renderer flattens everything on the primary plane. Since the
punch-through placeholder surface is both opaque and filled with
(0,0,0,0), the renderer automatically creates the punch-through hole as
intended, not rendering anything from beneath it, and rendering
everything that may be on top of it.

It's obviously not trivial, but I think Weston already computes
most of the things needed to do this, like opaque and non-opaque
occlusions. What we are mainly missing is how to choose between overlay
and underlay. And of course implementing the underlay bits in practice.
We already have region checks to see if overlays can be used not just
for the topmost surfaces, and underlays would need their own similar
checks/accumulation.

I also don't see how which client gives which surface would affect
this. There is no reason that the thing on top of an YUV surface must be
from any particular client.

So I think Wayland does support this. It would be possible to implement
in Weston IMO. OTOH, I cannot imagine what we should add, remove or
change in the Wayland protocol to help underlay support in addition to
overlay support, even if we were not bound by stability requirements.

The biggest problem I see is that there may be (and are) several
different solutions for a scenegraph to be laid onto hw planes, and
choosing the optimal configuration is not easy. That, and creating a
test suite to make sure things actually work.

Thanks,
pq

Pekka Paalanen

2016-03-15 09:02:48 UTC

Permalink

On Mon, 14 Mar 2016 14:44:34 +0200

Post by Pekka Paalanen
On Sat, 12 Mar 2016 11:33:40 -0800

Oh my.

I don't think it should be *that* hard, but of course I'm only talking
from Weston implementation point of view.

FYI,

I've filed https://phabricator.freedesktop.org/T7351

Thanks,
pq