Making Wayland accessible

[Matthias Clasen]

As part of the ongoing GNOME Wayland porting effort, the GNOME
accessibility (a11y) team has been looking into what it would take to make
our existing a11y tools (ATs) and infrastructure work under Wayland.

Most a11y features will probably end up being implemented in the
compositor:

- input tweaks like slow keys or bounce keys or hover-to-click naturally
fit in the event dispatching in the compositor

- display changes like zoom or color adjustments are already handled in
gnome-shell under X

- the text protocol should be sufficient to make on-screen keyboards and
similar tools work

The remaining AT of concern is orca, our screen reader, which does not
easily fit into the compositor. Here are some examples of the kinds of
things orca needs to do to support its users:

- Identify the surface that is currently under the pointer (and the
corresponding application that is registered as an accessible client)

- Warp the pointer to a certain position (see
https://bugzilla.gnome.org/show_bug.cgi?id=709999 for a description of how
this is used)

- Filter out certain key events and use them for navigation purposes. This
is currently done by capturing key events client-side and sending them out
again via at-spi, which will probably continue to work, even if it is
awkward and toolkit authors would really like to get rid of it

All of these features violate the careful separation between clients that
Wayland maintains, so that probably calls for some privileged interface for
ATs. I would appreciate feedback and discussion on this.

Has anybody else thought about these problems already ?

Are there better ways to do these things under Wayland ?


Matthias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20131015/dcf76455/attachment.html>

[Rui Tiago Cação Matos]

Hi,

- input tweaks like slow keys or bounce keys or hover-to-click naturally fit
in the event dispatching in the compositor

In the same spirit of having key repeat on the client side, I think
that slow, bounce and sticky keys should also be implemented by the
clients. Doing these on the compositor would in particular make things
more complex for xwayland which (I assume) should continue to provide
these features for X clients while being a regular wl_keyboard client
itself.

MouseKeys though, seems like it can be implemented in the wayland
compositor and we should probably remove the functionality from
xwayland. Hover-to-click can be done in the compositor as well.

Rui

[Bill Spitzak]

Personally I am very worried about clients doing this.

My main concern is clients acting differently than each other. This will
happen even if every one of them is using a carefully written toolkit
that implements all defined wayland behavior because they may be running
on different hosts with different versions of that toolkit. And unless
you are living in fantasy land you will know that there will be clients
that are not using perfect toolkits.

Also it makes it extremely difficult to remote to a host using a
different window system (rdp) that makes the opposite assumption,
because you have to "uncook" the events and then rely on the client
recooking them back to the same thing the use did on that host. Anybody
using NX will know this is a problem even between two X servers due to
it attempting to uncook keysyms into the scancodes used by the remote
server.

A literal interpretation of having the client do it all means you cannot
add a new type of hardware and have it emulate a mouse for software that
does not know about this new type of hardware.

In fact I think wayland should fully mark events with all kinds of
information, like "this is probably a double-click" and "the user has
dragged the mouse enough that it is not a click" and "the user has been
holding the mouse still long enough that you should do a hover event",
and everything else anybody can think of. Slight differences between
applications in deriving this information is a problem even on Windows.

It should be possible for a client to retrieve the raw events (if they
really exist at all, they won't in the rdp scenario). This mostly means
that any input processing cannot eat or delay an event, it must replace
it with something. And attaching the raw event info to it may be a good
idea.

Wayland is already "cooking" events, such as doing some of the keyboard
translation, the input methods, on-screen keyboards, the mapping of
other devices to the "seat pointer", the entire touch/gesture api, etc.
So I also see no reason to make repeat a special case.

Post by Rui Tiago Cação Matos
Hi,

- input tweaks like slow keys or bounce keys or hover-to-click naturally fit
in the event dispatching in the compositor

In the same spirit of having key repeat on the client side, I think
that slow, bounce and sticky keys should also be implemented by the
clients. Doing these on the compositor would in particular make things
more complex for xwayland which (I assume) should continue to provide
these features for X clients while being a regular wl_keyboard client
itself.
MouseKeys though, seems like it can be implemented in the wayland
compositor and we should probably remove the functionality from
xwayland. Hover-to-click can be done in the compositor as well.
Rui
_______________________________________________
wayland-devel mailing list
wayland-devel at lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/wayland-devel

[Piñeiro]

Hi Matthias,

thanks for starting the discussion. I will add some points below.

Post by Matthias Clasen
As part of the ongoing GNOME Wayland porting effort, the GNOME
accessibility (a11y) team has been looking into what it would take to
make our existing a11y tools (ATs) and infrastructure work under Wayland.

FWIW, most of the stuff we did on the recent Montreal Summit was related
with Wayland. You can find a summary on this wiki page:
https://wiki.gnome.org/Accessibility/Hackfests/Montreal2013

Post by Matthias Clasen
Most a11y features will probably end up being implemented in the
- input tweaks like slow keys or bounce keys or hover-to-click
naturally fit in the event dispatching in the compositor
- display changes like zoom or color adjustments are already handled
in gnome-shell under X
- the text protocol should be sufficient to make on-screen keyboards
and similar tools work
The remaining AT of concern is orca, our screen reader, which does not
easily fit into the compositor. Here are some examples of the kinds of
- Identify the surface that is currently under the pointer (and the
corresponding application that is registered as an accessible client)

FWIW2, there is a running conversation about that here:
https://mail.gnome.org/archives/gnome-accessibility-devel/2013-October/msg00006.html

Post by Matthias Clasen
- Warp the pointer to a certain position (see
https://bugzilla.gnome.org/show_bug.cgi?id=709999 for a description of
how this is used)

Also tracking mouse position (More about that here,
https://bugzilla.gnome.org/show_bug.cgi?id=710012, although it doesn't
include a description about how it is used).

Post by Matthias Clasen
- Filter out certain key events and use them for navigation purposes.
This is currently done by capturing key events client-side and sending
them out again via at-spi, which will probably continue to work, even
if it is awkward and toolkit authors would really like to get rid of it
All of these features violate the careful separation between clients
that Wayland maintains, so that probably calls for some privileged
interface for ATs.

Most of the people I asked (mostly after Wayland related presentations
on GUADEC and others) pointed to that direction.

Post by Matthias Clasen
I would appreciate feedback and discussion on this.
Has anybody else thought about these problems already ?
Are there better ways to do these things under Wayland ?

BR