[Appeal] Improving the Menu + Toolbar driven GUI withi adaptive element sizes.

[Jani-Matti Hätinen]

Hello,

I propose below an idea on how the current Menu + Toolbar -based GUI paradigm
might be improved upon (thus it won't represent any sort of radical
rethinking of the GUI). Right now I'll try to represent my ideas in text
form, but later on I'll try to create some mock-ups to illustrate my points.
First I'll cut to the chase and try to explain the idea as clearly as
possible. Then at the end I'll detail some of the earlier GUI developments
which inspired this idea.

Basically this boils down to having toolbar and menu items, which change their
size depending on how often you use them (or alternatively can be locked to
certain fairly free-flowing sizes).
So, if you, for example mainly use the back and reload buttons in
konqueror's web browser mode, the icons representing the actions would, at
first, enlarge to make is easier to spot and hit them, and eventually the
larger icons would be accompanied by a text label, which would further
differentiate them from the other visible GUI elements.
Similarly, if you were to often use the New Tab action from the Location
menu, it would get bigger and bigger over time (first accompanied by a bigger
icon and more vertical spacing, but later also with bold text and a bigger
font).
Also, if for some reason your usage habits were to change so that you
wouldn't use a certain action as often (for example you were to start using
the Ctrl-Shift-N shortcut for new tabs), the corresponding GUI element would
shrink back according to your usage habits.
Naturally the growing and shrinking of the GUI elements would be scaled
according to the user's font and icon size settings. So if you were to choose
small icons, the icons would never too big, or too small, if you were to
choose big icons.

The adaptive GUI element sizes could be further accompanied with hints, which
would make it easy for a user to add an often used menu item into the
toolbar, or remove a rarely used toolbar item, to be only accessible from the
menu. Note that the movement options should also be accessible from the same
place even when no hint would be visible (possibly through a context menu
which the hints could refer to)
The main point here would be that moving GUI elements would never happen
without the user's request. The automatic GUI adaptation would only affect
the proportions of the different GUI elements according to their use.

This approach could (and if possible, should) be applied to all GUI elements.
For example launchers in the panel would change size depending on their
usage, as well as programs in the KDE menu. The same could even apply to
context menus as well as files and folders in konqueror's icon view.

Now I'm not sure how difficult this would be to implement, but I'm guessing
pretty difficult. At minimum it will require SVG icon support to allow smooth
icon scaling, considerable changes to the current menu and toolbar handling
code, some sort of background process which would track and analyze the
user's usage habits and config file support to store the user's settings.
Whether this is at all doable in QT4 I can't really say.

Anyway here's the background & rationale section for this idea:

The main sources of inspiration for this idea (apart from every single GUI
software I've ever seen) are the adaptive menus in MS Office 2000 and later,
as well as somewhat by the ribbon-style GUI of MS Office 12. The final spark
however came from the new startup dialog in koffice-1.5.
Just like the two MS Office GUI ideas, this idea tries to solve the problem
of finding the wanted options/features in increasingly complex pieces of
software.

The Microsoft approach to this problem (first with adaptive menus and now the
ribbon-style GUI) is to hide unused features in order to make the oft-used
ones more visible.
However, this approach has the basic problem of 'If I can't see it, it's not
there'. As we all noticed with adaptive menus (and will probably notice with
the ribbon-thing) trying to remember where an invisible option is is a lot
harder than trying to find a visible one amongst even a lot of others
(especially if even the visible GUI elements are always moving as was the
case with adaptive menus).
Nevertheless IMHO both the Microsoft solutions as well as the koffice dialog
illustrate three important points:
First: moving GUI elements without user consent is just plain malice.
Second: infinite, repetitive menus (which offer no other distinction between
the different options apart from the actual text) aren't usable for anyone
(because if you were ever able to get the adaptive menus right, it was a hell
of a lot better than the traditional ones, until it changed again, that is).
Third: Correlating the frequency of use for an action with the size of the
corresponding GUI element is a good thing. (This is demonstrated really well
in the koffice-1.5 startup dialog in which the most often used buttons (Open
existing document... and Open this document) are by far the smallest GUI
elements in the whole dialog)

So, merely changing GUI element sizes is IMHO a fairly good compromise between
optimising a GUI for the user (which most people actually want) and not
playing games with his/her sense of direction.

[Chi Shang Cheng]

Hi Jani,

Post by Jani-Matti Hätinen
Hello,
I propose below an idea on how the current Menu + Toolbar -based GUI paradigm
might be improved upon (thus it won't represent any sort of radical
rethinking of the GUI). Right now I'll try to represent my ideas in text
form, but later on I'll try to create some mock-ups to illustrate my points.
First I'll cut to the chase and try to explain the idea as clearly as
possible. Then at the end I'll detail some of the earlier GUI developments
which inspired this idea.
Basically this boils down to having toolbar and menu items, which change their
size depending on how often you use them (or alternatively can be locked to
certain fairly free-flowing sizes).
So, if you, for example mainly use the back and reload buttons in
konqueror's web browser mode, the icons representing the actions would, at
first, enlarge to make is easier to spot and hit them, and eventually the
larger icons would be accompanied by a text label, which would further
differentiate them from the other visible GUI elements.
Similarly, if you were to often use the New Tab action from the Location
menu, it would get bigger and bigger over time (first accompanied by a bigger
icon and more vertical spacing, but later also with bold text and a bigger
font).
Also, if for some reason your usage habits were to change so that you
wouldn't use a certain action as often (for example you were to start using
the Ctrl-Shift-N shortcut for new tabs), the corresponding GUI element would
shrink back according to your usage habits.
Naturally the growing and shrinking of the GUI elements would be scaled
according to the user's font and icon size settings. So if you were to choose
small icons, the icons would never too big, or too small, if you were to
choose big icons.

Although I haven't seen a mock-up of this idea yet, I think there are some
issuess to be addressed:
- From a graphic design view, having an inconsistent icon (graphical
element) size throughout a certain toolbar results is graphically not
appealing (Gestalt principle 'alignment').
- Due to the changing nature of the icons in the toolbar, the position of a
certain toolbar item would be less easier to predict. A possible result of
this would be increased navigation time of the mouse cursor of point 'X' to
that certain toolbar item.
- The cognitive effects (such as speed, memory etc.) of icons with
inconsistent size in an array hasn't been scientifcally researched yet. That
is, I haven't any scientific literature on this specific subject yet.
- Relative values such as 'often', 'seldom' etc. cannot be quantitized, that
is, each user has a personal value that differs per user, per application,
per use case. For instance, when I'm designing a birthday card in KWord,
I'll be doing more design actions, in contrast when I'm writing an academic
report, where I'll be using more text formatting tools. What is 'often'?
Would the user interface adapt while running an application, or would it
adapt over a longer timespan? The usefullness of adaption is hard to predict
due to a great amount of variables as described above. Even if you would
conduct user testing with a huge amount of users, I'm not sure whether this
would result in useful data.
- Elaborating on the previous issue, the multi-purpose use a single
application is also an issue. Take Konqueror for example, it's both a file
browser as well as a web browser. The location bar in web browser mode would
be used a lot, whereas the user would use the icon view to navigate in file
browser mode.

The adaptive GUI element sizes could be further accompanied with hints,

Post by Jani-Matti Hätinen
which
would make it easy for a user to add an often used menu item into the
toolbar, or remove a rarely used toolbar item, to be only accessible from the
menu. Note that the movement options should also be accessible from the same
place even when no hint would be visible (possibly through a context menu
which the hints could refer to)
The main point here would be that moving GUI elements would never happen
without the user's request. The automatic GUI adaptation would only affect
the proportions of the different GUI elements according to their use.

Hints would be a nice addition.

This approach could (and if possible, should) be applied to all GUI

Post by Jani-Matti Hätinen
elements.
For example launchers in the panel would change size depending on their
usage, as well as programs in the KDE menu. The same could even apply to
context menus as well as files and folders in konqueror's icon view.

I'm not sure whether having inconsistent font size in a context menu (which
would also result in inconsistents vertical spacing) would enhance
readibility...

Now I'm not sure how difficult this would be to implement, but I'm guessing

Post by Jani-Matti Hätinen
pretty difficult. At minimum it will require SVG icon support to allow smooth
icon scaling, considerable changes to the current menu and toolbar handling
code, some sort of background process which would track and analyze the
user's usage habits and config file support to store the user's settings.

Again, I'm not sure whether you can draw semantics of a bunch of numbers,
without knowing the other variables (may be considered as meta information).


Whether this is at all doable in QT4 I can't really say.

Post by Jani-Matti Hätinen
The main sources of inspiration for this idea (apart from every single GUI
software I've ever seen) are the adaptive menus in MS Office 2000 and later,

Personally, I hated the adaptive menus in MS Office. Example: At the time of
Word 2000, printer ink was quite expensive (and it still is today), so I
tried to print only when it was really necessary. If I printed to times a
week, I would have considered that a lot. Of course, I would use Word more
often than I would print. Word would have hidden the print menu item,
although I didn't consided it not important. (Not sure whether this was a
true experience, I disabled personalized menus a long long time ago, but
you'll get the idea.)

Bottom line: because of the adaptation, the predictability of place and
position of UI elements decreased a lot.

as well as somewhat by the ribbon-style GUI of MS Office 12. The final spark

Post by Jani-Matti Hätinen
however came from the new startup dialog in koffice-1.5.
Just like the two MS Office GUI ideas, this idea tries to solve the problem
of finding the wanted options/features in increasingly complex pieces of
software.
The Microsoft approach to this problem (first with adaptive menus and now the
ribbon-style GUI) is to hide unused features in order to make the oft-used
ones more visible.

I'm not sure whether these approaches can be seen analogue to yours. First
of all, the adaptive menus weren't much of a success, outlined here:
http://blogs.msdn.com/jensenh/archive/2006/03/31/565877.aspx. Second, the
Ribbon is contextual, and as for as I know it doesn't hide non-important
items.

However, this approach has the basic problem of 'If I can't see it, it's

Post by Jani-Matti Hätinen
not
there'. As we all noticed with adaptive menus (and will probably notice with
the ribbon-thing) trying to remember where an invisible option is is a lot
harder than trying to find a visible one amongst even a lot of others
(especially if even the visible GUI elements are always moving as was the
case with adaptive menus).
Nevertheless IMHO both the Microsoft solutions as well as the koffice dialog
First: moving GUI elements without user consent is just plain malice.
Second: infinite, repetitive menus (which offer no other distinction between
the different options apart from the actual text) aren't usable for anyone
(because if you were ever able to get the adaptive menus right, it was a hell
of a lot better than the traditional ones, until it changed again, that is).
Third: Correlating the frequency of use for an action with the size of the
corresponding GUI element is a good thing. (This is demonstrated really well
in the koffice-1.5 startup dialog in which the most often used buttons (Open
existing document... and Open this document) are by far the smallest GUI
elements in the whole dialog)
So, merely changing GUI element sizes is IMHO a fairly good compromise between
optimising a GUI for the user (which most people actually want) and not
playing games with his/her sense of direction.

To summarize, your proposal raises many questions which cannot be answered
without doing any research. What is to be expected, is that the user
interface will look aesthetically less appealing, due to graphical design
issues such as inconsistency, balance, alignment etc. Of course, this would
not be the greatest drawback. If the nature of adaptation during the usage
lifecycle proves to be unable to enhance the usability of the user
interface, this idea should be discarded.

When a user-centered development methodology is applied when developing an
application, the user interface will be 'adapted' (or 'adjusted'), created
specifically for a group of users. Is further personalization required, and
if so, can it be done automatically? And if we were to implement an
auto-adaptative user interface, would the application be able to gather data
about the user that can be used usefully? If I had to answer that question,
I would have to honestly say that I don't think that it's doable.

Chi Shang

--

Post by Jani-Matti Hätinen
Jani-Matti H?tinen
_______________________________________________
Appeal mailing list
Appeal at kde.org
https://mail.kde.org/mailman/listinfo/appeal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/appeal/attachments/20060525/156df028/attachment.html

[Jani-Matti Hätinen]

Chi Shang Cheng kirjoitti viestiss??n (l?hetysaika torstai, 25. toukokuuta

Post by Chi Shang Cheng
Hi Jani,

Thank you for your excellent points. I'm still working on some mock-ups, but
in the mean time I'll try to answer to the issues you raised the best I can.

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
Hello,
I propose below an idea on how the current Menu + Toolbar -based GUI
paradigm might be improved upon (thus it won't represent any sort of
radical rethinking of the GUI). Right now I'll try to represent my ideas
in text form, but later on I'll try to create some mock-ups to illustrate
my points.
First I'll cut to the chase and try to explain the idea as clearly as
possible. Then at the end I'll detail some of the earlier GUI
developments which inspired this idea.
Basically this boils down to having toolbar and menu items, which change
their size depending on how often you use them (or alternatively can be
locked to certain fairly free-flowing sizes).
So, if you, for example mainly use the back and reload buttons in
konqueror's web browser mode, the icons representing the actions would, at
first, enlarge to make is easier to spot and hit them, and eventually the
larger icons would be accompanied by a text label, which would further
differentiate them from the other visible GUI elements.
Similarly, if you were to often use the New Tab action from the
Location menu, it would get bigger and bigger over time (first accompanied
by a bigger icon and more vertical spacing, but later also with bold text
and a bigger font).
Also, if for some reason your usage habits were to change so that you
wouldn't use a certain action as often (for example you were to start
using the Ctrl-Shift-N shortcut for new tabs), the corresponding GUI
element would shrink back according to your usage habits.
Naturally the growing and shrinking of the GUI elements would be scaled
according to the user's font and icon size settings. So if you were to
choose small icons, the icons would never too big, or too small, if you
were to choose big icons.

Although I haven't seen a mock-up of this idea yet, I think there are some
- From a graphic design view, having an inconsistent icon (graphical
element) size throughout a certain toolbar results is graphically not
appealing (Gestalt principle 'alignment').

Another option is that toolbar items would be enlarged only in the direction
of the toolbar, i.e. usually horizontally (just like menu items would only be
enlarged in the direction of the menu, i.e. vertically).
This would eliminate the problem of having the toolbar size adjust to the
height of the elements, but on the other hand would make it harder to find
the border between two elements (unless one were to stretch the corresponding
icons, which doesn't really seem like a good idea)
But obviously these will have to tested as well

Post by Chi Shang Cheng
- Due to the changing nature of the icons in the toolbar, the position of a
certain toolbar item would be less easier to predict. A possible result of
this would be increased navigation time of the mouse cursor of point 'X' to
that certain toolbar item.

I was thinking (but not writing, it seems) that the size differences would be
rather subtle. Even at maximum one icon/menu item size shouldn't IMHO be more
than 150% to maybe 200%. This'll obviously need to be tested and refined.
But in any case the main point here is that the automatic GUI adaptation
would not, under any circumstances, radically change the look or composition
of the GUI, but rather it would slowly evolve over time.

Post by Chi Shang Cheng
- The cognitive effects (such as speed, memory etc.) of icons with
inconsistent size in an array hasn't been scientifcally researched yet.
That is, I haven't any scientific literature on this specific subject yet.

That's true (i.e. me neither). And this is something that will definitely need
to be done as some point with some kind of a test application.

Post by Chi Shang Cheng
- Relative values such as 'often', 'seldom' etc. cannot be quantitized,
that is, each user has a personal value that differs per user, per
application, per use case. For instance, when I'm designing a birthday card
in KWord, I'll be doing more design actions, in contrast when I'm writing
an academic report, where I'll be using more text formatting tools. What is
'often'?

I was thinking that usage analysing would be based on clicks within the GUI
elements of an application (keyboard shortcuts won't IMHO need to be tracked,
or at least they shouldn't contribute to the sizes of the corresponding
toolbar/menu items).
IMHO the obvious solution is that the calculations would be based on
percentages. I.e. on the percentage of total clicks targetted at each
individual GUI element.
All the numbers (the size of the click history, usage percentage/element
size ratio, etc.) will obviously need to be refined through testing.

Post by Chi Shang Cheng
Would the user interface adapt while running an application, or would it
adapt over a longer timespan?

Yes to both actually. The adaptation would occur over a fairly long clickspan
(so if you use one application a lot, it changes faster than the others), but
also when running an application. This is because having big GUI changes
happen whenever you start/close an application is exactly the type of thing
that makes people hate adaptive GUIs. And besides, some people never close
their applications. For example Kontact has been continuously running on this
machine for at least two months, and I've used it a lot.

Post by Chi Shang Cheng
The usefullness of adaption is hard to predict due to a great amount of
variables as described above. Even if you would conduct user testing with a
huge amount of users, I'm not sure whether this would result in useful data.
- Elaborating on the previous issue, the multi-purpose use a single
application is also an issue. Take Konqueror for example, it's both a file
browser as well as a web browser. The location bar in web browser mode
would be used a lot, whereas the user would use the icon view to navigate
in file browser mode.

This is a good question indeed. If the size of an element is to be based on
the percentage of total clicks directed to it, how shall we define the total
clicks. Is it per application, per application profile (i.e. konqueror's file
browser profile vs, web browser profile) or edited document type (letter or
article in kword), or somehow else.

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
The adaptive GUI element sizes could be further accompanied with hints,
which would make it easy for a user to add an often used menu item into
the toolbar, or remove a rarely used toolbar item, to be only accessible
from the menu. Note that the movement options should also be accessible
from the same place even when no hint would be visible (possibly through a
context menu which the hints could refer to)
The main point here would be that moving GUI elements would never
happen without the user's request. The automatic GUI adaptation would only
affect the proportions of the different GUI elements according to their
use.

Hints would be a nice addition.

I'm working on a separate mock-up on the hints and the toolbar/menu context
menus, because they are a lot easier to implement than this whole shebang,
and can be done separately. Plus IMHO they would make GUI configuration (i.e.
changing which icons are in toolbars and which are not) a lot more
accessible.
They could even fit into the KDE4 timeframe. Whereas the whole adaptive
thing might be more realistic for KDE4.3 or KDE5

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
This approach could (and if possible, should) be applied to all GUI
elements. For example launchers in the panel would change size depending
on their usage, as well as programs in the KDE menu. The same could even
apply to context menus as well as files and folders in konqueror's icon
view.

I'm not sure whether having inconsistent font size in a context menu (which
would also result in inconsistents vertical spacing) would enhance
readibility...

True. It might have an adverse effect. The same goes for all menu elements.
IMHO subtlety is the key here.

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
Now I'm not sure how difficult this would be to implement, but I'm
guessing pretty difficult. At minimum it will require SVG icon support to
allow smooth icon scaling, considerable changes to the current menu and
toolbar handling code, some sort of background process which would track
and analyze the user's usage habits and config file support to store the
user's settings.

Again, I'm not sure whether you can draw semantics of a bunch of numbers,
without knowing the other variables (may be considered as meta
information).

That obviously depends on the problem like all engineering cases. One has to
define the problem and the environment and then just see whether the
quantifiable elements are a strong enough basis to work on.
And yes, I'm working on this.

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
Whether this is at all doable in QT4 I can't really say.
The main sources of inspiration for this idea (apart from every single
GUI software I've ever seen) are the adaptive menus in MS Office 2000 and
later,

Personally, I hated the adaptive menus in MS Office. Example: At the time
of Word 2000, printer ink was quite expensive (and it still is today), so I
tried to print only when it was really necessary. If I printed to times a
week, I would have considered that a lot. Of course, I would use Word more
often than I would print. Word would have hidden the print menu item,
although I didn't consided it not important. (Not sure whether this was a
true experience, I disabled personalized menus a long long time ago, but
you'll get the idea.)
Bottom line: because of the adaptation, the predictability of place and
position of UI elements decreased a lot.

Yes. I hated them too (and as far as I know so did everyone else). Precisely
due to the reasons you also stated. I.e. it hid unused elements (rather than
emphasized used ones), it constantly rearranged GUI elements and finally, the
changes it made were not incremental, but boolean (i.e. each menu item was
either on or off, nothing in between)

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
as well as somewhat by the ribbon-style GUI of MS Office 12. The final
spark however came from the new startup dialog in koffice-1.5.
Just like the two MS Office GUI ideas, this idea tries to solve the
problem of finding the wanted options/features in increasingly complex
pieces of software.
The Microsoft approach to this problem (first with adaptive menus and now
the ribbon-style GUI) is to hide unused features in order to make the
oft-used ones more visible.

I'm not sure whether these approaches can be seen analogue to yours. First
http://blogs.msdn.com/jensenh/archive/2006/03/31/565877.aspx. Second, the
Ribbon is contextual, and as for as I know it doesn't hide non-important
items.

But it does. It hides everything that the designers didn't see as a part of
each context. Obviously we don't know yet how it'll work, but I'm guessing
that there will be quite a few people whose idea of each context differs
quite a bit from the design.

Post by Chi Shang Cheng

Post by Jani-Matti Hätinen
However, this approach has the basic problem of 'If I can't see it, it's
not there'. As we all noticed with adaptive menus (and will probably
notice with the ribbon-thing) trying to remember where an invisible option
is is a lot harder than trying to find a visible one amongst even a lot of
others (especially if even the visible GUI elements are always moving as
was the case with adaptive menus).
Nevertheless IMHO both the Microsoft solutions as well as the koffice
First: moving GUI elements without user consent is just plain malice.
Second: infinite, repetitive menus (which offer no other distinction
between the different options apart from the actual text) aren't usable
for anyone (because if you were ever able to get the adaptive menus right,
it was a hell of a lot better than the traditional ones, until it changed
again, that is).
Third: Correlating the frequency of use for an action with the size of
the corresponding GUI element is a good thing. (This is demonstrated
really well in the koffice-1.5 startup dialog in which the most often used
buttons (Open existing document... and Open this document) are by far the
smallest GUI elements in the whole dialog)
So, merely changing GUI element sizes is IMHO a fairly good compromise
between optimising a GUI for the user (which most people actually want)
and not playing games with his/her sense of direction.

To summarize, your proposal raises many questions which cannot be answered
without doing any research. What is to be expected, is that the user
interface will look aesthetically less appealing, due to graphical design
issues such as inconsistency, balance, alignment etc. Of course, this would
not be the greatest drawback. If the nature of adaptation during the usage
lifecycle proves to be unable to enhance the usability of the user
interface, this idea should be discarded.
When a user-centered development methodology is applied when developing an
application, the user interface will be 'adapted' (or 'adjusted'), created
specifically for a group of users. Is further personalization required, and
if so, can it be done automatically? And if we were to implement an
auto-adaptative user interface, would the application be able to gather
data about the user that can be used usefully? If I had to answer that
question, I would have to honestly say that I don't think that it's doable.

Yes, as you described there are basically two big questions here. Are adaptive
interfaces doable and/or desirable? And if so, are adaptive toolbar/menu item
sizes the right approach (or one of them) to be taken in implementing them.

[Jani-Matti Hätinen]

Jani-Matti H?tinen kirjoitti viestiss??n (l?hetysaika keskiviikko, 24.

[SNIP]
The adaptive GUI element sizes could be further accompanied with hints,
which would make it easy for a user to add an often used menu item into the
toolbar, or remove a rarely used toolbar item, to be only accessible from
the menu. Note that the movement options should also be accessible from the
same place even when no hint would be visible (possibly through a context
menu which the hints could refer to)

I posted a mockup of a more flexible toolbar approach along with the context
menu described above to KDE-Look. You can check it out at:
http://www.kde-look.org/content/show.php?content=42474
I'll try post more mockups and updates on that topic fairly soon.