The setting
The ongoing innovations in tablets, virtual reality (VR), and smart glasses are revolutionizing the way we interact with digital content, signaling a critical need for new types of key inputs. Traditional mouse and keyboard setups, while effective for desktop computing, fall short in providing the intuitive, immersive experiences that modern technologies aim to deliver. As these cutting-edge devices offer more natural, gesture-based, and spatial interactions, input methods must evolve to fully leverage their capabilities, enhancing user engagement and efficiency. This shift necessitates the development of input technologies that can accurately interpret hand movements, voice commands, and even eye tracking, thereby offering a seamless integration between the user and the digital world. By embracing innovative key input solutions, we can unlock the full potential of current and future digital devices, ensuring they meet the demands of users in this rapidly evolving technological landscape.
Leveraging the multitouch capabilities and Bluetooth functionality of mobile phones presents a practical solution to the challenge of finding suitable input methods for emerging technologies like tablets, VR, and smart glasses. By transforming these ubiquitous devices into versatile input tools, we can utilize the sophisticated touch interfaces and sensors already embedded in smartphones to offer a wide range of interactive gestures and controls. This approach not only taps into the vast existing ecosystem of smartphones, minimizing the need for additional hardware investments but also provides a familiar, intuitive user experience. With smartphones acting as a bridge, users can effortlessly navigate, control, and interact with new digital environments, from swiping through virtual reality landscapes to selecting options in augmented reality menus. This solution embodies a forward-thinking strategy that harnesses the power and prevalence of mobile phones to meet the evolving demands of digital interaction in an efficient and user-friendly manner.
The keys
The actual characters are the following:
Alphanumeric Keys
- Letters: All 26 letters of the English alphabet (A-Z).
- Numbers: 0 through 9 (which often double as symbols when used with the Shift key).
Modifier Keys
- Shift: For accessing uppercase letters and secondary functions of number keys and other keys.
- Control (Ctrl): For keyboard shortcuts.
- Alt: For alternate character access and shortcuts.
- Windows Key (on Windows)/Command Key (on Mac): For operating system-specific functions.
- Option Key (on Mac): Similar to the Alt key on Windows, for accessing alternate characters and shortcuts.
Function Keys
- F1 through F12: Used for a variety of shortcuts and special functions in different applications.
Navigation Keys
- Arrow Keys: Up, Down, Left, Right for navigation.
- Home, End, Page Up, Page Down: For navigating text documents and web pages.
Utility Keys
- Enter/Return: For commands and entering information.
- Backspace: For deleting the character to the left of the cursor.
- Delete: For deleting the character to the right of the cursor (or the selected item).
- Tab: For indentation, navigation between fields in forms, and switching between elements in some applications.
- Escape (Esc): For canceling or closing a command/menu.
Punctuation and Special Characters
- Space Bar: For spaces between words.
- Comma (,), Period (.), Semicolon (;), Colon (:), Question Mark (?), Exclamation Mark (!)
- Single (‘) and Double (“) Quotes
- Slash (/) and Backslash ()
- Bracket pairs: Parentheses
(), Square Brackets[], Curly Braces{}, Angle Brackets<>(the last one might be less common and can be accessed through a combination of keys depending on the keyboard layout). -
Minus (-), Equal (=), Underline (_), Plus (+), Asterisk (*), Ampersand (&), Percent (%), Dollar Sign ($), Pound Sign (#), At Symbol (@), Tilde (~), Grave Accent (`), **Pipe ( ), **Caret (^)
Numeric Keypad (Optional for Full-Size Keyboards)
- Numbers 0 through 9: Separate from the alphanumeric section, used for numerical entry.
- Decimal Point (.)
- Plus (+), Minus (-), Multiply (*), Divide (/) keys
- Enter: Sometimes a separate Enter key is included in the numeric keypad for convenience.
Then if we look at the actual frequency of those characters, we can see that the difference between them is high (Figure 1 and Table 1). Therefore, we can afford different usabilities or easiness of reach for each of those characters.
Figure 1: Frequency of characters. Source(https://pi.math.cornell.edu/~mec/2003-2004/cryptography/subs/frequencies.html)
| Score per 1000 running words | |
|---|---|
| Average | |
| . full stop | 65.3 |
| , comma | 61.6 |
| ; semi-colon | 3.2 |
| : colon | 3.4 |
| ! exclamation | 3.3 |
| ? question | 5.6 |
| ’ apostrophe/ single quotation | 24.3 |
| “ double quotation | 26.7 |
| - hyphen | 15.3 |
| TOTAL | 208.7 |
| score = divided by total words in 1000s, .05 rounded up |
Table 1: Frequency of special characters. Source: http://www.viviancook.uk/Punctuation/PunctFigs.htm
We can also divide those characters by their usage (in an arbitrary and non rigurous way):
- High usage: Letters, Utility Keys, Space Bar
- Mid usage: Numbers, Modifier Keys, Navigation Keys, Punctuation and Special Characters
- Low usage: Function Keys
The solution
If you divide the screen in 2 grids of 3x2, you have 12 keys, 6 that can be used with the right hand and 6 that can be used with the left hand. Also, this grid is really simple and it can be used without having to look at the screen as per seen in figure 3.
Figure 3: Thumbs on simulated grid
Then by using one or the other hand, 12 letters can be represented. Then, by using both fingers 6x6 characters can be represented. So up to 48 characters could be triggered by the keyboard. Now, we would still need additional buttons for this.
Figure 4: Keys for most common letters
However, there’s a way to represent even more characters. We could allocate one of the buttons to make the other board to become a one stroke drawing board and then have an infinite number of character to be drawn as per figure 4. This would reduce the number of buttons to 5x6=30 +12 = 42 … but still plenty of characters for fast typing.
Figure 5: Gestures based characters
Regarding capital letters, wee were envisioning to have a small upward gesture instead of a press that would signify that the letter must be capital.
Also, we could envision as well a mouse mode with another gesture, but we can leave that for a future exercise.
The implementation
This could be implemented as a standard native app that captures the touch events on the screen. The only difficulty is the interface to the computer which would receive the input. Our first thought came to bluetooth as it is pervasive and all computers can have access to it.
The question is: can a native app make the phone appear as a bluetooth device to a nearby computer. Existing apps suggest that:
- Yes for Android: https://play.google.com/store/apps/details?id=io.appground.blek
- Nope for Iphone (need to install a helper app on the PC): https://apps.apple.com/us/app/remote-mouse-keyboard/id629019487
- Nope for web app: https://stackoverflow.com/questions/70361906/can-i-create-a-mouse-and-keyboard-simulator-with-the-bluetooth-web-api
For an android implementation, we can inspire on the following repos: