Abstract

Emerging surface haptic display technology holds promise for enhancing manual interactions such as contour drawing, a task that requires simultaneous generation and sensing for accurate control of the emerging shape. Existing haptic technology enabling tactile drawing is primarily restricted to niche and expensive products for blind accessibility, and the advantages and optimal strategies of engaging with tactile feedback remain unclear. This study compares three tactile interaction methods for contour drawing using a refreshable pin array: input directly on the pin surface with a finger, input on the surface using a stylus, and input on a peripheral screen while the tactile feedback is felt on the pin surface with one’s other hand. A user study investigating contour generation performance using these three different methods found that individual contour properties had greater impacts on drawing accuracy than interaction methods and that accuracy does not always correlate with user perception of performance across methods. Experimental observations highlight several strategies and hand postures users adopted to leverage each interaction method. These results can serve as a guide to developing robust feedback systems for contour drawing on future iterations of surface haptic interface technology.

References

1.
Friesen
,
R. F.
,
Klatzky
,
R. L.
,
Peshkin
,
M. A.
, and
Colgate
,
J. E.
,
2021
, “
Building a Navigable Fine Texture Design Space
,”
IEEE Trans. Hapt.
,
14
(
4
), pp.
897
906
.
2.
Grigorii
,
R. V.
,
Klatzky
,
R. L.
, and
Colgate
,
J. E.
,
2022
, “
Data-Driven Playback of Natural Tactile Texture via Broadband Friction Modulation
,”
IEEE Trans. Hapt.
,
15
(
2
), pp.
429
440
.
3.
Sadia
,
B.
,
Emgin
,
S. E.
,
Sezgin
,
T. M.
, and
Basdogan
,
C.
,
2020
, “
Data-Driven Vibrotactile Rendering of Digital Buttons on Touchscreens
,”
Int. J. Human Comput. Stud.
,
135
, p.
102363
.
4.
Bhatnagar
,
T.
,
Higgins
,
A.
,
Marquardt
,
N.
,
Miodownik
,
M.
, and
Holloway
,
C.
,
2023
, “
Analysis of Product Architectures of Pin Array Technologies for Tactile Displays
,”
Proc. ACM Hum. Comput. Interact.
,
7
(
ISS
), pp.
135
155
.
5.
Shultz
,
C.
, and
Harrison
,
C.
,
2023
, “
Flat Panel Haptics: Embedded Electroosmotic Pumps for Scalable Shape Displays
,”
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
,
Hamburg, Germany
,
Apr. 19
, Association for Computing Machinery.
6.
Suwa
,
M.
, and
Tversky
,
B.
,
1997
, “
What Do Architects and Students Perceive in Their Design Sketches? A Protocol Analysis
,”
Des. Stud.
,
18
(
4
), pp.
385
403
.
7.
Tversky
,
B.
,
2002
, “
What Do Sketches Say About Thinking
,”
2002 AAAI Spring Symposium
,
Palo Alto, CA
,
Mar. 25
, p.
151
.
8.
Tversky
,
B.
,
Suwa
,
M.
,
Agrawala
,
M.
,
Heiser
,
J.
,
Stolte
,
C.
,
Hanrahan
,
P.
, and
Phan
,
D.
,
2003
, “Sketches for Design and Design of Sketches,”
Human Behaviour in Design
,
Springer
, pp.
79
86
.
9.
Goldschmidt
,
G.
,
1994
, “
On Visual Design Thinking: The Vis Kids of Architecture
,”
Des. Stud.
,
15
(
2
), pp.
158
174
.
10.
Yasin
,
R. M.
,
Halim
,
L.
, and
Ishar
,
A.
,
2012
, “
Effects of Problem-Solving Strategies in the Teaching and Learning of Engineering Drawing Subject
,”
Asian Soc. Sci.
,
8
(
16
), p.
65
.
11.
Goel
,
V.
,
1995
,
Sketches of Thought Mit Press
,
The MIT Press
,
Cambridge, MA
.
12.
Maslych
,
M.
,
Taranta
,
E. M.
,
Aldilati
,
M.
, and
Laviola
,
J. J.
,
2023
, “
Effective 2D Stroke-Based Gesture Augmentation for RNNs
,”
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
,
Hamburg, Germany
,
Apr. 19
, Associationfor Computing Machinery.
13.
Vatavu
,
R.-D.
,
2017
, “
Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision
,”
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
,
Denver, CO
,
May 2
.
14.
Elsayed
,
H.
,
Barrera Machuca
,
M. D.
,
Schaarschmidt
,
C.
,
Marky
,
K.
,
Müller
,
F.
,
Riemann
,
J.
,
Matviienko
,
A.
,
Schmitz
,
M.
,
Weigel
,
M.
, and
Mühlhäuser
,
M.
,
2020
, “
Vrsketchpen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments
,”
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology
,
Virtual Event Canada
,
Nov. 1
, Association for Computing Machinery.
15.
Mohanty
,
R. R.
,
Castillo
,
R. M.
,
Ragan
,
E. D.
, and
Krishnamurthy
,
V. R.
,
2020
, “
Investigating Force-Feedback in Mid-Air Sketching of Multi-Planar Three-Dimensional Curve-Soups
,”
ASME J. Comput. Inf. Sci. Eng.
,
20
(
1
), p.
011010
.
16.
Yeh
,
L.
,
Chu
,
C.-H.
, and
Huang
,
Y.-T.
,
2024
, “
Assistive Sensory Feedback for Trajectory Tracking in Augmented Reality
,”
ASME J. Comput. Inf. Sci. Eng.
,
24
(
3
), p.
031001
.
17.
Richard
,
G.
,
Pietrzak
,
T.
,
Argelaguet
,
F.
,
Lécuyer
,
A.
, and
Casiez
,
G.
,
2021
, “
Studying the Role of Haptic Feedback on Virtual Embodiment in a Drawing Task
,”
Front. Virtual Real.
,
1
, p.
573167
.
18.
Dolphin
,
S.
,
Downing
,
M.
,
Cirrincione
,
M.
,
Samuta
,
A.
,
Leite
,
K.
,
Noble
,
K.
, and
Walsh
,
B.
,
2024
, “
Information Accessibility in the Form of Braille
,”
IEEE Open J. Eng. Med. Biol.
,
5
, pp.
205
209
.
19.
Phutane
,
M.
,
Wright
,
J.
,
Castro
,
B. V.
,
Shi
,
L.
,
Stern
,
S. R.
,
Lawson
,
H. M.
, and
Azenkot
,
S.
,
2022
, “
Tactile Materials in Practice: Understanding the Experiences of Teachers of the Visually Impaired
,”
ACM Trans. Access. Comput.
,
15
(
3
), pp.
1
34
.
20.
Raynal
,
M.
,
Ducasse
,
J.
,
Macé
,
M. J.-M.
,
Oriola
,
B.
, and
Jouffrais
,
C.
,
2024
, “
The Flexiboard: Tangible and Tactile Graphics for People With Vision Impairments
,”
Multimodal Technol. Interact.
,
8
(
3
), p.
17
.
21.
Lee
,
W.-C.
,
Hung
,
C.-W.
,
Ting
,
C.-H.
,
Chi
,
P.
, and
Chen
,
B.-Y.
,
2023
, “
TacNote: Tactile and Audio Note-Taking for Non-Visual Access
,”
UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
,
San Francisco, CA
,
Oct. 29
, Association for Computing Machinery.
22.
Ramôa
,
G.
,
Moured
,
O.
,
Schwarz
,
T.
,
Müller
,
K.
, and
Stiefelhagen
,
R.
,
2023
, “
Enabling People With Blindness to Distinguish Lines of Mathematical Charts With Audio-Tactile Graphic Readers
,”
PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments
,
Corfu, Greece
,
Aug. 10
, Association for Computing Machinery, pp.
384
391
.
23.
Zhao
,
Y.
,
Nacenta
,
M. A.
,
Sukhai
,
M. A.
, and
Somanath
,
S.
,
2024
, “
Tada: Making Node-Link Diagrams Accessible to Blind and Low-Vision People
,”
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
,
Honolulu, HI
,
May 11
, Association for Computing Machinery.
24.
Strohmeier
,
P.
, and
Hornbæk
,
K.
,
2017
, “
Generating Haptic Textures With a Vibrotactile Actuator
,”
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
,
Denver, CO
,
May 2
, pp.
4994
5005
.
25.
Culbertson
,
H.
,
Unwin
,
J.
, and
Kuchenbecker
,
K. J.
,
2014
, “
Modeling and Rendering Realistic Textures From Unconstrained Tool-Surface Interactions
,”
IEEE Trans. Hapt.
,
7
(
3
), pp.
381
393
.
26.
Bau
,
O.
,
Poupyrev
,
I.
,
Israr
,
A.
, and
Harrison
,
C.
,
2010
, “
TeslaTouch: Electrovibration for Touch Surfaces
,”
UIST '10: Proceedings of the 23nd Annual ACM Symposium on User interface Software and Technology
,
New York
,
Oct. 3
, Association for Computing Machinery, pp.
283
292
.
27.
Vardar
,
Y.
,
Güçlü
,
B.
, and
Basdogan
,
C.
,
2017
, “
Effect of Waveform on Tactile Perception by Electrovibration Displayed on Touch Screens
,”
IEEE Trans. Hapt.
,
10
(
4
), pp.
488
499
.
28.
Shultz
,
C.
,
Peshkin
,
M.
, and
Colgate
,
J. E.
,
2018
, “
The Application of Tactile, Audible, and Ultrasonic Forces to Human Fingertips Using Broadband Electroadhesion
,”
IEEE Trans. Hapt.
,
11
(
2
), pp.
279
290
.
29.
Wiertlewski
,
M.
,
Friesen
,
R. F.
, and
Colgate
,
J. E.
,
2016
, “
Partial Squeeze Film Levitation Modulates Fingertip Friction
,”
Proc. Natl. Acad. Sci. U. S. A.
,
113
(
33
), pp.
9210
9215
.
30.
Biet
,
M.
,
Giraud
,
F.
, and
Lemaire-Semail
,
B.
,
2007
, “
Squeeze Film Effect for the Design of an Ultrasonic Tactile Plate
,”
IEEE Trans. Ultrason., Ferroelectr. Freq. Control
,
54
(
12
), pp.
2678
2688
.
31.
Tanvas
,
2024
, “Tanvastouch.” https://tanvas.co/, Accessed February 28, 2024.
32.
Gershon
,
P.
,
Klatzky
,
R. L.
,
Palani
,
H.
, and
Giudice
,
N. A.
,
2016
, “
Visual, Tangible, and Touch-Screen: Comparison of Platforms for Displaying Simple Graphics
,”
Assist. Technol.
,
28
(
1
), pp.
1
6
.
33.
Sadia
,
B.
,
Sadic
,
A.
,
Ayyildiz
,
M.
, and
Basdogan
,
C.
,
2022
, “
Exploration Strategies for Tactile Graphics Displayed by Electrovibration on a Touchscreen
,”
Int. J. Hum. Comput. Stud.
,
160
, p.
102760
.
34.
Tennison
,
J. L.
, and
Gorlewicz
,
J. L.
,
2019
, “
Non-Visual Perception of Lines on a Multimodal Touchscreen Tablet
,”
ACM Trans. Appl. Percept.
,
16
(
1
), pp.
1
9
.
35.
Holloway
,
L.
,
Ananthanarayan
,
S.
,
Butler
,
M.
,
De Silva
,
M. T.
,
Ellis
,
K.
,
Goncu
,
C.
,
Stephens
,
K.
, and
Marriott
,
K.
,
2022
, “
Animations At Your Fingertips: Using a Refreshable Tactile Display to Convey Motion Graphics for People Who Are Blind or Have Low Vision
,”
ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility
,
Athens, Greece
,
Oct. 22
, Association for Computing Machinery.
36.
Yoshida
,
S.
,
Sun
,
Y.
, and
Kuzuoka
,
H.
,
2020
, “
Pocopo: Handheld Pin-Based Shape Display for Haptic Rendering in Virtual Reality
,”
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
,
Honolulu, HI
,
Apr. 23
, Association for Computing Machinery, pp.
1
13
.
37.
Jung
,
J.
,
Son
,
S.
,
Lee
,
S.
,
Kim
,
Y.
, and
Lee
,
G.
,
2021
, “
ThroughHand: 2D Tactile Interaction to Simultaneously Recognize and Touch Multiple Objects
,”
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
,
Yokohama, Japan
,
May 7
, Association for Computing Machinery.
38.
Siu
,
A. F.
,
Kim
,
S.
,
Miele
,
J. A.
, and
Follmer
,
S.
,
2019
, “
shapeCAD: An Accessible 3D Modelling Workflow for the Blind and Visually-Impaired Via 2.5D Shape Displays
,”
ASSETS '19: Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility
,
Pittsburgh, PA
,
Oct. 24
, Association for Computing Machinery, pp.
342
354
.
39.
Alzalabny
,
S.
,
Moured
,
O.
,
Müller
,
K.
,
Schwarz
,
T.
,
Rapp
,
B.
, and
Stiefelhagen
,
R.
,
2024
, “
Designing a Tactile Document UI for 2D Refreshable Tactile Displays: Towards Accessible Document Layouts for Blind People
,”
Multimodal Technol. Interact.
,
8
(
11
), p.
102
.
40.
Orbit Research
,
2024
, “Graphiti.” https://www.orbitresearch.com/product/graphiti/. Accessed January 20, 2024.
41.
Bornschein
,
J.
,
Bornschein
,
D.
, and
Weber
,
G.
,
2018
, “
Comparing Computer-Based Drawing Methods for Blind People With Real-Time Tactile Feedback
,”
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
,
Montreal, QC, Canada
,
Apr. 19
, Association for Computing Machinery, pp.
1
13
.
42.
Humanware Group
,
2024
, “Humanware Monarch.” https://www.humanware.com/en-usa/monarch. Accessed January 20, 2024.
43.
Dot Pad Inc.
,
2024
, “Dot Pad.” https://pad.dotincorp.com/, Accessed January 20, 2024.
44.
Picard
,
D.
,
Martin
,
P.
, and
Tsao
,
R.
,
2014
, “
iPads at School? A Quantitative Comparison of Elementary Schoolchildren’s Pen-on-Paper Versus Finger-on-Screen Drawing Skills
,”
J. Educ. Comput. Res.
,
50
(
2
), pp.
203
212
.
45.
Badam
,
S. K.
,
Chandrasegaran
,
S.
,
Elmqvist
,
N.
, and
Ramani
,
K.
,
2014
, “
Tracing and Sketching Performance Using Blunt-Tipped Styli on Direct-Touch Tablets
,”
AVI '14: Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces
,
Como, Italy
,
May 27
, Association for Computing Machinery, pp.
193
200
.
46.
Kara
,
L. B.
, and
Stahovich
,
T. F.
,
2005
, “
An Image-Based, Trainable Symbol Recognizer for Hand-Drawn Sketches
,”
Comput. Graph.
,
29
(
4
), pp.
501
517
.
47.
Hammond
,
T.
,
Kumar
,
S. P. A.
,
Runyon
,
M.
,
Cherian
,
J.
,
Williford
,
B.
,
Keshavabhotla
,
S.
,
Valentine
,
S.
,
Li
,
W.
, and
Linsey
,
J.
,
2018
, “
It’s Not Just About Accuracy: Metrics That Matter When Modeling Expert Sketching Ability
,”
ACM Trans. Interact. Intell. Syst.
,
8
(
3
), pp.
1
47
.
48.
Mangiafico
,
S. S.
,
2016
, “
Summary and Analysis of Extension Program Evaluation in R
,” Rutgers Cooperative Extension: New Brunswick, NJ, USA,
125
, pp.
16
22
. http://rcompanion.org/handbook/
49.
Kern
,
F.
,
Kullmann
,
P.
,
Ganal
,
E.
,
Korwisi
,
K.
,
Stingl
,
R.
,
Niebling
,
F.
, and
Latoschik
,
M. E.
,
2021
, “
Off-the-Shelf Stylus: Using Xr Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces
,”
Front. Virtual Real.
,
2
, p.
684498
.
50.
Wu
,
P.-C.
,
Wang
,
R.
,
Kin
,
K.
,
Twigg
,
C.
,
Han
,
S.
,
Yang
,
M.-H.
, and
Chien
,
S.-Y.
,
2017
, “
Dodecapen: Accurate 6D of Tracking of a Passive Stylus
,”
UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology
,
Québec City, QC, Canada
,
Oct. 20
, Association for Computing Machinery, pp.
365
374
.
51.
O’Modhrain
,
S.
,
Giudice
,
N. A.
,
Gardner
,
J. A.
, and
Legge
,
G. E.
,
2015
, “
Designing Media for Visually-Impaired Users of Refreshable Touch Displays: Possibilities and Pitfalls
,”
IEEE Trans. Hapt.
,
8
(
3
), pp.
248
257
.
52.
D’Angiulli
,
A.
, and
Maggi
,
S.
,
2003
, “
Development of Drawing Abilities in a Distinct Population: Depiction of Perceptual Principles by Three Children With Congenital Total Blindness
,”
Int. J. Behav. Dev.
,
27
(
3
), pp.
193
200
.
You do not currently have access to this content.