Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures

Chao Ming Wang; Hao Weng Pang

doi:doi:10.11648/j.hep.20200504.12

| Peer-Reviewed

Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures

Chao Ming Wang, Hao Weng Pang

Published in International Journal of Health Economics and Policy (Volume 5, Issue 4)

Received: 11 December 2020 Accepted: 23 December 2020 Published: 31 December 2020

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user’s physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user’s socio-pleasure. By public exhibitions, users’ feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.

Published in	International Journal of Health Economics and Policy (Volume 5, Issue 4)
DOI	10.11648/j.hep.20200504.12
Page(s)	80-95
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Visually Impaired, Interactive Technology, Sound Art, Somatosensory Interaction, User Interface, Hand-Gesture Recognition

References

[1]	H. Gardner, The Arts and Human Development. New York, NY, USA: Wiley, 1983.
[2]	P. Jennifer, R. Yvonne, and S. Hellen. Interaction Design: Beyond Human Computer Interaction, 5th ed. New York, NY, USA: Wiley, 2019.
[3]	L. C. Lee, K. C. Hao, and Y. H. Cheng, “A comparative study on game experience of searching and dodging missions between different control interfaces-design major college students as an example,” Journal of Design, vol. 3, pp. 1-22, 2012.
[4]	C. M. Wang, S. M. Tseng, and C. S. Huang, “Design of an interactive nostalgic amusement device with user-friendly tangible interfaces for improving the health of older adults,” Healthcare, vol. 8., article 179, 2020.
[5]	V. Levesque, “Blindness, technology and haptics,” Haptics Laboratory, Center for Intelligent Machines, McGill University, Montréal, Québec, Canada, Technical Report No. CIM-TR-05.08, pp. 19-21, 2005.
[6]	G. R. S. Murthy and R. S. Jadon, “A review of vision-based hand gestures recognition,” International Journal of Information Technology and Knowledge Management, vol. 2, pp. 405-410, 2009.
[7]	P. J. Lai (2012). “Digital Art Festival Taipei 2012.” Available: https://digitalartfestival.tw/daf12/home_en.html. Accessed: Apr. 7, 2018.
[8]	B. Traubeck (2012). “Years.” Available: http://traubeck.com/years/. Accessed: Apr. 14, 2018.
[9]	C. F. Wu, C. Y. Wu and Y. H. Tu, “A study on designing the optimal parameters of hierarchical menu system on 3C products for visually impaired people,” Journal of Ergonomic Study, vol. 10, pp. 31-39, 2008.
[10]	J. J. Gibson, The Senses Considered as Perceptual Systems. Boston, MA, USA: Hough Mifflin Harcourt, 1966.
[11]	J. C. Schacher, “Gesture control of sounds in 3D space,” in Proceedings of the 7th International Conference on New Interfaces for Musical Expression, New York, NY, UAS, June 6-10, 2007, pp. 358-362.
[12]	S. Hashimoto and H. Sawada, “A grasping device to sense hand gesture for expressive sound generation,” Journal of New Music Research, vol. 34, pp. 115-123, 2005.
[13]	B. Amento, W. Hill and L. Terveen, “The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface,” in Extended abstracts of the 2002 Conference on Human Factors in Computing Systems (CHI 2002), Minneapolis, MN, USA, Apr. 20-25, 2002, pp. 724-725.
[14]	B. Di Donato, J. Dooley, J. Hockman and S. Hall, “Myospat: A hand-gesture controlled system for sound and light projections manipulation,” in Proceedings of International Computer Music Conference (ICMC), Shanghai, China, Oct. 16-20, 2017, pp. 335-340.
[15]	A. Bouenard, M. M. Wanderley and S. Gibet, “Gesture control of sound synthesis: Analysis and classification of percussion gestures,” Acta Acustica united with Acustica, vol. 96, pp. 668-677, 2010.
[16]	A. Withagen, M. P. J. Vervloed, N. M. Janssen, H. Knoors and L. Verhoeven, “The tactual profile: development of a procedure to assess the tactual functioning of children who are blind,” The British Journal of Visual Impairment, vol. 27, pp. 221-238, 2009.
[17]	M. M. Wan, Visually Impaired Education. Taipei, Taiwan: Wu-Nan Book Inc., 2001.
[18]	R. L. Mandryk, “Objectively evaluating entertainment technology,” in Proceedings of 2004 Conference on Human Factors in Computing Systems, Vienna, Austria, Apr. 24-29, 2004, pp. 1057-1058.
[19]	K. Sanders and M. Casamassina (2006). “US Wii Price, Launch Date Revealed.” [Online]. Available: https://www.ign.com/articles/2006/09/14/us-wii-price-launch-date-revealed. Accessed: May 25, 2018.
[20]	J. Chen (2010). “Microsoft Xbox 360 Kinect Launches November 4.” [Online]. Available: https://gizmodo.com/microsoft-xbox-360-kinect-launches-november-4-5563148. Accessed: March 25, 2018.
[21]	ASUS (2011). “Xtion Pro.” [Online]. Available: https://www.asus.com/3D-Sensor/Xtion_PRO/. Accessed: March 25, 2018.
[22]	A. Davis (2014). “How to build your own leap motion art installation.” [Online]. Available: http://blog.leapmotion.com/how-to-build-your-own-leap-motion-art-installation/. Accessed: July 20, 2018.
[23]	North (2014). “Developing with Myo.” [Online]. Available: https://support.getmyo.com/hc/en-us/sections/200934739-Developing-With-Myo. Accessed: June 15, 2018.
[24]	Nintendo (2019). “Ring Fit Adventure.” [Online]. Available: https://ringfitadventure.nintendo.com/. Accessed: March 25, 2018.
[25]	O. Ashanti (2011). “This is Beatjazz.” [Online]. Available: https://www.ted.com/talks/onyx_ashanti_this_is_beatjazz. Accessed: April 7, 2018.
[26]	F. Visnjic (2011). “Kulbuto.” [Online]. Available: http://www.creativeapplications.net/sound/kulbuto-quarz-composer-sound/. Accessed: March 15, 2018.
[27]	F. Visnjic (2009). “Sound Storm.” [Online]. Available: http://www.creativeapplications.net/processing/multi-touch-sound-storm-processing-sound/. Accessed: March 20, 2018.
[28]	F. Zajeg (2013). “Patatap.” [Online]. Available: http://www.patatap.com/. Accessed: March 20, 2018.
[29]	T. Verbruggen (2009). “Crackle Canvas.” [Online]. Available: http://www.creativeapplications.net/environment/crackle-canvas-environment-sound/. Accessed: March 20, 2018.
[30]	C. Jannotta (2014). “12 - Sound sequencer comprised of 12 digitally controlled music boxes.” [Online]. Available: http://www.creativeapplications.net/sound/12-sound-sequencer-comprised-of-12-digitally-controlled-music-boxes/. Accessed: March 20, 2018.
[31]	M. Behrens (2006). “The Unknown.” [Online]. Available: http://www.marcbehrens.com/proj/the_unknown.html. Accessed: March 20, 2018.
[32]	User Studio (2017). “Dirty Tangible Interfaces.” [Online]. Available: https://www.smallab.org/dirti/. Accessed: July17, 2018.
[33]	P. W. Jordan, “Human factors for pleasure in product use,” Applide Ergonomics, vol. 29, pp. 25-33, 1998.
[34]	H. N. Schifferstein and E. P. Zwartkruis-Pelgrim, “Consumer-product attachment: measurement and design implications,” International Journal of Design, vol. 2, pp. 1-13, 2008.
[35]	D. A. Norman, Emotional Design: Why We Love (or Hate) Everyday Things. New York, NY, USA: Basic Books, 2004.
[36]	P. W. Jordan, Designing Pleasureable Products: An Introduction to the New Human Factors. Boca Raton, FL, USA: CRC Press, 2002.
[37]	F. Yuang, Social Research Methods. Taipei, Taiwan: Wu Nan Books 2002.
[38]	Wikipedia (2018). “Leap Motion.” [Online]. Available: https://en.wikipedia.org/wiki/Leap_Motion. Accessed: July 20, 2018.
[39]	Wikipedia (2018). “Pure Data.” [Online]. Available: https://en.wikipedia.org/wiki/Pure_Data. Accessed: July 20, 2018.
[40]	F. Weichert, D. Bachmann, B. Rudak and D. Fisseler, “Analysis of the accuracy and robustness of the Leap Motion controller,” Sensors, vol. 13, pp. 6380-6393, 2013.
[41]	https://jamiegledhill.tv/ (2019). “Being Human 2019 – Research.” [Online]. Available: https://jamiegledhill.tv/being-human-research/. Accessed: Dec. 20, 2019.

Cite This Article

Plain Text BibTeX RIS

APA Style

Chao Ming Wang, Hao Weng Pang. (2020). Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. International Journal of Health Economics and Policy, 5(4), 80-95. https://doi.org/10.11648/j.hep.20200504.12

Copy | Download

ACS Style

Chao Ming Wang; Hao Weng Pang. Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. Int. J. Health Econ. Policy 2020, 5(4), 80-95. doi: 10.11648/j.hep.20200504.12

Copy | Download

AMA Style

Chao Ming Wang, Hao Weng Pang. Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. Int J Health Econ Policy. 2020;5(4):80-95. doi: 10.11648/j.hep.20200504.12

Copy | Download

@article{10.11648/j.hep.20200504.12,
  author = {Chao Ming Wang and Hao Weng Pang},
  title = {Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures},
  journal = {International Journal of Health Economics and Policy},
  volume = {5},
  number = {4},
  pages = {80-95},
  doi = {10.11648/j.hep.20200504.12},
  url = {https://doi.org/10.11648/j.hep.20200504.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.hep.20200504.12},
  abstract = {Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user’s physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user’s socio-pleasure. By public exhibitions, users’ feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.},
 year = {2020}
}

Copy | Download

TY  - JOUR
T1  - Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures
AU  - Chao Ming Wang
AU  - Hao Weng Pang
Y1  - 2020/12/31
PY  - 2020
N1  - https://doi.org/10.11648/j.hep.20200504.12
DO  - 10.11648/j.hep.20200504.12
T2  - International Journal of Health Economics and Policy
JF  - International Journal of Health Economics and Policy
JO  - International Journal of Health Economics and Policy
SP  - 80
EP  - 95
PB  - Science Publishing Group
SN  - 2578-9309
UR  - https://doi.org/10.11648/j.hep.20200504.12
AB  - Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user’s physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user’s socio-pleasure. By public exhibitions, users’ feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.
VL  - 5
IS  - 4
ER  -

Copy | Download

Author Information

Chao Ming Wang

Department of Digital Media Design, National Yunlin University of Science & Technology, Yunlin, Taiwan
Hao Weng Pang

Department of Digital Media Design, National Yunlin University of Science & Technology, Yunlin, Taiwan

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

Chao Ming Wang, Hao Weng Pang. (2020). Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. International Journal of Health Economics and Policy, 5(4), 80-95. https://doi.org/10.11648/j.hep.20200504.12

Copy | Download

ACS Style

Chao Ming Wang; Hao Weng Pang. Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. Int. J. Health Econ. Policy 2020, 5(4), 80-95. doi: 10.11648/j.hep.20200504.12

Copy | Download

AMA Style

Chao Ming Wang, Hao Weng Pang. Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures. Int J Health Econ Policy. 2020;5(4):80-95. doi: 10.11648/j.hep.20200504.12

Copy | Download

@article{10.11648/j.hep.20200504.12,
  author = {Chao Ming Wang and Hao Weng Pang},
  title = {Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures},
  journal = {International Journal of Health Economics and Policy},
  volume = {5},
  number = {4},
  pages = {80-95},
  doi = {10.11648/j.hep.20200504.12},
  url = {https://doi.org/10.11648/j.hep.20200504.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.hep.20200504.12},
  abstract = {Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user’s physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user’s socio-pleasure. By public exhibitions, users’ feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.},
 year = {2020}
}

Copy | Download

TY  - JOUR
T1  - Designs of Interactive Sound Systems for the Visually Impaired to Enjoy Digital Art by Recognition of Hand Gestures
AU  - Chao Ming Wang
AU  - Hao Weng Pang
Y1  - 2020/12/31
PY  - 2020
N1  - https://doi.org/10.11648/j.hep.20200504.12
DO  - 10.11648/j.hep.20200504.12
T2  - International Journal of Health Economics and Policy
JF  - International Journal of Health Economics and Policy
JO  - International Journal of Health Economics and Policy
SP  - 80
EP  - 95
PB  - Science Publishing Group
SN  - 2578-9309
UR  - https://doi.org/10.11648/j.hep.20200504.12
AB  - Somatosensory interaction is a kind of man-machine interfacing technique for information acquisition through human postures, which are widely used in digital art nowadays. To create opportunities for the visually impaired to enjoy digital art, two sound art systems named Dynamic Sound and Concrete Sound, which are based on somatosensory technology, were designed in this study for the visually impaired to appreciate with pleasure. The former system emphasizes resonances between humans and sound, allowing the visually impaired user to trigger different sounds by hand gestures which promote the user’s physio-pleasure and ideo-pleasure. The latter system, also being controlled by hand gestures, combines sounds with three-state physical phenomena as feedbacks which are explained orally by an accompanying person to the visually impaired user, creating an inter-person communication that promotes the user’s socio-pleasure. By public exhibitions, users’ feedbacks were acquired via interviews, and evaluated to prove the effectiveness of the proposed systems with the following findings: 1) interactions by hand gestures offers the visually impaired with opportunities to enjoy digital art; 2) sound art provided by the systems allows the visually impaired to gain pleasure via man-machine interactions; 3) the systems innovatively integrate dynamic visual performances with auditory feedbacks in the interaction process; and 4) through the development of gesture operations, more performance techniques can be devised for sound art, allowing gesture motions to replace control interfaces in future designs.
VL  - 5
IS  - 4
ER  -

Copy | Download