Visual touch or sensing with the eyes

The notion that it is possible to ‘see with the hands’, as Descartes once put it in Dioptrique (1637, see Paterson 2016), chimes with the popular imagination of the sense of touch as somehow enhanced in people with impaired vision. It is linked to centuries of philosophical debate and scientific research of how blind people ‘visualise’, what their sense of spatiality might be, and how touch may be part of providing and processing wider environmental information. These are in parts deeply political questions: labelling people as blind and trying to allow them to see in other ways is considered by many as simultaneously constructing them as ‘lacking’ fundamental sensory functions (Paterson 2016), as having a defect. While this deficit model is problematised as indicative of societies in which emphasis is put on vision as a dominant sense, it has certainly fuelled a lasting interest in cross-modal perception and, in the last forty years or so, the development of sensory substitution technologies.

Paul Bach-y-Rita was a pioneer of neuroplasticity, or the idea that the brain continually adapts and reorganises itself through new neurological connections in the course of a lifetime. He was perhaps best known for his development of Tactile-Visual Sensory Substitution (TVSS) systems, including a chair that translates visual into tactile images and conveys these through vibration to the subject’s back.

More recent developments of this idea include tongue-machine electro-tactile interfaces, such as the commercially available BrainPort. As its manufacturer points out, it doesn’t enable vision per se but is classified as an ‘oral electronic vision aid’, working like a ‘refreshable Braille display from which you learn to interpret the bubble-like patterns on your tongue as representative of objects in [your] surroundings’ (http://www.wicab.com/faqs).

While the hands and finger tips are still commonly regarded as the main sensory mediators of touch, both in traditional Braille and with regard to emerging tactile displays (Xu et al. 2011), BrainPort and similar technologies demonstrate an interesting shift away from the hand to also incorporate other parts of the human body. In a related vein, Ingold (2004) and Macpherson (2009) have pointed to the ways in which blind people (and others) sense the world through their feet.

The above technologies and approaches to touch tap into a dialectic of both sensory isolation, that is, the notion of distinct modalities of touch, hearing or vision, and a merging of sensory categories. On the one hand, one sense or kind of sensory information is seen to replace the other. On the other hand, neuroscientific research points to ‘the extraordinarily rich interconnectedness and interactions of the sensory areas of the brain’, making it difficult to say ‘that anything is purely visual or purely auditory, or purely anything’ (Sacks 2011, 238)

To sensory anthropologists and particularly those engaging in phenomenological anthropology, the merging of senses is nothing new, partly because the Western 5-sensorium (of taste, touch, smell, vision and hearing) has been revealed as a cultural construct. Accordingly, sensory categories are not universal givens but are socially and culturally roduced and experienced. A multimodal social semiotic perspective recognises the social and cultural contingency of the senses. It attends to the ways in which people take up and use the sensory categories and the resources that are available to them to develop communicational modes that meet their social meaning functions (Jewitt, in press). Bearing in mind their cultural distinction, and social use, if touch apparently offers a route to visual imagery and imagination, what – if anything – can vision tell us about touch?

Within film studies, Laura Marks has advanced the concept of ‘haptic cinema’ or ‘haptic visuality’. The latter is described as ‘a sense of physical touching or being touched engendered by an organization of the film image in which its material presence is foregrounded and which evokes close engagement with the surface detail and texture’ (Kuhn and Westwell 2015). As Marks explains:

Haptic looking tends to rest on the surface of its object rather than plunge into depth, tends not to distinguish form so much as discern texture. It is a labile, plastic sort of look, more inclined to move than focus. The video works I propose to call haptic invite a look that moves on the surface plane of the screen for some time before the viewer realizes what it is she is beholding. Haptic video […] invites [a] caressing look. (Marks 1998: 338)

This notion of embodied viewing is also seen to test ‘viewer’s own sense of separation between self and image’ (ibid). Marks draws inspiration from the phenomenology of film and the sensation of ‘synaesthesia’ as advanced, amongst others, by Vivien Sobchack. ‘The lived-body’, Sobchack writes, ‘does not have [separate] senses. It is, rather, sensible’. (Sobchack 1991, 77)

Dyer and Pink have further argued that it is film’s ‘mimetic’ (Marks 2000) properties that render it ‘capable of drawing us into sensory participation with its world’ (Dyer and Pink 2015). As such, ‘the experience of film does not simply involve us looking at something that is external to us, but it is through the affordances of film that, in relation to the other constituents of our environments/worlds that viewing becomes meaningful’ (ibid). The visual, in this sense, is more than a text that can be viewed and interpreted, it is an environment that can be known and experienced:

Should the drawing or painting be understood as a final image to be inspected and interpreted, as is conventional in studies of visual culture, or should we rather think of it as a node in a matrix of trails to be followed by observant eyes? Are drawings or paintings of things in the world, or are they like things in the world, in the sense that we have to find our way through and among them, inhabiting them as we do the world itself? (Ingold 2010, 16)

While anthropological film has long sought to create empathetic entry points for viewers across cultural boundaries, this notion of inhabiting a visual medium goes beyond empathy to include a corporeal dimension akin to that described by filmmaker David MacDougall. Accordingly, visual media allow us to ‘construct knowledge not by “description” […] but by a form of “acquaintance”’ (MacDougall 2006, 220). Vision in itself has of course been conceptualised as a situated practice that can be trained to become ‘skilled’ (Grasseni 2010).

As we are developing our methodology for exploring the situated meanings and affordances of digital touch, the above considerations are at the forefront of our mind, specifically as regards the possibilities of employing arts-based and video-based methods in our fieldwork. Arts-based methods (e.g. cultural design probes, design provocations, future scenarios, and prototyping) will be employed to elicit a response (e.g. tactile sensations) from participants, engaging their imagination and encouraging a fuller physical response in ways that more formal interview and questionnaire techniques cannot achieve.

Videos will be useful to create audio-visual records of people’s interactions with touch technologies, which we can then revisit and explore as a team. Beyond this, however, we are interested in making use of video as part of a sensory-ethnographically inspired set of methods, including video tours or re-enactments (Pink and Leder Mackley 2014). Designed, framed and analysed through a sensory-ethnographic lens, these can provide additional routes towards empathic-embodied understandings of touch as they are performed and demonstrated in relation to the objects, technologies and environments in which touch occurs.

References

Dyer, Adrian G, and Sarah Pink. 2015. ‘Movement, Attention and Movies: The Possibilities and Limitations of Eye Tracking?’ Refractory: A Journal of Entertainment Media 25 (February). http://refractory.unimelb.edu.au/2015/02/06/dyer-pink/.

Grasseni, Cristina. 2010. Skilled Visions: Between Apprenticeship and Standards. New York; Oxford: Berghahn Books.

Ingold, Tim. 2004. ‘Culture on the Ground: The World Perceived Through the Feet’. Journal of Material Culture 9 (3): 315–40. doi:10.1177/1359183504046896.

———. 2010. ‘Ways of Mind-Walking: Reading, Writing, Painting’. Visual Studies 25 (1): 15–23. doi:10.1080/14725861003606712.

Jewitt, Carey. 2017 (in press). ‘Towards a Multimodal Social Semiotic Agenda for Touch’. In Advancing Multimodal and Critical Discourse Studies, edited by Sumin Zhao, Emilia Djonov, Anders Björkvall, and Morten Boeriis. London: Routledge.

Kuhn, Annette, and Guy Westwell. 2015. Dictionary of Film Studies – Oxford Reference. Oxford. http://www.oxfordreference.com/view/10.1093/acref/9780199587261.001.0001/acref-9780199587261.

MacDougall, David. 2006. The Corporeal Image: Film, Ethnography, and the Senses. Princeton, N.J.: Princeton University Press.

Macpherson, Hannah. 2009. ‘Articulating Blind Touch: Thinking through the Feet’. The Senses and Society 4 (2): 179–93. doi:10.2752/174589309X425120.

Paterson, Mark. 2016. Seeing with the Hands: Blindness, Vision and Touch After Descartes. 1 edition. Edinburgh: Edinburgh University Press.

Pink, Sarah, and Kerstin Leder Mackley. 2014. ‘Re-Enactment Methodologies for Everyday Life Research: Art Therapy Insights for Video Ethnography’. Visual Studies 29 (2): 146–54. doi:10.1080/1472586X.2014.887266.

Sacks, Oliver W. 2011. The Mind’s Eye. London: Picador.

Sobchack, Vivian. 1991. The Address of the Eye: A Phenomenology of Film Experience. Princeton, NJ: Princeton University Press.

Xu, Cheng, Ali Israr, Ivan Poupyrev, Olivier Bau, and Chris Harrison. 2011. ‘Tactile Display for the Visually Impaired Using TeslaTouch’. In CHI’11 Extended Abstracts on Human Factors in Computing Systems, 317–322. ACM. http://dl.acm.org/citation.cfm?id=1979705.