Imagine being capable of storing and retrieving scientific papers, pictures, emails, music files, and in general all sort of multimedia data from familiar things in the path to work: buildings, corners, a park bench, trees and flowers – even people. Everything around you can be a “scaffold” to store and retrieve personal data – in fact, everything around you is already priming the brain for a particular set of memories.
This project is an alternative to the omnipresent, intangible and semantically-based cloud database. We are investigating a format to represent, and an interface to browse dense databases inspired by the “method of loci”, a mnemonic technique that relies on human capacity to quickly and efficiently store new information on an imaginary 3d space (or “memory palace”). This is achieved by superimposing “virtual 3d data containers” (or “knowledge blocks” – hence “knoxels”) onto real space, leveraging our species’ powerful spatial memory. MBs can be opened, closed, sliced and arranged with respect to the others (very much like Minecraft). More concretely, each MB is a cubic volume (about 50x50x50 cm3 in our first prototype) onto which multimedia data can be spatially organised using natural gestures. These blocks/bricks can then be stacked together to fill larger spaces (homes – perhaps whole cities). The finesse of the classification follows naturally the overall 3d structure of the system, as it’is the case in a real bookshelf. Inside each block we find knoxels (that is, knowledge-voxels), arranged in a personal manner – a 3d sculpture representing clusters of data.
This virtual bookshelf can be completely contained in virtual space and function as a personal (walkable) library. However, our more ambitious goal is to make this “memory palace” a public place, that is: a shared, infinite whiteboard or 3d bookshelf. Users will be able to annotate, navigate and retrieve information by relying on their own personal frame of reference (as already demonstrated in our Volume Slicing Display). They will be able to add knoxels in the form of notes and hyperlinks (manually or semi-automatically). The shape and arrangement of knoxels may represent some aspects of the data itself, or represent relevant links between these – based on content or social network activity (see A-me and BrainCloud: Art-Science Interrogations of Localization in Neuroscience).
We implemented various instantiations of this volumetric virtual shelf; the simplest is technically similar to the Volume Slicing Display, but has a different form of interaction (knoxel slicing prompts songs or videos instead of images). Another experiment concentrated on the office space, and relied on AR goggles. An MB over an office desk is interesting because the same physical space can be used by different people without loosing each personal configuration (this is the equivalent of logging-in and out in a personal computer, changing and retrieving each familiar desktop space). Moreover, and since an MBs is portable, your individual desk can be deployed everywhere (we used a fiducial printed on a card to “invoke” the desk space in AR). In our preliminary experiment (below), each knoxel was represented in AR as a floating semi-transparent coloured blob – an easter egg containing visible images and text. As explained above, MBs can be arranged and composed to fill larger spaces, always in a meaningful way (for instance, we can have a MB filled with recipes in the kitchen). The whole system forms a fractal structure that can be navigated using spatial memory (something we are used to in the real world).
For more
- A-me and BrainCloud: Art-Science Interrogations of Localization in Neuroscience, Jordi Puig, Annamaria Carusi, Alvaro Cassinelli, Philippe Pinel, and Aud Sissel Hoel, Leonardo MIT Press, 51:2, 111-117, (2018).
- Puig J., Perkis A., Pinel P., Cassinelli A., Masatoshi I., The neuroscience social network project, SIGGRAPH ASIA 2013 (poster), 19-22 Nov. 2013, Hong Kong. (2013) [PDF-1MB]
- Puig J., Perkis A., Hoel A.S., Cassinelli A., A-me: Augmented Memories, SIGGRAPH ASIA 2013, (art paper), 19-22 Nov. 2013, Hong Kong. (2013) [PDF-35MB]
- JSPS Kiban Research Grant (2012 to 2014): Memory Blocks & Knowledge Voxels [PDF-1MB]