From digital to physical and back again

I get really annoyed when people start talking about the ‘real world’ in contrast to the ‘digital world’. We live an increasing amount of our lives connected to digital devices, using digital services, interacting with digital objects – in what sense is this not real? It also deflects attention away from the fact we are constantly criss-crossing the line between physical and digital. Whether it’s through geospatial applications, augmented reality, 3d printing, or guerilla projections, there are many fascinating opportunities for exploring this relationship within digital heritage. What interests me most of all are the possibilities of mobilising cultural heritage collections and taking them into the places, both physical and digital, where people already are.

That doesn’t mean, however, that we should just accept the hype that currently surrounds technologies such as virtual reality. As I’ve stressed throughout this unit, we have to critically appraise technologies – explore their limitations and assumptions as well as their possibilities. I have to admit to being a little perturbed, for example, when people start talking about VR as an ‘empathy engine’. What does this actually mean?

While I think there are some exciting possibilities ahead for 3D imaging, I’m not sure about AR and VR in the cultural heritage setting. Perhaps it’s because I feel like we’ve been there before. I remember playing around with VRML in the ’90s; when Second Life was going to be the next big thing for museums; and when everybody was going round sticking QR codes on things. Are these technologies different? Will they change the way we experience cultural heritage? I’d like to know what you think. Instead of trying to tie these things up in a neat bundle for you, I thought I’d just offer examples and raise questions. As you explore each of the examples I’d like you to think about what they might contribute to the way we experience and understand cultural heritage. Are they gimmicks or groundbreaking? The future or mere fad?

3D imaging

As I said, I think 3D imaging is becoming an established part of the digitisation toolkit. Basic photogrammetry – extracting spatial data from photographs – has moved from research projects to consumer products. Indeed, if you have a smartphone you can try it for yourself. Download Trnio for iOS or Scann3D for Android. They’re both pretty simple to use, and once you’ve created a 3D model you can publish it on SketchFab – the main site for sharing 3D content.

Here’s a 3D model I created at home with my kids using another (now defunct) photogrammetry app. I cleaned it up a bit using Meshmixer before uploading to SketchFab.

Terry by wragge on Sketchfab

Of course there are much sophisticated and accurate tools available for 3D imaging. Using technologies such as LIDAR it’s possible to create 3D models of complete heritage sites.

Cultural institutions are starting to make selected collection items available as 3D models:

Not only can you view the models online, you can download them and feed them into a 3D printer to make your own physical copies. This seems to me to have important educational uses, particularly in remote areas – what do you think? Does the ability to handle an item (even a 3D printed plastic version) add to our understanding of it?

Virtual tours

This is not really 3D and not augmented reality, but I thought it was worth mentioning here. As you may know, Google’s StreetView technology has moved inside some institutions, allowing people to take virtual tours. Here’s the National Museum of Australia.

The NMA also provides laser-guided virtual tours courtesy of two mobile telepresence robits.

I suppose one theme across all of the technologies we’re looking at today is the attempt to create new understandings or experiences of space that blend the physical and the digital. But how new is this? How are technologies such as augmented reality different to old-fashined audio guides? Is access a key point – the fact that these experiences can be delivered remotely?

3D worlds

As I noted above, 3D scanning technologies can be used to help build up models of complete sites. These can then be populated with reconstructed buildings, and even animated people.

The Visualising Angkor project is a particularly impressive example, where researchers have not only tried to reconstruct a site, they’ve tried to represent to rhythms of everyday life – the sun rises and sets, people and animals move about their lives. All of this is based on detailed archaelogical evidence. Unfortunately the computing power required to run the simulation means it can’t be accessed online, but you can find some videos on the Google Cultural Institute page.

How about a tour of ancient Rome? The Rome Reborn project has created a detailed 3D model of Rome in 320 CE.

Rome Reborn 2.2: A Tour of Ancient Rome in 320 CE from Bernard Frischer on Vimeo.

Distance in time is not the only reason to build 3D worlds. Sites can be inaccessible for a variety of reasons, and 3D models can provide a way of documenting and experiencing locations that might be at risk.

Researchers in Western Australia have created a 3D model of the wreck of the HMS Sydney. As you can see from this ABC report it offers a powerful emotional experience.

You’ve probably also heard of attempts to use 3D modelling to document and possibly even reconstruct parts of Palmyra destroyed by ISIS. Recently a replica of the Palmyra Arch of Triumph, created using 3D modelling techniques, was unveiled in London.

It’s also possible to create models of places that never actually existed. Researchers on the Digital Panopticon project have created a 3D model of Jeremy Bentham’s ideal prison, designed in the 18th century, but never built. This them to think about how the space might have worked, and what the experiences of prisoners might have been within such a space.

But not everyone thinks these types of projects are a good idea. Critics of the reconstructed Palmyra arch, for example, have pointed to the danger of creating a Disneyfied version of the past. Might such reconstructions obliterate memory rather than preserve it? How do we experience and document the processes of destruction and decay alongside such freshly-minted versions of the past? Are these reconstructions all a bit too clean?

AR & VR

Nowadays if you want to explain augmented reality to someone it’s easy – you just say it’s like Pokemon Go. AR is a blend of physical and digital, with new experiences, information and activities layered on top of our physical environment. There’s some definitions and a good dash of added hype in this article about ‘Augmented Museums’. Clearly some people can see the business opportunities.

Location-enabled applications that present historical photographs based on your location have been around for a number of years. A couple of years ago the Museum of London created an iPhone app called StreetMuseum that blended past and present.

Earlier this year the Art Gallery of NSW presented an immersive 3D experience that allowed visitors to explore intricate cave murals from the UNESCO World Heritage listed Mogao Grottoes in China. This work has been presented in a variety of forms around the world. It’s not just a 3D reconstruction, because various elements of the paintings come to life. Here’s a video showing it on display in HongKong.

Augmented reality apps can be used within a museum context to provide additional layers of information and interactity, as in this example.

If you view this video on YouTube you’ll see that the caption proclaims:

Visiting a museum is no longer a one-sided, passive experience, as we’ve developed an interactive exhibition tablet app using the newest innovative mobile technologies…

Does giving a visitor an iPad app make the experience less ‘passive’? Isn’t their experience still carefully curated? Clearly there’s value in giving visitors access to more information, but is this a good way to do it?

Virtual reality, on the other hand, immerses you in a totally manufactured digital environment. And you have to wear funny-looking goggles. All the big players are working on VR technologies – most notably FaceBook and Microsoft. (Let’s not mention Google Glasses).

Microsoft is building up to the consumer release of its HoloLens and there’s much hype about the ‘Minority Report’ style of interactivity it will enable. It’s interesting, however, that Microsoft has used the possibilities for cultural heritage as part of its pitch to potential developers:

One of the problems with VR is of ourse that you need special hardware for it to work. Videos don’t capture the experience. If you’d like to explore some of the possibilities and you’ve got a smartphone handy you can order Google Cardboard. It’s a simple VR viewer that uses your existing phone – for $30 or so you can start experimenting.

The Google Cardboard app provides some fun examples, including museum exhibits. You can also search the App Store or Google Play for ‘Google Cardboard’ to find other VR apps. Beware – the experience can be a bit unsettling!

You might we use this sort of technology? One of the nice features of SketchFab is that its 3D viewer is compatible with VR devices, including Google Cardboard. If you created a model with Trnio or Scann3D, you can easily upload to SketchFab and view using Google CardBoard. (Yes you’ll need to create a free SketchFab account.)

For further experimentation download Google’s Cardboard Camera app. You can use it to create 360 degree ‘globe’ images of any location that you can then explore in VR using Google Cardboard.

This is all a lot of fun. But how might you use this technology in a cultural heritage setting? Will we really want to spend our time immersed in a virtual museum? What sort of experiences do we want to create?

Perhaps the really exciting thing is not what cultural institutions will do with this technology, but what ordinary people will do with cultural collections using these sorts of tools.

Portfolio alert! Read through the sections above, watching the videos and exploring the examples. In a minimum of 200 words reflect on the possibilities and problems of these technologies. How do you think cultural heritage institutions will make use of AR and VR? What will these technologies add to our experience and understanding of cultural heritage collections? Are they the future, or just a fad?

Games

There is a big overlap between VR/AR and the world of gaming. Indeed, the game creation platform Unity is often used as the platform for building and exploring 3D worlds.

If you’re interested in the connection between cultural heritage and gaming, keep an eye on the Play the Past blog.

Of course, games don’t need to dependent on fancy graphics. There’s a series of posts on Play the Past that explore the use of Twine to create interactive text-based historical narratives.

One final video describes a project undertaken by the Auckland War Memorial Museum. Working with a local school they ‘recreated’ the Gallipoli campaign using Minecraft.

Build a bot

But there are other sorts of interventions we can make to challenge ways of thinking within the digital realm. Remember in the first module when you created your own ‘hacked’ versions of institutional websites? We can play with conventions, manipulate contexts, and expose alternatives.

One way of changing the conversation and putting new ideas into circulation is through Twitter bots. Twitter bots are just computer programs that interact with Twitter. Some are evil, such as spambots trying to capture your clicks, or fake news bots trying to manipulate public opinion. But others generate art, open collections, and expose political activities to scrutiny.

Some Twitter bots just post tweets, while others interact with Twitter users. Their behaviours can be quite complex and surprising.

TroveNewsBot tweets random articles from Trove’s digitised newspapers. But if you tweet keywords to it the bot will search for them in Trove and tweet back the most relevant result. You can search Trove without ever leaving Twitter! TroveNewsBot does a lot more as well.

  • For a random article, just tweet the hashtag ‘#luckydip’ to @TroveNewsBot.
  • What about all the resources in Trove that aren’t newspapers? Try @TroveBot!

But Twitter bots can also make a point. @StL_Manifest shares images of Jewish refugees who were turned away from the US in 1939 and became victims of the Holocaust. Every3Minutes helps us understand the scale of the slave trade in pre Civil War America. I created Operation Random Words in response to the Australian government’s implementation of Operation Sovereign Borders. For more on the political potential of Twitter bots read A protest bot is a bot so specific you can’t mistake it for bullshit by Mark Sample.

Your final mission is to create a Twitter bot. You’ll be using a site called Cheap Bots Done Quick that makes it easy to build bots with a minimum of fuss.

Your bot will combine words and phrases within a pre-designed template to generate random messages. Operation Random Words is an example of this approach. Its tweets generally follow the pattern:

Operation [ADJECTIVE] [NOUN]: [EITHER 'PROTECTING' OR 'DEFENDING'] Australia from [PLURAL NOUN]

To build your bot, you’ll need to design a template something like this, and provide sets of words to fill the slots.

  • First decide on a concept for your bot – what would you like it to do? This should help you come up with a name.

  • Once you have a name for your bot, create an account for it on Twitter. Note that you’ll need either a mobile phone number, or an email that’s not already associated with Twitter to verify the new account.

  • Once you’ve created your new account, go to Cheap Bots, Done Quick, and click on the Sign in with Twitter button.

  • Click to give the site permission to use your bot’s Twitter account.

Cheap Bots, Done Quick uses a Javascript library called Tracery to generate the random messages. Most of the work is done behind the scenes, but you have to provide Tracery with the data it needs in a format called JSON (Javascript Object Notation). JSON is used all over the place to move data around.

Fortunately, there’s a visual editor that makes it simple to generate data in the format Tracery expects.

  • Open the Tracery visual editor.

  • If it’s not already selected, choose ‘tinygrammar’ from the dropdown list.

  • Click on show colors and then reroll. You’ll see how the values on the left side of the editor are being combined to create the phrases on the right side.

  • The template for your tweets is always called ‘origin’. The placeholders in your template (where the randomly selected words will be slotted in) are indicated by a hash (#) at the beginning and end. Note that the default example has two placeholders ‘#name#’ and ‘#occupation#’.

  • Try reversing the positions of ‘#name#’ and ‘#occupation#’ in the template and click reroll. What happens?

  • You’ll see that the values to fill the placeholders are drawn from lists called ‘name’ and ‘occupation’. Try adding some extra values to the list of names and click reroll.

  • Click on new symbol to create a new list.

  • Click on the label of the list and change it to ‘mood’.

  • Click on ‘rule’ in the new list and change it to ‘happy’.

  • Click on the ‘+’ sign to add ‘sad’ to the list.

  • Now edit the origin template and add ‘is #mood#’ to the end. Click reroll. What happens?

At this point you should have a good idea of how the templates work. Now you need to design your own! Think about the types of things you want to combine. Then start creating lists of those things. Keep in mind that the complete messages need to be under 140 characters.

Once you created your templates and word lists in the visual editor, it’s time to copy the data across to your bot.

  • In the visual editor, click on the JSON button. Your template and lists will be displayed in JSON format.

  • Select and copy all of the JSON data.

  • Go back to Cheap Bots, Done Quick and paste your data in place of the default values.

  • A sample tweet should appear in the text box below the data. Click on the reload button a few times to check that it’s working the way you expect.

  • If all seems ok – fire off your first tweet! Just click on the Tweet button next to the sample tweet.

  • Check your bot’s twitter profile to see if the tweet was sent. (You might have to reload the page.)

  • All good? Now go back to Cheap Bots, Done Quick to set up a schedule.

  • In the dropdown box next to ‘post a tweet’ select ‘Every hour’.

  • Now click on the Save button at the bottom of the page.

Congratulations, you’ve created your first bot. Share the Twitter account through Slack so we can all enjoy it!

Portfolio alert! In your portfolio save a link to your bot’s Twitter account. In a minimum of 200 words, explain the concept behind your Twitter bot, then describe the Twitter bot you would build if you had the technical skill – what would it do and why?