Mobile-First Indexing or a Whole New Google? How Media & PWAs Fit Into the Larger Picture at Google – Article 2 of 4February 2, 2018
By: Cindy Krum
In the past couple of years, the internet has seen massive growth in the streaming and downloading of high-quality digital media. This is especially true for legitimately licensed (not pirated), legal, on-demand content including music, podcasts, audiobooks, video, animation and games. There is a lot of money to be made in this part of the Internet, and the potential is only growing, as people continue to abandon static media and cut cables and transition to online subscription services and on-demand options instead. According to a study by SensorTower last year, the U.S. consumer spending in the top 10 subscription video-on-demand apps (SVOD) surged 77% y/y to $781 million on iOS and Android. The final quarter of the year was even stronger with an 88% jump to $242 million for the same group.
With that, Google has a significant vested interest in securing its position as a digital media provider during this transition. The new Internet of Thing (IoT) and Connected Home technologies increase the stakes and the potential gains for Google immensely. Google Home and Google Assistant are already used heavily for media consumption. Often, they’re combined with Chromecast to make the mobile device the most capable TV remote available on the market. Now with Google Assistant, the phone becomes the TV remote; one that can even leverage voice commands, much like less-capable but competitive remotes from the cable companies.
The phone, of course, can do much more than cable remotes. Lots of big brands are already jumping on the Google Assistant bandwagon. Dish and Qualcom technology are both getting Google Assistant (Dish already works with Alexa) and you will soon also be able to get Sony Headphones and Bose noise canceling headphones with Google Assistant built in as well. There are also rumors that Google has begun a renewed round of efforts with GoogleGlass, which would presumably function using Google Assistant.
With all the consumer demand for media on so many different devices, it will be important for Google to help users find the exact content that they are looking for, or find something new and get it to the right device quickly and easily. It will be important for users to be able to do it seamlessly, even if they are transferring between devices with different levels of connectivity, processing power or no visual interface – even if they are in the middle of a streaming session. Consumer demand for media across different devices is creating unique technological challenges, and Google needs to be proactive about establishing a framework in the search engine for these evolving demands.
This article series dives into how Mobile-First Indexing, Google Actions, and PWAs may all be part of Google’s framework. The previous article in this series talked about Mobile-First Indexing and how it was probably related to Google Home and Google Assistant. This article will take that further, and put Mobile-First Indexing into a media-search and consumption context. It will talk about the pressures in the marketplace that Google is trying to address and how they are trying to address them using PWAs and other similar technology, even when in a ‘voice-only’ or ‘eyes-free’ context. The next and article in this series will speculate about Mobile-First Indexing in a e-commerce, shopping and payment context. The final article in the series will give information about how Mobile-First Indexing may impact local and international searches in maps and Google at large.
All in all, my belief is that Google’s shift to the Mobile-First Index is meant to lay the groundwork for many more significant and strategic changes in the future. Most of these changes are already in the works and they will be based on leveraging the Mobile-First Index, AI and the Internet of Things (IoT). The Mobile-First Index and the search technology that uses it is just what Google needs to create a constant stream of data to feed their many new AI-driven technologies that are all currently being developed.
More Details about the Fundamentals of Mobile-First Indexing
-Mobile-First Index Basics
One of the most misunderstood points about the Mobile-First Index is that Google intends it to be the primary index – for both mobile and desktop searches. It is called ‘mobile-first’ because the crawler is evaluating the content as a mobile phone rather than a desktop computer, so content that would not be linked or visible for a mobile phone might not get included in this index. The idea is that Google wants us to consider the content in a ‘mobile-first’ setting, but not necessarily a ‘mobile-only’ setting.
In terms of algorithms and ranking, Google has confirmed, at least in a pagespeed context that results will be device specific – faster pages will rank better on mobile, but the speed guidelines will be less strict on desktop – as illustrated in the Twitter conversation with John Mueller from Google shown to the right. This lends itself well to the theories presented in the previous article in the series, which suggested that while the index may stay the same across the many different devices and search options, the algorithms will likely adapt to the device, context and personalized needs of the searcher.
-The Importance of Collaborative Use & AI in the Mobile-First Index
Google seems to be banking on the idea that users will use their devices in a collaborative way, as a singular, personal unit, and as a larger societal unit, providing feedback to the search engine in order to refine the information they receive. There are hints that in a voice-only search context, websites might not be present at all, replaced only by rich answers, Featured Snippets, and other Mobile-First, ‘eyes-free’ style content, and saving the web interaction for later in the Google Home or the Google Assistant app. This is how Danny Sullivan suggested a user might get more information from a Featured Snippet that was conveyed by voice, but still accessible later, archived in the app. It may also be that in some cases Google will try to review web results in ‘eyes-free’ search experiences. This could be the incentive for the increasing the length of HTML description tags in search results that Dr. Pete observed in his research. In edge cases where a voice search needs to surface a web result without a screen, Google could read the title tag and description tag to the user, and then just save the URL in the app for later viewing, or potentially cast it to a screen, to be viewed immediately.
The more devices that are enabled with Google Assistant, the better coverage and penetration Google will eventually have with the new Mobile-First Index. High-dollar purchases like cars, TV’s and stereo systems that are enabled with Google Assistant from the beginning, lay the foundation for the Google Assistant software to be available, whenever the user is ready to integrate or interact with the technology. This is even true if the Google Assistant capabilities that are built into these many devices are not leveraged right away, but instead lay dormant until Google is really ready to start pushing for more engagement or until the user chooses to engage on their own. At that point, Google can always remotely update the hardware with the latest software, and encourage engagement through marketing on all the other connected channels. The fact that these kinds of devices are generally kept for longer than a typical mobile phone makes the integration that much more valuable for Google.
What are PWAs & How do they Solve the Deep Linking & App Indexing Problem for Mobile-First Indexing?
Before we dive deep into media SEO, it is important to get a basic understanding of PWAs. Put simply, a PWA is a website that looks and behaves like an app. It can do this primarily because of two files that are hosted on the domain with the website code:
- App Manifest: An ‘App Manifest’ holds the app icon, start screen and basic app-shell styling information
- ServiceWorker: The ‘ServiceWorker’ is more complicated than the App Manifest. Basically, it helps the website look like an app by saving local copies of critical assets, to make the website experience feel faster and more seamless on the phone – in other words, more like an app. With a ServiceWorker, once important content has been loaded on the phone the first time, it never has to be loaded again. It can even allow the app to work offline, and then when the web connection is restored, systems sync-up all the assets in the background, to make sure everything on the server and the user-facing side of the app are both up-to date.
To get a better understanding of how PWAs fit into Google’s larger plans, you have to understand that one of the biggest struggles Google has faced has been crawling and indexing native app content. It has been tough with Android apps and nearly impossible with iOS apps because they are built in different code than Android. To crawl and index native apps, Google has always had to use APIs to help translate and verify the content, but it has never been an effective or efficient solution.
Instead of actually understanding the app content, Google has always ended up relying heavily on associating app content with web content, which can be more easily understood. This association process made it easier for Google to understand what was going on in the native app so that it could be indexed, but it also meant that anything in the native app also needed to exist on the web to be indexed by Google. It also meant that native app content had to be actively and intentionally mapped back to web content, which sometimes proved quite difficult.
In large companies, with large websites and apps, the iOS app team, Android app team and web teams often work as independent bodies, with minimal overlap. Not only do they not know how the other systems work, they often didn’t even know what content was available in the other systems or where it would be found. Most large websites – especially those whose websites operate using a different CMS than their apps, still have not been able to achieve full deep linking or app indexing, even after significant effort. PWAs diminish the need for deep linking and app indexing in the short term, and potentially eliminate the need for many native apps in the long run, hopefully making the deep level of complexity that deep linking caused some companies simply go away.
Just like most software can now be converted to web-ware, most native apps can now be converted to PWAs. This makes the creation, maintenance and distribution of ‘app-like’ content much cheaper for everyone. Once a website has an app manifest and a ServiceWorker, it is technically a PWA. From there, it is a matter of adding app-like behaviors and designs to the website and this can be done in small changes as things are updated over time.
Android Instant Apps (Previously Google Streaming Apps) have not gotten much attention, but were an interesting attempt at allowing app indexing for app only content without requiring web parity. These were launched at about the same time as something Google called Dynamic Links (also at about the same time that Google launched Google Now-on-Tap, which draws in a potential connection with AI, entity understanding and the contextual awareness of the phone – as discussed in the last article.) Basically, Android Instant Apps were originally designed to be a way to have an app be indexed without a website.
Similarly, Google Dynamic links were a way to link to and share the app content without having to build a website. They were essentially temporary short-link that included a link-disambiguation engine. If an app had iOS and Android versions, those could both be stored in the Dynamic Link and shared. Since the Instant App lived in ‘chunks’ on the web, the Dynamic link would link people to the deep content in the native app and people without the app would get a web-view that showed just the small portion of the native app that was deep-linked to. The idea was to achieve cross-device sharing without requiring web-parity, but it never really caught on. The process of making a native Android app ready to be ‘chunked’ into Action-oriented Feature Modules is still very hard, and still required labor-intensive mapping between iOS and Android apps, even though the iOS app can never be updated into an Instant App.
What is interesting though, is that Google now appears to be using Dynamic Links all over the place in their content – especially with things like Knowledge Graph content that have no URL at all. It is almost like the reverse process of creating an Android Instant App, and it seems to be much easier. It appears that Google is simply ingesting databases or data sets that are marked up with JSON-LD – they published the instructions on how to do this about 8 months ago – and allowing these to index on their own, with no URLs and using styling instructions that come directly from the browser. At least, it appears possible with Google-owned assets like Knowledge Graph and similar database style assets.
With this concept of the technology, when a database is properly marked up and modified it becomes an Instant Apps, not only because of how fast the app experience is without extraneous code, but because with minimal additional code or development, the database itself becomes the app, nearly instantly.
NOTE: This description of how “Instant Apps” might be theoretically created in the future sound very similar to Google’s theoretical description of PWAs. According to Google, “PWAs are websites that progressively become apps.”
This kind of indexing is especially great for media databases, because there is no crawling involved. That makes it very fast, and items are surfaced in a search based on their relationships to other items, which is exactly how most media, like music, TV, movies, books and art are consumed. As long as the media can be easily surfaced and played, that is what matters in Mobile-First indexing. As discussed in the first article of this series, it seems that URLs are not required for the Mobile-First Index because it leans heavily on databases for content. In fact, URLs may be one of the least efficient ways that Google finds and indexes content. If a user wants to share this non-URL’s content, they can click on the triangular ‘share’ icon to create a Dynamic Link for it. Then, the asset is uploaded to a server and a temporary link is created and copied to the clipboard. It is clear that this is happening on the fly, because it can take time, if a lot of content is included in the ‘share.’
While it is not as fast or exciting, Google has been encouraging large media sites to mark up their content with JSON-LD ‘watch’ and ‘play’ action schema for more than a year. This markup was originally designed to be included in the body of the HTML or in the head tag of the HTML. However, it seems more and more likely that Google will begin suggesting developers build external JSON-LD files like that are linked from the head tag in the future. Currently this option is in ‘Partner Only’ mode for websites, so you have to be approved, and then will be given more comprehensive instructions, but for native Android Apps to leverage this same option, they use the Voice Interaction API. This separation of files reinforces the idea that Mobile-First Indexing is more about avoiding burdensome crawling in favor of ingesting content whenever they can.
What Makes PWAs & Instant Apps Great for Cross-Device Media?
PWA’s create a more seamless experience across the different devices by ensuring simple, web-based transitions, that don’t require software, app or OS-specific programming. PWA’s use ServiceWorkers to help modulate what data is stored to the phone, and what data is stored to the server. That functionality is also what makes them so great for media.
PWA’s also are able to store app-states in the cloud, including the location of ‘stop’ and ‘start’ in a media file, so that the user can pick up where they left off in an PWA-media interaction, even if it is on a different device. Google Home and Google Assistant have continued to build out the software with new voice command capabilities that you can use to control media. For instance, now you can use Google Home to start and stop shows in Netflix, query TV schedules, set reminders for TV shows or associate specific artists, songs or playlists with alarms, so these kinds of integrations may be important as the Google Assistant becomes more able to interact with media from websites and especially PWAs..
PWA’s require less data transfer and less storage space, and are built to be device independent, so they are great for devices that were not built with lots of storage and data transfer in mind, like web-enabled TV’s, Android Auto devices, ChromeBooks, watches and lots of other devices. Since PWAs are hosted on the web and require Responsive Design, any device with a browser or OS that supports ServiceWorkers will do. With that minimum requirement met, PWAs can work on the largest group of devices possible, regardless of OS, including low-cost, web-enabled dummy-terminals created by Google and other manufacturers as well. Let’s dig in to those devices here:
Windows & Other Desktop OS: Since PWA’s are just websites with extra capabilities, all desktop browsers can display them. Many desktop browsers are starting to support more features of PWA’s – especially push notifications, but some may also begin including some support of ServiceWorkers. Remember, the Windows App Store was the first store to list PWAs alongside regular OS apps, and they use the same app store for desktop and mobile. The integration of PWA functionality on desktop as well as mobile is an important step forward to making PWAs the standard of development, to make them more efficient and effective than device-specific options like native apps or non-responsive websites. Once all of the major tech players are able to effectively support PWAs in mobile and desktop and tablet scenarios, it will be easier and more likely that they will begin pushing them for use on less traditional devices like web-connected car systems, game systems and especially TV’s, where web display and interactivity has historically been problematic and unwieldy.
Apple Desktop & Mobile OS: Apple was a long hold-out on PWA’s. Most assumed that Apple’s previous resistance supporting PWA’s was likely due to their desire to maintain mobile market-share for their products and services in the App Store, and protect against Google’s encroachment, since Google has always thrived on the web and Apple has always done best in their ‘walled garden’. Now Apple has announced support ServiceWorkers in iOS 11.3 and macOS 10.13.4 (although top names in the space like Maximiliano Firtman have already some technical issues with PWA’s on iOS that still need to be ironed out.)
Apple also recently made a surprisingly bold move by suggesting that PWAs were a better option for developers than using app templating software to build low-quality native apps. Since Apple’s revenue has always focused more on the iTunes and the AppStore, their sudden support of PWAs likely means that Apple has figured out how they can monetize PWA’s. It also likely indicates that PWA’s will be accessed through iTunes, the AppStore or through some aspect of AppleSearch (Siri, Safari, Spotlight). Recent announcements even indicate that Apple could let you run iOS apps on your Mac soon – Further indications of convergence across multiple ecosystems, so this change seems likely to further broaden the appeal of PWAs as well.
Low-Cost, Web-Enabled Smart Displays: The last important concept here is Google Smart Displays, which allows Google Assistant and Google Home to cast video content. All different types of screens including TVs and computer screens but it also seems to include Android Auto screens and ‘Smart Displays’ designed exclusively for this purpose. Different from TV’s that might be associated with a variety of other components, dummy screens might only be used for receiving casted information from other devices. It seems likely that these ‘Smart Displays’ will actually be more like dummy terminals that have minimal processing power and which lean heavily on Google Assistant and the cloud for most or nearly all of their functionality. This might be an interesting middle ground where even though there is a visual interface, there is no capability to download apps or even PWAs, and the screens are more like projectors that can never be altered or personalized.
How do PWAs Fit with Google Actions, Media & Mobile-First Indexing?
As described in the previous article in this series, the Google Assistant and the Google Now utility currently pull information from a variety of sources that include News Feeds, Knowledge Graph, Google Actions and website results. Until a search is submitted, the screen focuses on information cards that it determines you might find useful based on your established preferences and habits. This includes information about the weather in your location, sports scores, news and entertainment information. When a voice-oriented search is submitted, search results focus mostly on similar results that include Google-owned content like news, weather and sports scores, but also might include Featured Rich Snippets, Answer Boxes or a Knowledge Graph. When a more traditional query is submitted, it is more likely to generate a list of website results with blue links. The Schema markup that Google has been pushing websites to add is equally apparent here, but not more-so than it might be in a regular desktop or mobile search result.
Google has begun turning many of the stock Android apps into PWA’s. This includes their Contacts, Photos, News, Maps, YouTube. This gives a couple good examples of what PWAs can do, but it also give Android users the added benefit of allowing account-specific information to sync to the cloud across multiple devices. There are also some more niche, non-stock Google apps like Google+, Google My Business and The AMP Playground that are also available as PWAs, probably for similar reasons.
We can probably expect Google to continue spinning off entity-specific parts of the Knowledge Graph into their own PWAs as the markup is finalized and the assets become available.
There are also some PWAs with app manifests, but no unique web urls to link to, including: Weather, Sports, Dining, Drive, GooglePlay TV and Traffic. The process for adding the Google Dining App/PWA to an Android phone is illustrated below. It is possible that these are examples of Google Instant Apps or something else that Google has yet to document for public consumption. The idea that it is a PWA is based on how the app download is triggered, and how quickly the download is complete (adding only an app manifest and ServiceWorker rather than an entire app takes only about 1 full second).There are also some Google search results that have special formatting and PWA-like qualities in the search results, but for which there are no URLs or app manifests for a PWA including: Events, Calculator, Knowledge Graph. These apps may leverage ServiceWorkers and app manifests that are part of the Chrome app itself to enable this behavior without a separate download.
The voice-only capability of a Google Action is a bar that most apps and websites can’t manage yet, but this could quickly change for PWAs, making them a great potential option for Mobile-First Indexing. Google has already announced that they will begin repurposing existing web content – especially recipes, podcasts and articles into Actions for Google Assistant. (ht/t to Aaron Bradley (@aaranged) for calling this out on Google+ and Twitter). ServiceWorkers force PWAs do a very good job of separating text content from the rest of the code for caching, so to generate a voice-only interaction Google might be able to read the text of a PWA directly from the ServiceWorker cache. This might even be made easier with the announcement that Chrome will be automatically lazy-loading images, preventing them from slowing down access to text content that could be conveyed through voice, when images might slow down the transfer. In the long run, the same may also be true of AMP’d content and Instant Apps but this is a bit harder to envision.
PWAs & Instant Apps in Google Play
Google has already indicated that they will include PWA’s and Android Instant Apps in the Google Play Store, so this means that in both web and app search results, they will be expected to compete side by side. While this is not an overt merger (yet), it is a clear change of direction to allow web content to cross into their app store, where the web was previously forbidden. PWA’s also use ServiceWorkers to manage caching of the most important content on the site, and this is likely how Google will crawl and index this content – saving considerable time over traditional crawling. In this, PWA’s have another significant advantage over both native apps and websites, in that they might be much faster and easier for Google to crawl and index, because it is all done with the ServiceWorker API.
Google Play’s recent focus on evaluating and ranking apps based on technical factors like crash rate and errors, as well as the recent massive purge of 700,000 low quality apps from the store is probably an important signal in the preparation for changes here. Adding PWAs and Instant Apps to Google Play is a signal, not only that Google Play is capable of listing web content in the store, but that it is capable of comparing the web content with app content, and determining an appropriate ranking of all the relevant options. It is an indication that anyone in the App Search Optimization space (ASO) should expect some significant changes soon, but it also is likely an indication that Google is prepared to monetize all of these different offerings simply, through a unified system.
The Mixing of Google’s Media Assets
In terms of media assets, Google owns YouTube, which is the largest media search engine in the world, and the second largest search engine in the world in terms of volume. It also has Google Play which offers TV, movies, books, audiobooks, games, music, podcasts and news. Both have been coming out with a number of new subscription options and announcements, which makes it seem like this type of asset is explicitly intended to be part of the long term picture in terms of Mobile-First Indexing and Google’s movement into IoT, along with casting, playlists and lists of favorites. It seems that both might soon be capable of being used as DVR and set-top-box replacements. You can see in the progression below, that Google is really ready to monetize digital media, so Mobile-First Indexing will be the critical part to surfacing it quickly and easily.
It seems like there may be some major consolidation brewing between YouTube and Google Play. Google Play has historically been targeted at Android users, but as described above, it can now take web content in the form of PWA’s. Potentially, this is true for the entire store, and not just the ‘app’ aspect of the store. YouTube, on the other hand, has always been more web-focused, reaching both Android and iOS users, but is now encouraging people into a subscription service – which was something previously only seen in Google Play. Consider the merger of YouTube Red and Google Play Music – YouTube Red is Google’s Video subscription service for cable-cutters and Google Play Music is where Google’s streaming and music library solution lives.The real benefit that Google has over its main competition (Apple, Bing and Amazon) is search. It is what they have always been good at, but as any seasoned developer knows, legacy can be a mixed blessing. The motivation to create a new Mobile-First Index may have come from their desire to leverage their biggest advantage over their biggest rivals – Apple and Amazon have their own indexes, which are focused heavily on media, but in media, but Google has the rest of the internet, and that will be quite powerful. Google also needs to compete with companies like Netflix and Hulu, which have their own subscriber bases and repositories of content, but need partnerships because they don’t have the voice-oriented or sophisticated search technology.
As the two platforms become nearly inter-operable, it becomes more likely that they will consolidate to build the power of the brand and reduce the friction in cross-device interactions. There is also suspicious duplication and cross-linking of the Google Play and Google Play Movies & TV app, as shown on the right. The Google Play Movies & TV app appears to use some Instant App or PWA capabilities though it is not clear which – it seems that the transition may be only partially complete. Regardless, consolidation of media assets is an important way that Google will protect and grow its revenue and market share, to pull customers away from strong competitors like Netflix, Hulu, Pandora and Amazon Video/Music.
For years, Google Play video purchase records have been cross-populated directly with YouTube. Before the MyAccounts disambiguation system was put in place YouTube seemed to host Google’s account management arbitration system, and this is important because YouTube was not just focused on the Android audience like Google Play was – it has always been one of the stock apps on iOS devices. This helped get iOS users into Google’s account funnel (if they didn’t already have a Gmail account or some other Google account set up.)
YouTube and Google Play also both seem to be enabling future in-roads for further social connections in the platforms. While rating, commenting and sharing have always been part of both systems, they both now allow Android users to search their Google contacts to find friends and to share content to other users, in any other apps on the phone, using Dynamic Links. It is possible that once YouTube and Google Play are merged into one, they could also bring Google+ into the fold, in another (possibly ill-fated attempt to launch a viable social network). This probably wouldn’t be successful, but as the level of media that is shared in social networks continues to increase, it remains firmly in the realm of possibilities.
It feels like we are in the midst of a land-grab for voice-oriented technologies that allow consumers to interact directly with the internet. This is an exciting time to be in search – but potentially scary if you are not ready for change. Many SEO’s have settled into the idea that their job will always be about optimizing websites to be found based on the keyword optimization of the content on a website, with URLs and links. While those strategies may work for a bit longer, it seems like in the near future, those types of search results may be relegated to a very small portion of the total number of results that are served to users. As more and more devices go online, and users can search using only their voice, the types of results that they want will change dramatically. Queries will become more action-oriented and conversational. It could only be a small set of research style circumstances where a blue-link web result is an appropriate type of result.
The Mobile-First Index will be the index of all content, but I believe that different algorithms will rank the content differently, based on the queries, but also based on the user intent, circumstances, and history. Some devices may have a strong understanding of a users tastes and preferences and will be able to personalize recommendations based on entity relationships of content in their index. Databases and feeds of content – especially media content – will be preferred over HTML, because they can be easily ingested through API’s rather than crawled. The existing structure of a database, along with JSON-LD markup, can be used to build on entity understanding for the content, rather than creating it from scratch in the index. With the intense competition for users’ attention in the media space, this kind of capability and agility will be heavily in Google’s favor.
Mobile-First Indexing is a change that is laying the foundation for much greater changes at Google, and SEO’s who are not prepared for the change could be left in the dust. Where our old jobs focused on optimizing websites, HTML tags and content, our new jobs might focus much more heavily on optimizing JSON-LD markup in databases and feeds, and training AI. The first article in this series focused on the importance of Voice for Mobile-First Indexing, and this article focused how Mobile-First Indexing could change our interaction with media in the near future. The next article in this series will focus on e-commerce, and how how Mobile-First Indexing may change how we shop and pay for physical goods, and the final article in this series will focus on the impact that Mobile-First Indexing is expected to have on local and international SEO.
- More web-enabled devices are being produced, and many leverage voice controls rather than, or in addition to traditional screens, buttons or remote controls.
- With the demand for media across many devices, it’s important that Google find the best content or achieve the desired result quickly and easily, with the device out-put and user-intent in mind. This is the main motivator of Mobile-First Indexing and the integration Google Actions into their framework.
- Progressive Web Apps (PWAs) are websites that behave like an app. They appear to be key in bridging the feel and function of an app with the discoverability and easy maintenance of a website. PWAs are great for cross-device media experiences.
- Google has begun re-purposing existing web content (podcasts, recipes, and articles) into Google Actions, presumably for surfacing in the Mobile-First Index.
- Google also indicated that PWAs and Android Instant Apps will be permitted in the Google Play Store, allowing web content to cross into the app marketplace.