Anatomy behind the Shadow DOM {Chapter 1}

Anatomy behind the Shadow DOM {Chapter 1}

Reading Time: 8 minutes

Since the birth of Selenium Webdriver {now W3C Webdriver}, the first time a global survey has been conducted across the W3C WebDriver community and all of the participants have shared their key challenges faced during their automation. Let’s look at the responses of one of the most important challenges asked “What one thing would you improve about W3C Webdriver?”

It’s crystal clear that the Shadow DOM is one of the key challenges for all of the Test Engineers across the community. On the other hand, the rise of Shadow DOM has become popular among the developers’ community and has been used within almost every software development organization. To spread the knowledge about Shadow DOM, I have also presented it at the virtual Selenium International Conference 2020. And I know, the conference session has been missed by many of the test engineers. But there is good news for those who have missed it. The Selenium International Conference session video recording has been released on youtube and moreover, I will be covering every nitty-gritty of Shadow DOM in this blog series. It’s time to ride on Shadow DOM — so buckle up and let’s break together with the Shadow DOM 😊😊

Before cracking the Shadow DOM, let’s put some light on its BIG BROTHER i.e. main DOM.

Peek at DOM {Document Object Model}

Whenever a web application is rendered on a web browser, then it always encapsulates three things — Markup {HTML}, Styling {CSS} and JavaScript {JS}. And after loading the web page on the browser, it creates a Document Object Model of that page. The DOM model is constructed as a tree of Objects. Let’s understand the code snippet that demonstrates HTML to DOM structure:

// Browser rendered Web Page HTML
<html>
 <head>
 <title>My Title</title>
 </head>
 <body>
 <h1>My Header</h1>
 <a> href=”My Link”</a>
 </body>
</html>

[ DOM tree structure of above HTML ]

In technical terms, The DOM is an Object Model for HTML that defines — HTML elements as objects, properties for all HTML elements, methods for all HTML elements, and events for all HTML elements. On the other hand, The DOM is an API for JavaScript that allows JS to add/change/remove HTML elements/attributes/events as well as CSS styles.

📍📍DOM elements are accessible to the document interface — ‘an entry point to DOM’ and DOM elements find by using document.querySelector method.

Nitty gritty of Shadow DOM

In reference to objects and structures, there is no difference between DOM and Shadow DOM. But Shadow DOM has a few special properties that differentiate it from the main DOM. It is a kind of web standard that developers use to encapsulate their custom HTML code and their style components. Because of the encapsulation, Shadow DOM elements are NOT visible to document.querySelector method because it renders separately from the DOM.

Shadow DOM is also created by the browser itself, but in a completely isolated DOM tree called the Shadow DOM Tree. The Shadow DOM Tree encapsulates its own custom elements and styling that is appended to an element as a child. Line of control, where the shadow DOM ends and the main DOM begins, we call it Shadow Boundary. The element to which the tree is appended is called the Shadow Host and the root is called Shadow Root. In the picture below, we can clearly understand the Shadow DOM structure in respect of DOM:

If it renders separately then What is the solution to reach out to Shadow DOM custom elements?

📍📍We can reach out to Shadow DOM elements through shadowRoot interface i.e. by using shadowRoot.querySelector method. The Shadow Root has two different modes

📍📍Open mode allows the shadow root to be accessible from JavaScript outside the shadow root but Closed mode does not.

So far we have briefed Shadow DOM with their custom elements, what if some standard elements have their own Shadow DOM and which is not visible through default dev-tools settings? Browser itself create Shadow DOM for few of the standard elements e.g. <video> and <select> tags. You cannot access a Shadow DOM created by the browser to display standard elements, which is called a #shadow-root (user-agent) in the Dev Tools. The #shadow-root (user-agent) is a browser vendor’s native implementation, so they are not documented and will never be accessible outside the Shadow DOM.

📍📍 This #shadow-root (user-agent) is not visible with the default properties of Chrome dev-tools. So to reveal this, we have to select the ‘user-agent Shadow DOM’ checkbox under dev-tools settings. Then only, a user would be able to see #shadow-root (user-agent). Let’s take a <video></video> tag in your HTML and try to access it through default settings and then by selecting ‘user-agent Shadow DOM’ checkbox under dev-tools settings.

Why is Shadow DOM rising in the developer community?

Reason #1 — Though the Web-Component-driven architecture becoming more popular among the developers, there is still one hitch — accidental intrusions i.e. globally applied CSS could create problems across the rest of our web application code. So any workaround? Yes, the simple solution is Encapsulation. Encapsulation will surely help us prevent any outside side intrusion whilst keeping our component code clean, minimal, and easy to extend.

Reason #2 — Old school days, <iFrame> was quite handy among the developers for the sake of encapsulation. But there are few serious security setbacks i.e. malicious plugins can be injected through <iFrame>.

So these are the major reasons behind the rising of Shadow DOM ✌️✌️

Let’s see how Shadow DOM renders on the browser and also see its Shadow tree structure with the following JavaScript code:

For a deep understanding of Shadow DOM, we also need to go through the concept of Web Components and Custom HTML elements.

Peek at Web Components

Webcomponent is a W3C standard feature that is being incorporated in HTML5, Google’s Polymer, and LitElement libraries. Web components are a set of APIs that facilitate the creation of new custom, reusable HTML elements that can be used in web pages and web apps with their functionality and also isolates them from the rest of your application code. Its characteristics:

- Reusable: Web Components are the same as other HTML tags, they can be reused n number of times without conflicting with each other’s functionalities. In the picture below, we can clearly see the same Header/Footer web components reused at two different places.
- Encapsulation: Web Components wrap their business logic, its styling within the component. By attaching Shadow DOM to that web component we can completely encapsulate its styling and business logic.
- All encapsulated web components can interact with each other but can not access each other’s properties. But still, global Inheritable properties are easily applicable on all web components e.g. color is an inheritable global property that always impacts all of the web component elements but we can override it within the web component by writing inline HTML code {‘background-color’ is also global property but NOT inheritable to encapsulated web components}.
- One component can integrate another component e.g. Header inside Blog.

Web components have four building blocks:

- Custom HTML Elements
- Shadow DOM
- HTML Templates
- HTML Imports

But in this article, we touch base on only two {Custom HTML Elements and Shadow DOM}, which are required to understand our test automation problem.

Peek at Custom HTML Elements

By working on HTML5, we can create our own custom elements [with or without appending Shadow DOM], which would be encapsulated by web components. Let’s understand and create custom elements:

- customElements — are HTML elements with custom templates, behaviors, and tag names made with a set of JavaScript APIs (e.g. <my — custom — element>)
- To define a custom element, create a class that says ‘MyCustomElement’ that extends HTMLElement class and passes the class to the customElements.define the method.
- HTMLElement class on HTML — is used for defining a custom element and teaching the browser about a new tag.
- customElements.define method — To register a custom element on the page.

// Code snippet Without attaching the Shadow DOM to custom HTML elements
<script>
    	class MyCustomElement extends HTMLElement {
      	connectedCallback() {
        		const button = document.createElement("button");
        		button.onclick = () => alert("Button is clicked");
        		button.innerText = "Button inside the Custom Element";
        		this.append(button);
      		}
    	}
    	customElements.define("my-custom-element", MyCustomElement);
  </script>
  	<div><my-custom-element></my-custom-element></div>

Let’s open this JavaScript code snippet in the browser and verify it under the dev tools console, it is clearly showing the custom HTML element and that element contains inside another standard web element i.e. button:

<my-custom-element>
   <button>Button inside the Custom Element</button>
</my-custom-element>

Now let’s attach Shadow DOM to a custom HTML element:

// Code snippet after attaching the Shadow DOM to above custom HTML element
<script>
    	class MyCustomElement extends HTMLElement {
      	connectedCallback() {
        		const shadow = this.attachShadow({ mode: "open" });
        		const button = document.createElement("button");
        		button.onclick = () => alert("Button is clicked");
        		button.innerText = "Button Inside Shadow DOM";
        		shadow.append(button);
      		}
    	}
    	customElements.define("my-custom-element", MyCustomElement);
 </script>
 <div><my-custom-element></my-custom-element></div>

Let’s open this code snippet in the browser and verify it under the dev-tools console, it clearly showing the custom HTML element appended with Shadow Root:

<my-custom-element>
	#shadow-root (open)
   		<button>Button Inside Shadow DOM</button>
</my-custom-element>

Are all browsers supporting Shadow DOM?

The majority of the latest browsers (Chrome, Firefox, Edge, Safari, and mobile browsers) support the latest version of Shadow DOM i.e. Version 1. Have a quick look:

Key Takeaways

- Did grasp about the DOM, Web components, and Custom Elements
- Recognized the difference between DOM and Shadow DOM
- Decipher Shadow DOM structure and its two modes
- Cracked User-Agent Shadow DOM
- Understood reasons behind the popularity of Shadow DOM
- Shadow DOM attachment with Custom elements
- Decoded DOM and Shadow DOM entry points
- Shadow DOM browser support

In chapter #1 — we have covered the nitty-gritty of Shadow DOM in the context of development. Still, Test Engineers are unaware — how it will impact their automation. It has raised so many questions for every Test Engineer:

- Is it possible to interact with custom elements WITH or WITHOUT appending Shadow DOM?
- Role of Open/Closed mode of Shadow Root?
- Can we handle User-Agent Shadow DOM in automation?
- Can we use regular selectors {Xpath, CSS} to find Shadow DOM elements?
- Is there a way to interact with multi-level Shadow Root elements?

Half of the battle is already won so let’s move to Chapter 2 {W3C Webdriver conquering automation of Shadow DOM}, which is already rolled out to clear all above open questions!! Keep winning, Keep learning, and spread automation 👍👍👍👍

December 24, 2015

Anatomy behind the Shadow DOM {Chapter 1}