Solving Data Wrangling for Dashboards

This publish is about Dashify, the Cisco Observability Platform’s dashboarding framework. We are going to explain how AppDynamics, and companions, use Dashify to construct customized product screens, after which we’re going to dive into particulars of the framework itself. We will describe its particular options that make it essentially the most highly effective and versatile dashboard framework within the business.

What are dashboards?

Dashboards are data-driven person interfaces which can be designed to be considered, edited, and even created by product customers. Product screens themselves are additionally constructed with dashboards. For this motive, an entire dashboard framework gives leverage for each the top customers trying to share dashboards with their groups, and the product-engineers of COP options like Cisco Cloud Observability.

In the observability house most dashboards are centered on charts and tables for rendering time collection information, for instance “average response time” or “errors per minute”. The picture under reveals the COP EBS Volumes Overview Dashboard, which is used to grasp the efficiency of Elastic Block Storage (EBS) on Amazon Web Services. The dashboard options interactive controls (dropdowns) which can be used to further-refine the state of affairs from all EBS volumes to, for instance unhealthy EBS volumes in US-WEST-1.

Several different dashboards are supplied by our Cisco Cloud Observability app for monitoring different AWS programs. Here are just some examples of the quickly increasing use of Dashify dashboards throughout the Cisco Observability Platform.

• EFS Volumes
• Elastic Load Balancers
• S3 Buckets
• EC2 Instances

Why Dashboards

No observability product can “pre-imagine” each method that clients wish to observe their programs. Dashboards permit end-users to create customized experiences, constructing on current in-product dashboards, or creating them from scratch. I’ve seen giant organizations with greater than 10,000 dashboards throughout dozens of groups.

Dashboards are a cornerstone of observability, forming a bridge between a distant information supply, and native show of information within the person’s browser. Dashboards are used to seize “scenarios” or “lenses” on a specific downside. They can serve a comparatively fastened use case, or they are often ad-hoc creations for a troubleshooting “war room.” A dashboard performs many steps and queries to derive the info wanted to handle the observability state of affairs, and to render the info into visualizations. Dashboards may be authored as soon as, and utilized by many various customers, leveraging the know-how of the writer to enlighten the viewers. Dashboards play a vital position in low-level troubleshooting and in rolling up high-level enterprise KPIs to executives.

The objective of dashboard frameworks has all the time been to supply a method for customers, versus ‘developers’, to construct helpful visualizations. Inherent to this “democratization” of visualizations is the notion that constructing a dashboard should in some way be simpler than a pure JavaScript app growth strategy. Afterall, dashboards cater to customers, not hardcore builders.

The downside with dashboard frameworks

The diagram under illustrates how a standard dashboard framework permits the writer to configure and organize elements however doesn’t permit the writer to create new elements or information sources. The dashboard writer is caught with no matter elements, layouts, and information sources are made accessible. This is as a result of the areas proven in crimson are developed in JavaScript and are supplied by the framework. JavaScript is neither a safe, nor simple know-how to study, due to this fact it’s hardly ever uncovered on to authors. Instead, dashboards expose a JSON or YAML primarily based DSL. This sometimes leaves discipline groups, SEs, and energy customers within the place of ready for the engineering crew to launch new elements, and there’s nearly a deep function backlog.

I’ve personally seen this state of affairs play out many occasions. To take an actual instance, a crew constructing dashboards for IT companies needed rows in a desk to be coloured in response to a “heat map”. This required a function request to be logged with engineering, and the core JavaScript-based Table element needed to be modified to help warmth maps. It grew to become typical for the core JS elements to grow to be a mishmash of domain-driven spaghetti code. Eventually the code for Table itself was onerous to search out amidst the handfuls of props and hidden behaviors like “heat maps”. Nobody was proud of the state of affairs, however it was typical, and core element groups principally spent their dash cycles constructing area behaviors and making an attempt to grasp the spaghetti. What if dashboard authors themselves on the power-user finish of the spectrum might be empowered to create elements themselves?

Enter Dashify

Dashify’s mission is to take away the barrier of “you can’t do that” and “we don’t have a component for that”. To accomplish this, Dashify rethinks among the foundations of conventional dashboard frameworks. The diagram under reveals that Dashify shifts the boundaries round what’s “built in” and what’s made fully accessible to the Author. This radical shift permits the core framework crew to deal with “pure” visualizations, and empowers area groups, who writer dashboards, to construct area particular behaviors like “IT heat maps” with out being blocked by the framework crew.

To accomplish this breakthrough, Dashify needed to resolve the important thing problem of how one can simplify and expose reactive habits and composition with out cracking open the proverbial can of JavaScript worms. To do that, Dashify leveraged a brand new JSON/YAML meta-language, created at Cisco within the open supply, for the aim of declarative, reactive state administration. This new meta-language known as “Stated,” and it’s getting used to drive dashboards, in addition to many different JSON/YAML configurations throughout the Cisco Observability Platform. Let’s take a easy instance to point out how Stated permits a dashboard writer to insert logic immediately right into a dashboard JSON/YAML.

Suppose we obtain information from a knowledge supply that gives “health” about AWS availability zones. Assume the well being information is up to date asynchronously. Now suppose we want to bind the altering well being information to a desk of “alerts” in response to some enterprise guidelines:

1. solely present alerts if the proportion of unhealthy situations is bigger than 10%
2. present alerts in descending order primarily based on proportion of unhealthy situations
3. replace the alerts each time the well being information is up to date (in different phrases declare a reactive dependency between alerts and well being).

This snippet illustrates a desired state, that adheres to the foundations.

But how can we construct a dashboard that repeatedly adheres to the three guidelines? If the well being information modifications, how can we ensure the alerts will probably be up to date? These questions get to the center of what it means for a system to be Reactive. This Reactive state of affairs is at finest tough to perform in in the present day’s standard dashboard frameworks.

Notice now we have framed this downside by way of the info and relationships between completely different information gadgets (well being and alerts), with out mentioning the person interface but. In the diagram above, notice the “data manipulation” layer. This layer permits us to create precisely these sorts of reactive (change pushed) relationships between information, decoupling the info from the visible elements.

Let’s have a look at how simple it’s in Dashify to create a reactive information rule that captures our three necessities. Dashify permits us to switch *any* piece of a dashboard with a reactive rule, so we merely write a reactive rule that generates the alerts from the well being. The Stated rule, starting on line 12 is a JSONata expression. Feel free to attempt it your self right here.

One of essentially the most fascinating issues is that it seems you don’t need to “tell” Dashify what information your rule will depend on. You simply write your rule. This simplicity is enabled by Stated’s compiler, which analyzes all the foundations within the template and produces a Reactive change graph. If you alter something that the ‘alerts’ rule is taking a look at, the ‘alerts’ rule will hearth, and recompute the alerts. Let’s shortly show this out utilizing the acknowledged REPL which lets us run and work together with Stated templates like Dashify dashboards. Let’s see what occurs if we use Stated to alter the primary zone’s unhealthy depend to 200. The screenshot under reveals execution of the command “.set /health/0/unhealthy 200” within the Stated JSON/YAML REPL. Dissecting this command, it says “set the value at json pointer /health/0/unhealthy to value 200”. We see that the alerts are instantly recomputed, and that us-east-1a is now current within the alerts with 99% unhealthy.

By recasting a lot of dashboarding as a reactive information downside, and by offering a strong in-dashboard expression language, Dashify permits authors to do each conventional dashboard creation, superior information bindings, and reusable element creation. Although fairly trivial, this instance clearly reveals how Dashify differentiates its core know-how from different frameworks that lack reactive, declarative, information bindings. In truth, Dashify is the primary, and solely framework to function declarative, reactive, information bindings.

Let’s take one other instance, this time fetching information from a distant API. Let’s say we wish to fetch information from the Star Wars REST api. Business necessities:

1. Developer can set what number of pages of planets to return
2. Planet particulars are fetched from star wars api (https://swapi.dev)
3. List of planet names is extracted from returned planet particulars
4. User ought to be capable of choose a planet from the checklist of planets
5.  ‘residents’ URLs are extracted from planet information (that we received in step 2), and resident particulars are fetched for every URL
6. Full names of inhabitants are extracted from resident particulars and introduced as checklist

Again, we see that earlier than we even take into account the person interface, we will forged this downside as a knowledge fetching and reactive binding downside. The dashboard snippet under reveals how a worth like “residents” is reactively sure to chosePlanet and the way map/cut back type set operators are utilized to whole outcomes of a REST question. Again, all of the expressions are written within the grammar of JSONata.

To show how one can work together with and check such a snippet, checkout This github gist reveals a REPL session the place we:

1. load the JSON file and observe the default output for Tatooine
2. Display the reactive change-plan for planetName
3. Set the planet identify to “Coruscant”
4. Call the onSelect() perform with “Naboo” (this demonstrates that we will create features accessible from JavaScript, to be used as click on handlers, however produces the identical consequence as immediately setting planetName)

From this concise instance, we will see that dashboard authors can simply deal with fetching information from distant APIs, and carry out extractions and transformations, in addition to set up click on handlers. All these artifacts may be examined from the Stated REPL earlier than we load them right into a dashboard. This exceptional economic system of code and ease of growth can’t be achieved with some other dashboard framework.
If you might be curious, these are the inhabitants of Naboo:

What’s subsequent?

We have proven lots of “data code” on this publish. This is just not meant to suggest that constructing Dashify dashboards requires “coding”. Rather, it’s to point out that the foundational layer, which helps our Dashboard constructing GUIs is constructed on very stable basis. Dashify not too long ago made its debut within the CCO product with the introduction of AWS monitoring dashboards, and Data Security Posture Management screens. Dashify dashboards are actually a core element of the Cisco Observability Platform and have been confirmed out over many advanced use circumstances. In calendar Q2 2024, COP will introduce the dashboard enhancing expertise which gives authors with inbuilt visible drag-n-drop type enhancing of dashboards. Also in calendar Q2, COP introduces the flexibility to bundle dashify dashboards into COP options permitting third occasion builders to unleash their dashboarding abilities. So, climate you skew to the “give me a gui” finish of the spectrum or the “let me code” way of life, Dashify is designed to fulfill your wants.

Summing it up

Dashboards are a key, maybe THE key know-how in an observability platform. Existing dashboarding frameworks current unwelcome limits on what authors can do. Dashify is a brand new dashboarding framework born from many collective years of expertise constructing each dashboard frameworks and their visible elements. Dashify brings declarative, reactive state administration into the palms of dashboard authors by incorporating the Stated meta-language into the JSON and YAML of dashboards. By rethinking the basics of information administration within the person interface, Dashify permits authors unprecedented freedom. Using Dashify, area groups can ship advanced elements and behaviors with out getting slowed down within the underlying JavaScript frameworks. Stay tuned for extra posts the place we dig into the thrilling capabilities of Dashify: Custom Dashboard Editor, Widget Playground, and Scalable Vector Graphics.

Related sources

Share: