Evaluation Guide: Universal Catalog

Internal Developer Portals are experiencing a wave of adoption; however, evaluating the capabilities of these platforms to determine which one is right for you can be difficult. In this post, we explore how to evaluate the universal catalog component of an internal developer portal.

An internal developer portal is comprised of 3 core capabilities:

• Universal catalog
• Analytics, like Scorecards
• Self-serve actions (i.e., reusable golden paths)

Think of a universal catalog as the database for your developer portal. It needs to be comprehensively populated in order to enable the full range of insights and actions that will improve your developer experience. It also needs to have the expressiveness necessary to map to how you think about your systems. Finally, it needs to be easy to set-up and drift-free. 

Let’s explore these concepts, starting with comprehensiveness.

Comprehensiveness

Comprehensiveness refers to the completeness of the knowledge about your system captured in the catalog. We unpack this based on the layers of your system:

Applications: Is the catalog aware of applications and the services that comprise them, or does its data model start with microservices? Application awareness is important because it enables engineers and managers to understand at a glance all of the services that comprise an application and key information about them, such as health and other key measures. It also enables analytics to be presented in ways that have meaning to your stakeholders (i.e., seeing cloud costs at the application level).

Services: 

• Breadth: What breadth of service types is the catalog aware of? Microservice only? Or does it also support monoliths, serverless functions, data pipelines, and third-party services like Twillio? Breadth impacts your ability to comprehensively represent the components of an application.
• ‍Context: How much context does the catalog enable about a given service? Is it simply owners and documentation? Or is it effortlessly organizing service-specific Jira tickets, CI/CD runs, security metrics, SLOs, resource monitors, repository information, and other contextual information engineers need?

Environment: Environments are groupings of resources that power an individual service. Environment “awareness” is important for self-serve actions (i.e., which environment does a developer need access to for a given service?) as well as for analytics (i.e., are there orphaned resources not mapped to an environment burning your cloud budget?) and a developer’s understanding during an incident, learning a system, and other scenarios.‍

Resources: Here are some helpful evaluation dimensions to aid in your review of a catalog’s resource “awareness”:

• Clouds: Does it support your cloud(s)?
• Resources: Which cloud resources are supported? Only a few common ones like EKS, S3 and EC2, or many if not all of the resources you use across cloud providers as well as any custom resources? This is important in terms of supporting scenarios related to analytics, self-serve actions, and catalog use cases for your ops team and developers.
• Data Breadth, Visibility and Freshness: Is the catalog ingesting critical information about a resource (i.e., an EKS cluster’s deep configuration data, pod and container relationships, which VPC it is in and what else is in that VPC, or is it simply showing that you have an EKS cluster and you need to context shift and search a cloud console in order to find the information you’re seeking or take an action? How current is the resource data? Can you see what changed over time?

Data Model Flexibility

It’s important to diligence the flexibility of a catalog’s data model in order to understand if it is sufficiently expressive to map to how you think about your system (or will it lock you into an predetermined schema that is at odds with your organization’s mental model). This has implications for the value of the analytics you are able to run in addition to the intuitiveness of the portal for developers. Does the data model allow you to create your own custom relationships?

Ease of Set-Up and Drift Management

Set-up and drift management can be two non-obvious costs with the potential to dramatically impact the total cost of ownership and usefulness of your internal developer portal. It’s important to understand these burdens to gauge your time to value internally as well.  Here are some helpful questions you can ask to better understand the relative ease of a given platform:

• Data Ingestion: Do you have to send in all the data yourself, or can it be ingested based on single-click integrations with your tools and clouds?
• Associations: Do you need to manually map resources to environments, or does your catalog do this automatically? Do you need to manually build out services and declare dependencies and configuration information, or can your catalog automatically map these items to minimize set-up as well as minimize drift and the associated maintenance burden?
• Drift: Is the catalog able to keep itself in sync with your services, dependencies, and resources automatically, or does it require maintenance? How often do manual elements drift? For automated syncs, do they capture changes at an interval that is appropriate for your system?

Conclusion

Internal developer portals are transforming how organizations improve their developer experience. By making this knowledge and functionality available on a self-serve basis, developer portals can offer rapid time-to-value as well as significantly improve your organization’s development velocity and ability to achieve best practices. We hope this guide helps you conduct a rigorous assessment of the catalog component of an internal develop portal in order to find the solution that is right for the needs of your organization.