Observability is the ability to measure the internal states of a system by examining its outputs. The three pillars of observability (metrics, logs, and traces) can be used to get insights into system behavior, identify and resolve performance issues to improve system efficiency, and provide responsiveness into complex systems on bottlenecks and failures.

In the previous introduction, we explored how ScopeDB can be used to store and manage observability data at a petabyte scale. In this post, we will discuss how we observe ScopeDB and how its self-observability helps us evolve it into a more reliable and efficient system.

SQL has dominated structured data processing for 50 years. However, its age reveals significant design flaws, notably non-composable and inconsistent syntax.

When developing the query language for ScopeDB, we decided to go against SQL and design a new language, ScopeQL, from scratch to fix SQL's problems. In this post, we will discuss where SQL falls short and the design principles behind ScopeQL.

TL;DR​

• Both SQL and ScopeQL are based on relational algebra, a powerful, elegant, proven theory.
• SQL query is out of its semantic order; ScopeQL fixes it with a linear pipelined syntax.
• SQL has a rigid and arbitrary syntax; ScopeQL's syntax is consistent, composable, and reduced.
• SQL relies on subqueries heavily, while ScopeQL eliminates most of them.

Here is a showcase:

SELECT
  country,
  MAX(salary) AS max_salary
FROM
  employees
WHERE
  start_date > '2025-01-01 00:00:00+00:00'::TIMESTAMPTZ
GROUP BY
  country
HAVING
  MAX(salary) > 100000

FROM employees
WHERE start_date > '2021-01-01 00:00:00+00:00'::timestamp
GROUP BY country AGGREGATE max(salary) AS max_salary
WHERE max_salary > 100000

What is an Algebraic Data Type?​

An algebraic data type is a structured type that is formed by combining other types. Most programming languages support algebraic data types in some form. The two most common types are:

1. Product Types: A product type is a type that is composed of other types, such as tuples and records. For example, a Person type might be composed of a name and an age:

   struct Person {
       name: String.
       age: u64,
   }
   
2. Sum Types: A sum type is a type that can be one of several types, such as tagged or disjoint unions. For example, a Shape type might be a Circle or a Square.

   enum Shape {
      Circle { radius: f64 },
      Rectangle { length: f64, width: f64 }
   }
   

Algebraic data types are a common and powerful tool for modeling complex data structures in programming. While you can map some product types to database tables, sum types are more challenging to represent in a relational database. This lack of expression causes impedance mismatches between the application and the database. In this article, we'll explore how sum types can be implemented in ScopeDB using variant data types.

TL;DR​

ScopeDB is a database built directly on top of S3 and unleashes the power of cloud elasticity and workload isolation. ScopeDB servers are launched with a single, stateless Rust binary; thus, it does not need to manage local disks, data sharding, or complex leader election. ScopeDB achieves ten times cost efficiency and performance improvement compared to databases built with shared-nothing architecture by delegating data replication to S3 and dynamically adjusting the number of servers based on the workload.

If you're eager to dive right in, you can try out our playground:

git clone https://github.com/scopedb/community.git
docker compose -f ./community/playground/docker-compose.yml up -d
docker run -it --rm --entrypoint /bin/scopeql --network host scopedb/scopedb

Otherwise, let's begin with the story!