Architecture

Motivation

With our architecture, we wanted to achieve 3 main things:

1. Keep the part of our system that processes developers’ data within their infra. They can modify and improve any code that accesses or processes their data. Anything that queries or modifies sensitive data needs to go through this.
2. Developers hosts the code they write and can also import their existing scripts and libraries.
3. Make it easy for developers to get the latest improvements, components, and updates from Dropbase, without breaking their deployments each time we push changes.

Overview

To understand the architecture of the Dropbase platform, it's important to understand the different components and how they interact with each other.

Let's go through some of the major components of the Dropbase platform:

• Dropbase Server: The Dropbase Server is the main server that hosts all aspects of the platform that involves management of users and app permissions. This includes:

  • User authentication
  • Permissions
  • Worker settings

• Dropbase Client: The Dropbase Client is the main UI that users interact with. It is currently hosted on Dropbase's infrastructure and is available to be self-hosted. The client is responsible for:

  • Hosting the code editor
  • Providing a Python LSP server for auto-completion and linting
  • Managing app components and updates

• Dropbase Worker: The Dropbase Worker is hosted on customers’ infrastructure. It hosts all the scripts that access and process user data and holds most of the data of a workspace. This includes:

  • SQL scripts to query user databases
  • Python scripts to fetch and process data
  • Any other scripts that users write
  • All information inside an app, including tables, functions, and UI components

Here’s an architectural diagram:

Note that the parts of the platform that interact with user data (Dropbase Worker and to an extent the Client), are hosted on a users infrastructure, isolated from Dropbase's server.

Component Deep Dive

Now that we have an overview of the architecture, let's dive into each component in more detail.

Dropbase Server

The Dropbase server is primarily responsible for managing user authentication and authorization. This means that we store all user information, including user credentials, in the Dropbase server. We also store information to make correct authorization decisions to enforce the permissions set by the user.

The Dropbase server also stores information about app meta data, such as an app's ID so that we can correctly identify and manage the app.

Dropbase Client

The Dropbase client is the main UI that users interact with. We currently have two versions of the client. The first version is available on app.dropbase.io and is hosted on Dropbase's infrastructure. The second version is available to be self-hosted.

Currently, the recommended way to use the two versions of Dropbase client is to use the hosted version on app.dropbase.io to sign up, setup your self hosted worker, and manage user permissions. Once you have your worker setup, we recommend that you use the self-hosted client to interact with apps and app building.

Dropbase Worker

The Dropbase worker is hosted on customers’ infrastructure. It is primarily responsible for actions that access or modify user data, and for storing and managing the structure of your workspace.

The worker itself can be split into a couple of different components. Let's briefly go through each of them:

The Web Server

The web server processed requests from the Dropbase client. It is responsible for:

• Querying user data sources
• Running user functions
• Serving static files from the workspace directory
• Managing the workspace with CRUD operations
• Syncronizing the State and Context objects between the client and the worker

The Workspace Directory

The workspace directory is where the structure of your workspace is stored. This directory will house all the info about your app, and everything in it including tables, functions, and UI components.

Here's a sample structure of a workspace directory:

    workspace/
    └── test_app_1/
        └── page_1/
            ├── context.py
            ├── state.py
            └── scripts/
                ├── do_something.py
                └── fetch_customers.sql
            └── properties.json
        └── properties.json
    └── __init__.py
    └── properties.json

note

Rather than storing the workspace structure in a database, we store it in a directory. This allows users to easily version control their workspace and potentially share it with others. The speed of reading and writing to the workspace directory is also much faster than calling Dropbase server to fetch and modify the structure of your workspace from a database.

LSP Server

The LSP server is a Python Language Server Protocol server that provides auto-completion and linting for Python code. It is used by the Dropbase client to provide a better coding experience for users. With this server, we give three big benefits to users:

1. Users can write and modify their own Python code with a similar experience to their favorite code editor.
2. Users can use autocomplete to use state and context objects in their Python or SQL code.
3. Code is automatically saved and synced to the workspace directory.

Data Flow

Now, that we have an idea of what the different components are, let's take a look at how they interact with each other.

Fetching Data

Let's go through an example where a user opens a table on an app. Assuming the table is bound to a SQL data fetcher, here's what happens:

1. The user opens the app, and the data fetcher bound to the table is triggered. This sends a request to Dropbase Worker
2. The worker sends user, resource, and action information to Dropbase Server to check if the user has the correct permissions to access the data
3. If the user has the correct permissions, Dropbase Worker continues to process the request. It goes to the workspace directory to find the SQL data fetcher bound to the table
4. The worker then runs the SQL data fetcher on the relevant data source
5. The worker then sends the data back to the Dropbase Client, which displays the data in the table

Adding UI Components

Let's go through an example where a user adds a new UI component to an app. Here's what happens:

1. The user adds a new UI component to the app. This sends a request to Dropbase Worker
2. The worker sends user, resource, and action information to Dropbase Server to check if the user has the correct permissions to add the UI component
3. If the user has the correct permissions, Dropbase Worker continues to process the request. It goes to the workspace directory to add the new UI component by modifying relevant app structure files in the workspace directory
4. The worker then sends the updated app structure back to the Dropbase Client, which displays the new UI component