Skip to main content

Architecture

note

As of Nov 27, 2023, this document is mostly out of date. Dropbase now allows a local setup for both the Dropbase Client and Dropbase Worker via Docker. This means that the proxy is no longer needed. When setup locally, your local Dropbase client directly talks to your local Dropbase Worker.

We will update this page soon.

Motivation

With our architecture, we wanted to achieve 3 main things:

  1. Keep the part of our system that processes developers’ data within their infra. They can modify and improve any code that accesses or processes their data. Anything that queries or modifies sensitive data needs to go through this.
  2. Developers hosts the code they write and can also import their existing scripts and libraries.
  3. Make it easy for developers to get the latest improvements, components, and updates from Dropbase, without breaking their deployments each time we push changes.

Overview

At a high level, the Dropbase cloud is designed with a classical client-backend setup, where metadata is stored in a SQL database. The client calls user’s scripts via a worker server, hosted in our customers’ infrastructure. To create a better development experience, the built-in code editor is connected to a Python LSP server, which provides auto-completion and linting. Finally, a proxy is used to securely connect servers running on users’ local infra with the Dropbase Cloud.

There are 2 main sides of the architecture, each hosting different metadata:

  • Dropbase Cloud: hosted on Dropbase’s infrastructure, it hosts all aspects of the platform that involves management of users and app components including:
    • User authentication and permissions
    • Properties and of UI components
    • App settings and proxy tokens
  • Dropbase Worker: hosted on customers’ infrastructure, it hosts all the scripts that access and process user data including:
    • SQL scripts to query user databases
    • Python scripts to fetch and process data

Here’s an architectural diagram:

Dropbase architecture overview

Note that sensitive parts of the platform (pieces that interact with user data), are isolated from Dropbase server. Not only are they hosted on the user infra, but they are also gated via worker server.

Any request that involves execution of a function or fetching of data is sent to the worker server, which routes them to workspace scripts or Dropbase server.

note

This architectural diagram is only applicable to local development deployments. For self-hosted cloud deployments, a proxy isn't necessary.

Now, let’s dive into each in more details.

Dropbase Cloud

Dropbase Client

It is the main entry point for the web application (client) hosted at app.dropbase.io. It is hosted on Dropbase infra and it is responsible for user/resource management and rendering UI components with data. Users trigger requests through the client UI from their browsers.

Dropbase Server

User and resource information (metadata) is stored in Dropbase database. this information is accessed by client via Dropbase backend (classical frontend-backend setup).

Proxy server

This is the server side of the proxy that accepts connections from user’s local machines.

note

Proxy server is only needed for local development. For self-hosted cloud deployments, we don't use proxies.

Dropbase Worker

All the actions that involve accessing or processing user data are handled by the Dropbase Worker. Worker consists of 4 parts:

Worker Server

A web server that processes client requests. The requests server handles falls into the following categories:

  • Querying user data sources (data fetching from database and APIs)
  • Running UI or generic Python functions
  • Serving static files from workspace directory (for studio mode)
  • CRUD operations on files in the workspace directory
    • CRUD SQL files
    • CRUD app/page directories
  • Synchronize State and Context for a page
note

Database credentials and api tokens are also stored on the user side in .env file.

Workspaces

A directory where functions and data models are stored in a form of files. workspace directory is structured as follows:

workspace/
└── app_name/
└── page_name/
├── context.py
├── state.py
└── scripts/
├── functions.py
└── queries.sql

LSP Server

Language Server Protocol based on the python-lsp-server that provides linting and autocomplete for code editor in developer studio mode

Proxy Client

Client side of the proxy that exposes server and lsp to the proxy server on Dropbase Cloud

note

To avoid conflicts, user should only modify files that interact with their data (data fetchers, UI functions) and do not modify server, proxy or LSP code.

note

Proxy client is only needed for local development. For self-hosted cloud deployments, we don't use proxies.

Dataflow diagrams

Running Data Fetcher function

Data fetcher dataflow

  1. User opens a page, which loads a table and triggers data fetcher function bound to the table
  2. Dropbase Client sends a POST request to Worker Server’s /query endpoint
  3. Worker Server starts a new Worker Process and passes request payload as input arguments
  4. Worker Process fetches required files from Workspace
  5. With Data Fetching content in hands, Worker Process queries Data Source
  6. Once complete, Worker Process returns results back to Worker Server
  7. Worker Server passes the response back Dropbase Client to update the table

Note: for longer running tasks, a queue system will be implemented

Running a UI function

UI function dataflow

When user clicks on a UI component that is bound to a UI function, the following steps are taking place:

  1. User clicks on the button requesting to run a UI function
  2. Dropbase Client sends a POST request to Worker Server’s /function endpoint
  3. Worker Server starts a new Worker Process and passes request payload as input arguments
  4. Worker Process fetches required files from Workspace and runs requested function
  5. Once complete, Worker Process returns results back to Worker Server
  6. Worker Server passes the response back Dropbase Client to update UI and State, Context

Adding UI Components

UI component dataflow

  1. User adds a new component
  2. Dropbase Client sends a POST request to Worker Server’s /components endpoint
  3. Worker Server calls Dropbase Server with the same payload
  4. Dropbase Server adds components to the Dropbase Database
  5. Dropbase Server generates a new State and Context for the component’s page and returns them to Worker Server
  6. Worker Server updates state.py and context.py files in Workspace
  7. Worker Server returns State and Context back to the Dropbase Client
  8. Dropbase Client fetches new components from Dropbase Server and updates UI with the latest changes