Announcing DoltLab v2.1.0 and its new Installer
We recently released DoltLab v2.1.0, that contains a brand new tool for configuring a DoltLab instance: the installer. The DoltLab installer is a single binary that will generate all the assets DoltLab needs to run. For those familiar with previous versions of DoltLab, this is a HUGE improvement.
Prior to the installer, DoltLab would ship a zip folder containing a variety of configuration files, templates, and scripts required for DoltLab to run via Docker Compose. Whenever DoltLab administrators needed to make changes to their setup, they'd have to manually edit these files and keep track of what changes they made, as they'd have to also perform these edits again after upgrading their DoltLab instance.
But now, DoltLab's zip contains only the installer. It handles all of the asset generation for you in a deterministic way, which is a huge improvement to administrator experience, but also a big change from previous versions of DoltLab. Today were gonna take a deep dive on the installer and how it can be used to simplify your DoltLab setup moving forward.
Before jumping into the specifics of the installer and how to use it, I think it's important to have a bit more context about DoltLab, its services, and how they work together in order to demystify some of the assets the installer generates and how DoltLab will use them. I've written about these services before, but I think a refresher will serve nicely here.
DoltLab Overview
Put simply, DoltLab is a collection of services that run via Docker Compose on a single Linux host. The services that make up DoltLab are the same services (with a few exceptions) that make up DoltHub.com, since DoltLab's services share their code with DoltHub. The following is a break down of the services that comprise DoltLab and their role. Each section will also contains snippets of the base docker-compose.yaml file that the installer tool generates in order to run DoltLab using Docker Compose.
DoltLab's Application Database (doltlabdb)
DoltLab's application database is a Dolt server (formerly a PostgreSQL server) that backs DoltLab's main API, DoltLab API (doltlabapi). This database stores DoltLab users, repository (database) metadata, pull requests, credentials and anything else that doltlabapi needs to perform its various operations or serve to the frontend. It does not store repository (database) data or data from user uploaded files. These are stored on disk in separate Docker volumes, but we will touch more on that later.
The doltlabdb server image is actually the Dolt SQL Server Docker image called dolthub/dolt-sql-server with the following statements used to initialize the database:
CREATE USER 'dolthubadmin' IDENTIFIED BY '$DOLT_PASSWORD';
CREATE USER 'dolthubapi' IDENTIFIED BY '$DOLTHUBAPI_PASSWORD';
CREATE DATABASE dolthubapi;
GRANT ALL ON *.* TO 'dolthubadmin';
GRANT ALL ON dolthubapi.* TO 'dolthubapi';This creates an admin role dolthubadmin and a dolthubapi role (artifacts of the code that DoltLab shares with DoltHub). The dolthubadmin role is only used by database administrators when they need to run a shell against the running server. The dolthubapi role is used by doltlabapi.
As mentioned above, the installer will produce a docker-compose.yaml file used to run DoltLab with Docker Compose, and the following section corresponds to the doltlabdb service:
doltlabdb:
image: public.ecr.aws/dolthub/doltlab/dolt-sql-server:v2.1.0
command: |-
-l debug
environment:
DOLT_PASSWORD: "${DOLT_PASSWORD}"
DOLTHUBAPI_PASSWORD: "${DOLTHUBAPI_PASSWORD}"
networks:
- default
volumes:
- doltlabdb-dolt-data:/var/lib/dolt
- doltlabdb-dolt-root:/.dolt
- doltlabdb-dolt-configs:/etc/dolt
- doltlabdb-dolt-backups:/backupsBy default, the server is run with -l debug, or log level DEBUG. This is so the server logs contain the queries that were run for easier debugging.
DOLT_PASSWORD and DOLTHUBAPI_PASSWORD are configured with environment variables used by the initialization script of the doltlabdb server and various Docker volumes are mapped to corresponding paths on the container so that data is persisted to disk. You'll see the following section for these volumes in the docker-compose.yaml file as well:
volumes:
doltlabdb-dolt-backups: {}
doltlabdb-dolt-configs: {}
doltlabdb-dolt-data: {}
doltlabdb-dolt-root: {}doltlabdb-dolt-data stores all of the Dolt database data, while doltlabdb-dolt-configs stores Dolt server configuration files and doltlabdb-dolt-root contains anything stored at the DOLT_ROOT_PATH. doltlabdb-dolt-backups can be used to create file system based backups of the Dolt server, but is otherwise unused by DoltLab.
DoltLab's Main API (doltlabapi)
DoltLab main API server, doltlabapi, is the powerhouse of DoltLab and by far its most complex service. It is a gRPC service that supports all of the functionality you see on DoltLab including database creation, file importing, searching, pull request merging, and online data editing. This is also the service responsible for emailing users at various times, and thus requires arguments for connecting to an existing SMTP server, but we'll go over that more shortly.
When the service first comes up, it connects to doltlabdb and is the only service that reads from it and writes to it. After establishing a connection, it then runs a series of database schema migrations. These migrations are the result of new features and bug fixes supported in both DoltLab and DoltHub and are referenced in the schema_migrations table. Details about the schema migrations executed by doltlabapi are not crucial for setting up or maintaining a DoltLab instance, but it's important to note that new versions of DoltLab often run new schema migrations against the doltlabdb. If for some reason those migrations fail, this service will crash and require manual updates to the schema_migrations table.
The doltlabapi section of the docker-compose.yaml file generated by the installer is by far the largest and most complex section of the file, and will change depending on the installer arguments supplied.
doltlabapi:
depends_on:
- doltlabdb
- doltlabenvoy
- doltlabfileserviceapi
- doltlabremoteapi
image: public.ecr.aws/dolthub/doltlab/dolthubapi-server:v2.1.0
command: |-
-doltlab
-outboundInternalServiceEndpointHost doltlabenvoy
-iterTokenEncKeysFile /iter_token.keys
-iterTokenDecKeysFile /iter_token.keys
-processAsyncTasks
-port 60051
-noReplyEmail "me@email.com"
-emailAuthMethod plain
-emailHost email.smtp.host.com
-emailPort 587
-emailUsername "${EMAIL_USERNAME}"
-emailPassword "${EMAIL_PASSWORD}"
-emailOauthToken "${EMAIL_OAUTH_TOKEN}"
-defaultUser "admin"
-defaultUserPassword "${DEFAULT_USER_PASSWORD}"
-defaultUserEmail "me@email.com"
-metricsHost "eventsapi.dolthub.com"
-migrateUp
-importJobContainerImage "public.ecr.aws/dolthub/doltlab/file-importer:v2.1.0"
-mergeJobContainerImage "public.ecr.aws/dolthub/doltlab/pull-merge:v2.1.0"
-sqlReadJobContainerImage "public.ecr.aws/dolthub/doltlab/query-job:v2.1.0"
-jobEnvoyHost "doltlabenvoy"
-dolthubWhitelistAllowAll
-doltUser dolthubapi
-doltHost doltlabdb
-doltPort 3306
-remoteApiHostNameOverrideKey "doltlab.dolthub.com:100"
-userImportUploadsFileServiceBrowserHost "http://doltlab.dolthub.com:4321"
-websiteURL "http://doltlab.dolthub.com"
-doltremoteapi.url "doltlab.dolthub.com"
-doltremoteapi.port "50051"
-doltlabRemoteApiHost "doltlab.dolthub.com"
-doltlabRemoteApiPort "50051"
-remoteApiHostNameOverrideValue "doltlabremoteapi:100"
-userImportUploadsFileServiceServerHost "http://doltlabenvoy:4321"
-fileserviceapiServiceEndpoint "http://doltlabenvoy:4321"
environment:
BUILD_SCM_REVISION: "doltlabapi-v2.1.0"
DOLT_DOLTHUBAPI_PASSWORD: "${DOLT_PASSWORD}"
DOLTHUB_METRICS_ENABLED: "true"
DOLTHUBAPI_PASSWORD: "${DOLTHUBAPI_PASSWORD}"
MAILCHIMP_API_KEY: "does_not_work"
MAILCHIMP_LIST_ID: "does_not_work"
STRIPE_API_KEY: "does_not_work"
networks:
- default
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/ubuntu/doltlab/doltlabapi/iter_token.keys:/iter_token.keysAs you can see there's quite a lot going on with doltlabapi, too much to cover in this blog alone. And prior to DoltLab's installer, this section would need to be hand edited for configuration changes, yikes!
Anyway, two things to make note of are the volumes mounted to the container. First the Docker socket /var/run/docker.sock is mounted to the container. This is so that doltlabapi can deploy DoltLab Jobs, that allow file imports, pull request merges, and large read queries to run as standalone Docker containers.
Second, a local file called doltlabapi/iter_token.keys, generated by the installer, is used to encrypt outbound tokens and decrypt inbound tokens produced by doltlabapi.
Aside from these volumes mounted on the container, all other doltlabapi configuration options are supplied either by argument or by environment variable, and the installer is now the main interface for changing these options.
DoltLab Remote API (doltlabremoteapi)
The next service we'll cover, another gRPC service, is DoltLab's remote service called doltlabremoteapi. This service manages all the remote data stored on a DoltLab instance.
Let's say a user creates a new database on DoltLab called user1/test, and they push data to that DoltLab database. The data that they push is not stored in the application database doltlabdb, but is instead written to disk by doltlabremoteapi. Similarly, if a different user clones user1/test, they clone the data (using their Dolt client) through doltlabremoteapi.
This service is responsible for serving remote data to authorized parties, and performs request authentication by interacting with doltlabapi. Here's what you'll find in the doltlabremoteapi section of the docker-compose.yaml created by the installer.
doltlabremoteapi:
depends_on:
- doltlabdb
- doltlabenvoy
- doltlabfileserviceapi
image: public.ecr.aws/dolthub/doltlab/doltremoteapi-server:v2.1.0
command: |-
-http-port "100"
-outboundInternalServiceEndpointHost doltlabenvoy
-repoTokenEncKeysFile /iter_token.keys
-repoTokenDecKeysFile /iter_token.keys
-backing-store local
-dir /doltlab-remote-storage
-metricsHost eventsapi.dolthub.com
-backingStoreHostNameOverrideKey ":100"
-doltlab
-allow-nbf-dolt
-backingStoreHostNameOverrideValue "doltlabremoteapi:100"
-http-host ":100"
environment:
BUILD_SCM_REVISION: "doltlabremoteapi-v2.1.0"
DOLTHUB_METRICS_ENABLED: "true"
networks:
- default
volumes:
- /home/ubuntu/doltlab/doltlabremoteapi/iter_token.keys:/iter_token.keys
- doltlab-remote-storage:/doltlab-remote-storage
ports:
- "50051:50051"Here you'll see that the port 50051 is exposed on the container. This port must also be accessible on the DoltLab host so that Dolt clients can interact with this service during cloning, pushing, pulling, and fetching.
This service also gets a key file mounted to the container. /doltlabremoteapi/iter_token.keys is created by the installer and doltlabremoteapi uses this file, like doltlabapi, to encrypt and decrypt tokens.
There's an additional volume mount as well called doltlab-remote-storage. This Docker volume is where doltlabremoteapi actually writes all database data on local disk, and serves it from disk to doltlabapi and Dolt clients alike. Although not seen explicitly in the service definition above, doltlabremoteapi serves this data on port 100 so this port must also be exposed on the DoltLab host. Connections to this port are made through DoltLab's reverse proxy, doltlabenvoy, discussed later.
You can find the definition of the doltlab-remote-storage volume in the docker-compose.yaml top level volumes section.
DoltLab File Service API (doltlabfileserviceapi)
DoltLab's next service is called the doltlabfileserviceapi and this service is unique to DoltLab. This service is responsible for storing and retrieving files uploaded by users of the DoltLab instance, most often during the online file import process. doltlabfileserviceapi writes all of these files to DoltLab's local disk and authenticates incoming requests against doltlabapi.
DoltHub does not have an equivalent service, since it uploads user uploaded files directly to AWS S3.
The following section of the docker-compose.yaml file corresponds to this service:
doltlabfileserviceapi:
depends_on:
- doltlabenvoy
image: public.ecr.aws/dolthub/doltlab/fileserviceapi-server:v2.1.0
command: |-
-outboundInternalServiceEndpointHost doltlabenvoy
-iterTokenEncKeysFile /iter_token.keys
-iterTokenDecKeysFile /iter_token.keys
-dir /doltlab-user-uploads
-doltlab
environment:
BUILD_SCM_REVISION: "doltlabfileserviceapi-v2.1.0"
DOLTHUB_METRICS_ENABLED: "true"
networks:
- default
volumes:
- /home/ubuntu/doltlab/doltlabfileserviceapi/iter_token.keys:/iter_token.keys
- doltlab-user-uploads:/doltlab-user-uploadsSimilarly to doltlabapi and doltlabremoteapi, we can see that the installer will generate doltlabfileserviceapi/iter_token.keys so it can be mounted in the container for token encryption and decryption.
The Docker volume doltlab-user-uploads is also mounted to the container and is where the service will write all uploaded files.
Not shown in this service definition is the port that doltlabfileserviceapi uses for incoming requests, port 4321. This port must be accessible on the DoltLab host so that browsers can upload files to doltlabfileserviceapi. This port will be exposed by doltlabenvoy, DoltLab's reverse proxy described below.
DoltLab GraphQL API (doltlabgraphql)
Next we'll look at the first part of DoltLab's frontend services, the GraphQL service called doltlabgraphql. This service is a data layer API, that provides a uniform interface for DoltLab UI requests to and responses from doltlabapi. This service has very low user facing impact, as it primarily acts as a middleware service between DoltLab's frontend, doltlabui, and doltlabapi.
In the generated docker-compose.yaml file, the following corresponds to the doltlabgraphql section:
doltlabgraphql:
depends_on:
- doltlabdb
- doltlabenvoy
- doltlabremoteapi
- doltlabapi
- doltlabfileserviceapi
image: public.ecr.aws/dolthub/doltlab/dolthubapi-graphql-server:v2.1.0
environment:
DOLTHUBAPI_URL: "http://doltlabenvoy:9443"
NODE_ENV: "development"
networks:
- defaultNot too much else to say about this service except that the NODE_ENV=development environment variable tells GraphQL to run a GUI at /graphql which can be used to execute queries against doltlabapi manually, although this endpoint is not exposed on the container definition by default.
DoltLab UI (doltlabui)
DoltLab's UI, called doltlabui is a Next.js application that provides the DoltLab user interface. This service interacts only with doltlabgraphql, which in turn interacts with doltlabapi.
Here is the service definition for doltlabui:
doltlabui:
depends_on:
- doltlabdb
- doltlabenvoy
- doltlabfileserviceapi
- doltlabremoteapi
- doltlabapi
- doltlabgraphql
image: public.ecr.aws/dolthub/doltlab/dolthub-server:v2.1.0
environment:
DOLTHUB_METRICS_ENABLED: "true"
DOLTHUB_VERSION: "v2.1.0"
EVENTSAPI_URL: "https://eventsapi.dolthub.com"
INTERNAL_GRAPHQLAPI_URL: "http://doltlabenvoy:10080/graphql"
IS_DOLTLAB_INSTANCE: "1"
NODE_OPTIONS: "--max-old-space-size=16384"
STRIPE_API_KEY: "does_not_work"
networks:
- defaultSimilar to doltlabgraphql, not a lot of configuration going on here, although DoltLab Enterprise customers will see this service definition change if they use the installer to customize their DoltLab's UI.
We'll touch more on that in the DoltLab Enterprise section below.
DoltLab's Envoy Reverse Proxy (doltlabenvoy)
DoltLab runs behind an Envoy Proxy that performs a variety of functions including routing ingress requests to their proper targets as well as facilitating internal requests between services.
The service definition for doltlabenvoy is as follows:
doltlabenvoy:
image: envoyproxy/envoy:v1.28-latest
command: |-
-c /envoy.json
networks:
- default
volumes:
- /home/ubuntu/doltlab/envoy.json:/envoy.json
ports:
- "100:100"
- "4321:4321"
- "7770:7770"
- "2001:2001"
- "80:80"doltlabenvoy requires a configuration file to be mounted on the container, and the installer will generate this envoy.json for us. Unfortunately the configuration of the Envoy proxy is outside the scope of this post, so we won't detail the generated envoy.json file here. There are some important things to note about the port definitions in the docker-compose.yaml file above, though.
We can see from the ports exposed on the container above that requests for remote data files will go through this proxy on port 100 to doltremoteapi, requests for doltlabfileserviceapi will go through this proxy on 4321, and requests for our DoltLab's instance's homepage will go through this proxy to doltlabui on port 80.
Port 7770 is used for service metrics as discussed in a DoltLab monitoring blog and port 2001 can be used to health check the proxy, as it will return a 200 to GET requests.
This service is a crucial piece of DoltLab's infrastructure, so it's good to be aware of its role within the suite of services, but like doltlabgraphql, when its functioning properly, it has low end-user impact.
Getting Started with the Installer
Now that you have an updated overview of DoltLab's services, it's time to use the installer to get a DoltLab instance up and running! Let's start by configuring the most basic version of DoltLab v2.1.0.
First, ensure you have the minimum requirements for DoltLab, which are a Linux host with ports 100, 50051, 4321, and 80 open for ingress.
Next, make sure DoltLab's dependencies are installed and configured. This primarily means Docker, Docker Compose, and amazon-ecr-credential-helper.
Next, on your provisioned host, download DoltLab v2.1.0 and unzip the contents into a doltlab directory:
$ curl -LO https://doltlab-releases.s3.amazonaws.com/linux/amd64/doltlab-v2.1.0.zip
$ unzip doltlab-v2.1.0.zip -d doltlab
Archive: doltlab-v2.1.0.zip
inflating: doltlab/smtp_connection_helper
inflating: doltlab/installer
inflating: doltlab/installation_guide.md
inflating: doltlab/administrators_guide.mdYou're now ready to run the installer:
$ ./installer \
--host=doltlab.dolthub.com \
--smtp-auth-method=plain \
--smtp-host=doltlab.dolthub.com \
--smtp-port=587 \
--no-reply-email=me@email.com \
--default-user-email=me@email.com
2024-03-29T18:19:28.164Z INFO metrics/emitter.go:111 Successfully sent DoltLab usage metrics
2024-03-29T18:19:28.164Z INFO cmd/main.go:345 Successfully configured DoltLab {"version": "v2.1.0"}
2024-03-29T18:19:28.164Z INFO cmd/main.go:349 You can now start DoltLab using the generated script {"start_script": "/home/ubuntu/doltlab/start.sh"}
2024-03-29T18:19:28.164Z INFO cmd/main.go:354 Use the generated stop script to stop DoltLab {"stop_script": "/home/ubuntu/doltlab/stop.sh"}The arguments used in the snippet above are the minimum required arguments to successfully run the installer.
--host, is the host name or IP address of the host running DoltLab. This argument is used in a number of DoltLab's services.
--smtp-auth-method, is your SMTP server's authentication method, one of plain, login, external, anonymous, oauthbearer, and disable. This is used by doltlabapi for sending emails to end users.
--smtp-host, is your SMTP server's host name, and is also used by doltlabapi.
--smtp-port, is your SMTP port, and is also used by doltlabapi.
--no-reply-email, is the email address to use as the "from" address for emails sent by DoltLab. Ensure that your SMTP server allows you to send emails from this address. This value will be used by doltlabapi.
--default-user-email, is the email address to use for DoltLab's default user admin. DoltLab creates a default user automatically in case there are problems with DoltLab's connection to your SMTP server, you can still use DoltLab's features as user admin. This value is used by doltlabapi as well.
If we run ls, we can see that the installer generated the expected files we need to run DoltLab:
$ ls
administrators_guide.md doltlabapi doltlabfileserviceapi envoy.json installer start.sh
docker-compose.yaml doltlabdb doltlabremoteapi installation_guide.md smtp_connection_helper stop.shThe generated files include the docker-compose.yaml file which contains our service definitions, the envoy.json file for configuring DoltLab's reverse proxy, and three key files used for token encryption and decryption for each respective service: doltlabapi/iter_token.keys, doltlabremoteapi/iter_token.keys, and doltlabfileserviceapi/iter_token.keys.
Additionally, the installer generated a script to easily start DoltLab and a script to easily stop it, start.sh and stop.sh.
If we inspect start.sh, we'll see:
#!/bin/bash
set -e
export EMAIL_USERNAME
export EMAIL_PASSWORD
export EMAIL_OAUTH_TOKEN
export DEFAULT_USER_PASSWORD=${DEFAULT_USER_PASSWORD-DoltLab1234}
export DOLT_PASSWORD
export DOLTHUBAPI_PASSWORD
check_env() {
if [[ -z "$DOLT_PASSWORD" || -z "$DOLTHUBAPI_PASSWORD" ]]; then
echo "Must supply DOLT_PASSWORD, and DOLTHUBAPI_PASSWORD"
exit 1
fi
check_plain_auth_env
}
check_plain_auth_env() {
if [[ -z "$EMAIL_USERNAME" || -z "$EMAIL_PASSWORD" ]]; then
echo "Must supply EMAIL_USERNAME and EMAIL_PASSWORD"
exit 1
fi
}
create_docker_network_if_not_exists() {
docker network inspect doltlab >/dev/null 2>&1 || \
docker network create --driver bridge doltlab
}
update_images() {
local config="/home/ubuntu/doltlab/docker-compose.yaml"
docker compose -f "$config" pull
}
start_services() {
local config="/home/ubuntu/doltlab/docker-compose.yaml"
EMAIL_USERNAME="$EMAIL_USERNAME" \
EMAIL_PASSWORD="$EMAIL_PASSWORD" \
EMAIL_OAUTH_TOKEN="$EMAIL_OAUTH_TOKEN" \
DEFAULT_USER_PASSWORD="$DEFAULT_USER_PASSWORD" \
DOLT_PASSWORD="$DOLT_PASSWORD" \
DOLTHUBAPI_PASSWORD="$DOLTHUBAPI_PASSWORD" \
docker compose -f "$config" up -d
}
_main() {
check_env
create_docker_network_if_not_exists
update_images
start_services
}
_main "$@"The contents of this script will change depending on how the installer was run, but you can expect this script to require the environment variables necessary to successfully start and run DoltLab based on the installer configuration.
For example, we ran the installer with --smtp-auth-method=plain. As a result, the generated start.sh script requires EMAIL_USERNAME and EMAIL_PASSWORD to be set in your environment.
We can also see that it's possible to configure the default user of a DoltLab instance, admin, with a specific password. This can be set by supplying DEFAULT_USER_PASSWORD, or if unset, this will default to DoltLab1234.
We are now ready to run ./start.sh and see our DoltLab instance go live.
You should now be able to navigate to http://doltlab.dolthub.com (or whatever your --host was) and see your DoltLab homepage!
You might have noticed that the installer also generated assets in doltlabdb. They are doltlabdb/shell-db.sh and doltlabdb/dolt_db_cli.sh. doltlabdb/dolt_db_cli/sh can be used to connect a Dolt CLI client to doltlabdb, for very rare occasions when the Dolt CLI needs to be used instead of the server itself.
More important is doltlabdb/shell-db.sh which opens a shell to the running doltlabdb server. This can be used to run queries against the server or even whitelist emails to create accounts against your DoltLab instance if you've enabled email whitelisting.
More Installer Power
Hopefully it's clear how much simpler the installer has made configuring something as complex as DoltLab. Here are some more very useful installer options that can help you set up DoltLab for your specific use case.
SMTP Implicit TLS
Some SMTP servers use implicit TLS connections which DoltLab will need to handle. To configure DoltLab's SMTP client to use implicit TLS, run the installer with --smtp-implicit-tls=true. Similarly, to configure DoltLab's SMTP client to skip TLS verification, use --smtp-insecure-tls=true.
Prevent Open Account Creation
DoltLab for non-enterprise use currently supports explicit email whitelisting to prevent account creation by unauthorized users. To enable email whitelisting (disallowing open account creation) use --white-list-all-users=false with the installer. To whitelist an email and allow it to create an account on your DoltLab instance use the doltlabdb/shell-db.sh script to run the following query:
$ DOLT_PASSWORD=******** ./shell-db.sh
...
mysql> INSERT INTO email_whitelist_elements (email_address, updated_at, created_at) VALUES ('example@address.com', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);A user with email example@address.com can now create an account on your DoltLab instance.
Connect DoltLab to an External Dolt Server
DoltLab runs doltlabdb on the same host as the application database on the same host as the other services by default, but DoltLab can also use an external Dolt server as it's application database.
Ensure you run the following statements against your external Dolt server.
CREATE USER 'dolthubadmin' IDENTIFIED BY '$DOLT_PASSWORD';
CREATE USER 'dolthubapi' IDENTIFIED BY '$DOLTHUBAPI_PASSWORD';
CREATE DATABASE dolthubapi;
GRANT ALL ON *.* TO 'dolthubadmin';
GRANT ALL ON dolthubapi.* TO 'dolthubapi';Then, run the installer with --doltlabdb-host referencing your external database's host name, and --doltlabdb-port referencing its port.
Serve DoltLab natively over HTTPS
To do so, make sure that port 443 is open on the host running DoltLab (as well as the other required ports 100, 4321, and 50051) and that you have a valid TLS certificate that uses the --host value you supplied to the installer. We recommend creating a TLS certificate using certbot.
You can then run the installer with --https=true, --tls-cert-chain=/path/to/tls/certificate/chain, and --tls-private-key=/path/to/tls/private/key and use the start.sh script to run DoltLab over HTTPS.
DoltLab Enterprise and Beyond
A major motivation for creating DoltLab's installer tool was to better support our DoltLab Enterprise clients, and make it simpler for them to configure various Enterprise features.
Previous versions of DoltLab required Enterprise feature configuration with a separate file called admin-config.yaml. Thankfully we've done away with this additional file, and can simply use the installer to enable Enterprise features.
For those not yet aware, we offer an Enterprise version of DoltLab the has exclusive features and support, not included with the free version of DoltLab. This includes, but is not limited to: UI customization options including use of custom colors, logo, and email templates, single-sign-on integration with your identity-provider, and 24/7 engineering support, and more coming soon! If you're interested in finding out more about DoltLab Enterprise, contact us on Discord for more information.
If you're already a DoltLab Enterprise customer, using the installer to unlock your exclusive features very simple.
In addition to supplying the minimum required installer arguments, simply include the following arguments to configure DoltLab to run in Enterprise mode:
$ ./installer \
...
--enterprise-online-product-code=******** \
--enterprise-online-shared-key=************** \
--enterprise-online-api-key=************ \
--enterprise-online-license-key=********** \
...All Enterprise features will require these arguments. With these supplied to the installer, here's how you configure the following features:
Custom Emails
DoltLab Enterprise allows you to customize the emails sent to users by your DoltLab instance. You can do this by editing email templates that DoltLab uses for these automated emails.
Using the installer supply the argument --custom-email-templates=true. The installer will generate template files you can edit at doltlabapi/templates/email. For more information about how to edit these files, check out our Custom emails section in DoltLab's Administrator guide.
Custom Logo
You can also use a custom logo on your DoltLab instance, which helps ensure users of your DoltLab instance know it's yours. To use a custom logo, supply --custom-logo=/path/to/custom/logo.png to the installer.
Super Admins
Its often the case that DoltLab administrators need super user permissions on their DoltLab instance so they can add or remove users, modify various databases, and perform various actions to meet their use case. We call these users "super admins", and they can be configured with the installer by supplying the argument --super-admin-email with the email address associated with the user who should have super admin privileges.
This argument can also be supplied multiple times, ie:
$ ./installer \
...
--super-admin-email=yours@email.com \
--super-admin-email=mine@email.com \
--super-admin-email=ours@email.com \
...SAML Single Sign-on
To configure SAML single sign-on (SSO) with DoltLab, there are a few prerequisite steps you'll need to take in order to ensure you're ready to integrate your DoltLab instance with your identity provider. For those we have a section in our documentation that walks you through what to do.
Once you have the SAML metadata from your identity provider downloaded, you can configure your instance to use SSO by using the --sso-saml-metadata-descriptor=/path/to/downloaded/metadata/descriptor argument and the --sso-saml-cert-common-name=mydoltlabinstance argument with the installer. The --sso-saml-cert-common-name is used during the SAML certificate generation of your DoltLab instance.
Automated Backups
DoltLab Enterprise also enables automated backups of the application database doltlabdb to various cloud providers. Using the power of dolt backup under-the-hood, automated backups can be configured to write to an AWS, GCS, or OCI remote. For detailed information about automated DoltLab backups, see the section in DoltLab's Administrator guide.
In brief, AWS backups can be configured with the installer with the arguments:
$ ./installer \
...
--automated-dolt-backups-url="aws://[test-doltlab-backup-application-db-manifest:test-doltlab-application-db-backups]/my_doltlab_backup" \
--aws-shared-credentials-file="/absolute/path/to/aws/credentials" \
--aws-config-file="/absolute/path/to/aws/config" \
--aws-region="aws-region" \
--aws-profile="doltlab_backuper" \
...GCS backups can be configured with:
$ ./installer \
...
--automated-dolt-backups-url="gs://test-doltlab-application-db-backup/my_doltlab_backup" \
--google-creds-file="/absolute/path/to/gcloud/credentials" \
...And OCI backups can be configured with:
$ ./installer \
...
--automated-dolt-backups-url="oci://test-doltlab-application-db-backup/my_doltlab_backup" \
--oci-config-file="/absolute/path/to/oci/config" \
--oci-key-file="/absolute/path/to/oci/private/key.pem" \
...Automated backups will be pushed by default every 24 hours, but this schedule can be augmented by supplying --automated-dolt-backups-cron-schedule with a cron string, like "0 * * * *", to push backups every hour, for example. DoltLab will also email the administrator if a backup fails for some reason.
Conclusion
Hopefully our work on the latest release of DoltLab and its new installer interface has encouraged you to download and try DoltLab for yourself. We're always working to make the experience of running DoltLab simple and straightforward and would love to hear your feedback.
We are also offering free, 30-day trials of DoltLab Enterprise so you can demo this powerful product for the rest of your team. To find out more information hit us up on Discord in the #doltlab server.
Not sure which product is right for you? You can check out each of our different product offerings below:
- Dolt—it's Git for data.
- DoltHub—it's GitHub for data.
- DoltLab—it's GitLab for data.
- Hosted Dolt—it's RDS for Dolt databases.