Almost a month ago, we announced that Private Deployments were on their way. With the final development and testing completed, we're excited to announce that Private Deployments are now available for all Hosted Dolt users. In this short blog post, I'll quickly walk through what Private Deployments are, why you might want one, and how to create one and get started using one.

Hosted Dolt Private Deployments

A Private Deployment is a Hosted Dolt Deployment which is only accessible through AWS PrivateLink. Only an authorized list of AWS accounts can create VPC Endpoints which connect to the endpoint services published by the Private Deployment. In this way, customers can launch Hosted Deployments which are only acceptable from their AWS infrastructure and not from the public internet.

For now, Private Deployments only support Dolt deployments, not MySQL deployments with Dolt replicas. Private Deployments are also only supported on AWS, not on GCP.

How to Create a Private Deployment

Creating a Private Deployment on Hosted Dolt looks almost exactly like creating a normal one. From the create deployment page, give your new deployment a name and be certain to choose a cluster type of "Dolt."

On the next page, leave the cloud provider as AWS and choose a zone which is in the same region as your AWS VPC. Your private deployment will be accessible from any zones within the region, but the VPC Endpoints which you will create to connect to your Private Deployment must be in subnets within subnets that are in the same region as your Private Deployment. After choosing your instance type, storage type and volume size, click Next.

On the next page, there is a check box for "Private deployment". If you select it, you will be prompted for a list of AWS account IDs which will be authorized to connect to the service. Go ahead and enter at least one AWS account ID which you would like to be authorized. You can change this list of Account IDs in the future, if you need to add new accounts, for example.

Click next to review your options and click "Create deployment" if everything looks right.

Private Deployments can also be created for a Trial instance. After choosing the deployment name and clicking "Next", you will be taken directly to the "Confirmation" page. Simply click "Edit" next to the "Advanced" section to go to the page where you can choose the "Private deployment" option and enter the authorized account IDs.

After creating your deployment, it will enter a Starting phase. Private Deployments take quite a bit longer to launch than regular deployments, because some of the VPC network infrastructure which is provisioned as a part of their creation takes a bit of time to create and become available. You can expect the launch to take up to 20 minutes. After the network infrastructure is created, the instances themselves will be launched. At this point, your private deployment is fully available and you can access it through the SQL workbench and interact with it through the Hosted Dolt website. To access it from your VPC, you will need to a bit of work.

How to Connect to Your Private Deployment

To access your Private Deployment from your VPC, you need to create VPC Endpoints for accessing it. Each Private Deployment creates two VPC Endpoint Services whose names are listed on the deployment details page. The primary endpoint service will always have the current primary of the cluster behind it. This is the only endpoint that should receive write traffic from your application. The cluster endpoint service will always have every deployed instance in the cluster behind it. It should receive read-only traffic if you're attempting to scale reads for your application. Because deployments in Hosted Dolt can always add replicas, we create the cluster endpoint services even for deployments that currently have zero replicas.

The first thing you will need to do to connect to your Private Deployment from your VPC is to make certain that the account IDs you gave when you created the deployment include the AWS account ID in which the VPC is located. If the AWS account ID is not currently included, feel free to add it on the database page and click "Update" for your changes to take effect.

Then, make certain you have the necessary prerequisites for creating the VPC Endpoint. You need the following:

1. Your VPC should have "DNS hostnames" and "DNS resolution" both Enabled. If this is not already the case, you can change these settings under "Edit VPC Settings" in the AWS console, or you can use AWS CLI with something like aws ec2 modify-vpc-attribute --vpc-id [VPC_ID] --enable-dns-support && aws ec2 modify-vpc-attribute --vpc-id [VPC_ID] --enable-dns-hostnames.

2. Your VPC should have an appropriate security group which you can use for the VPC Endpoints you create. This security group should allow ingress on port 3306, the MySQL port, from any instances which are supposed to access it. If you want to use your Hosted SQL instance as a Dolt remote for clone and pull, it should also allow ingress on port 443. You can attach multiple security groups to the VPC Endpoints if you need to.

3. Your VPC should have at least one subnet in which the VPC Endpoint will create network interfaces. You can give one subnet per zone to the VPC Endpoint. Giving your VPC Endpoint endpoint multiple zones within the region can make access to your endpoint more resilient and cheaper.

If you have all of the above, the only thing left to do is to create the VPC Endpoint. In the section for "Private Link Information" on your Deployment's Database page, you copy the Service Name for which you want to create the Endpoint. Then you create the endpoint through whatever mechanism you want: AWS CLI, AWS Console, Terraform, etc. Using AWS CLI, creating the endpoint looks like:

aws ec2 create-vpc-endpoint \
  --vpc-endpoint-type Interface \
  --vpc-id [VPC_ID] \
  --service-name [SERVICE_NAME] \
  --security-group-ids [SECURITY_GROUP_ID_1] [SECURITY_GROUP_ID_2] ... \
  --subnet-ids [SUBNET_ID_1] [SUBNET_ID_2] ...

The endpoint will start a Pending state. It should quickly transition to Available. From that point forward, the DNS names associated with the endpoint should resolve within the VPC and you will be able to connect to the database through them. The DNS name which Hosted Dolt manages is listed on the deployment database details page, as "Primary Host" and "Cluster Host" for the two different endpoint services. You can take the connect strings or the MySQL client invocations and use those as well from any host that has access to the newly created VPC Endpoint and the VPC's DNS resolution services, for example, from an EC2 instance which is within the VPC.

If you destroy a Private Deployment then any VPC Endpoints currently attached to its endpoint services will transition to a "Rejected" status. You should delete these from your AWS account, since they are no longer functional or necessary.

Troubleshooting

If you are having any issues with a newly created or existing Private Deployment, please don't hesitate to reach out to us. The two most common issues we have seen so far are:

1. Attempting to create the VPC Endpoint results in InvalidServiceName.

If the service name provided to AWS was correct, then this indicates that the AWS Account does not currently have access to this Private Deployment. Try updating the list of authorized AWS Account IDs on the Deployment Database page under the Private Link Information section.

2. The VPC Endpoint is Available but attempts to connect to the host hang and eventually fail.

This typically means there is a problem with the security group and authorizing the traffic to port 3306 from the instance which is attempting to connect to the VPC Endpoint itself. The VPC Endpoint must access the 3306 traffic as ingress. The instance that is attempting the connection must allow the 3306 traffic as egress.

Conclusion

With Private Deployments, Hosted Dolt can create deployments which are only accessible from within your VPC. Private Deployments might be appropriate for you due to regulatory requirements, security requirements, your existing networking topology and infrastructure or simply because you're more comfortable with leaving your private infrastructure off the public internet. Private Deployments have all of the features of Hosted Dolt, including clustering support, support for remotes access, and the SQL workbench. If you're interested in Private Deployments or have any questions about them, please don't hesitate to contact us or reach out to our team on Discord.