DoltHub is a web-based UI built in React to share, discover, and collaborate on Dolt databases. We've recently been adding more features to make it easier to add and change data in your Dolt database on the web. One of these features is file upload, where you can upload an existing file or use our built-in spreadsheet editor to create new tables or update existing ones.

We recently changed our file upload process so that each of the five steps is a separate page. This required moving a lot of the state from that process to client-side storage, so that the state persists in the browser as the pages change. This blog will cover how we used client-side storage with React to achieve this change.

Client-side storage

Client-side storage is a way to store data on a user's computer to be retrieved when needed. This has many use cases and is commonly used in addition to server-side storage (storing data on the server using some kind of database). Client-side storage is great for passing temporary information along in a process like file upload, which requires navigating through multiple pages before storing it in the server.

There are a few different ways you can store data on the client, and we will cover both web storage and IndexedDB.

Cookies

Before we dive into some examples of how we use client-side storage on DoltHub, it's worth mentioning the OG method of client-side storage: cookies 🍪. Cookies are commonly used for storing user information to improve the user's experience across websites. While cookies are great for certain use cases, they are less convenient in others because they must be generated by a web server and they have smaller storage limits.

Web Storage API

The web storage API stores data as simple name/value pairs that can be fetched as needed. This is ideal for storing simple, smaller data like strings and numbers. There are two mechanisms within web storage: localStorage and sessionStorage. Data saved to sessionStorage persists while a page session is active (i.e. for as long as a browser is open), while data saved to localStorage persists across page sessions (i.e. even after the browser is closed). The storage limit for these is larger than for a cookie, but maxes out at around 5MB.

An example: pull request comments

Similar to GitHub, pull requests on DoltHub consist of two pages: the details page with a description, commits, and comments and the diff page with the schema and data changes. As a reviewer, you're commonly clicking between both pages to give feedback on changes. We ran into an issue on DoltHub where sometimes a reviewer will start typing out a comment and navigate to the diff, and then when they go back to the comment it's gone. This is a great use case for sessionStorage.

We don't want our users to lose their pull request comment if they navigate away from that page. To prevent this, we can store the comment in sessionStorage before pushing the comment through to our server to be stored in our database.

Initially we stored the comment state in a useState hook, and the comment was updated on changes to the comment input. We first need to create a new hook for getting and updating state from session storage:

// hooks/useStateWithSessionStorage.ts

import { Dispatch, SetStateAction, useEffect, useState } from "react";

type ReturnType = [string, Dispatch<SetStateAction<string>>];

export default function useStateWithSessionStorage(
  storageKey: string // Needs to be unique
): ReturnType {
  const [value, setValue] = useState(sessionStorage.getItem(storageKey) ?? "");

  useEffect(() => {
    sessionStorage.setItem(storageKey, value);
  }, [value, storageKey]);

  return [value, setValue];
}

Then we can replace our useState hook with our new useStateWithSessionStorage hook in our comment form component:

// components/CommentForm.tsx

import React, { SyntheticEvent } from "react";
import useStateWithSessionStorage from "../hooks/useStateWithSessionStorage";
import { PullParams } from "../lib/params";

type Props = {
  params: PullParams;
  currentUser: CurrentUserFragment;
};

export default function CommentForm({ params, currentUser }: Props) {
  const [comment, setComment] = useStateWithSessionStorage(
    `comment-${params.ownerName}-${params.repoName}-${params.parentId}`
  );

  const onSubmit = (e: SyntheticEvent) => {
    // Create pull comment
  };

  return (
    <form onSubmit={onSubmit}>
      <img src={currentUser.profPic.url} alt="" />
      <textarea
        rows={4}
        value={comment}
        onChange={(e) => setComment(e.target.value)}
        placeholder="Leave a comment"
      />
    </form>
  );
}

So now when we start typing text in our pull comment form input:

And then navigate away from that page before submitting the comment to view the diff, we can still see the comment stored in our session storage in the developer tools:

Then when we navigate back to the main pull request page the pull comment will be retrieved from session storage and populated in the input.

IndexedDB API

Web storage is great for storing small-sized strings and numbers. However, sometimes you need to store larger data of more complex types.

Our file upload process has fives steps, split into pages, that aggregates data in many different forms along the way. The first step looks like this:

It starts with a branch (1. Branch), then a table name and import operation enum (2. Table), and then starts to get more complicated with storing the content of an uploaded file (3. Upload). We send this information to our server in this third upload step, and in the next (4. Review) we show a diff of the resulting changes from the upload with the option to update the schema. At the very end (5. Commit) you can create a commit with the changes from the upload. You can read more about the specifics of file upload on DoltHub here.

Web storage does not work for this process due to size constraints and needing to store more complex object and array types. IndexedDB (IDB) seemed like it could be a good solution for file upload. IDB is a client-side storage API for larger amounts of structured data. Data can be retrieved asynchronously so it does not block other processes. The one downside is that it provides an extra layer of complexity that you don't have to deal with when using web storage.

Deciding to use an outside library

As I continued to research, I found a library called localForage in the MDN documentation for IDB. localForage is a library that uses IDB to provide an asynchronous data store with a simpler, localStorage-like API. It also provides fallback stores (WebSQL and localStorage) for browsers that don't support IDB.

There are a few things that weigh into our decision for including an outside library in our React application. Having too many dependencies can be difficult to manage (learn more about why here), so we need to be intentional with these kinds of decisions. These are some questions we ask ourselves:

1. Is there a way to achieve what I want without this library? Is the time saved by adding another dependency worth the future technical debt of maintaining it?
2. Is this package actively maintained by the owners? Is it being used by other companies and individuals?
3. Is the code open source? How quickly do they respond to and address issues and questions?
4. Do they have useful and up-to-date documentation?
5. Are there any obvious security issues that could be introduced by using this package?

We ultimately decided having an easy to use client-side storage solution for both the file upload process and future features was worth the tradeoff. Local forage is actively maintained and used by almost 200k users and organizations on GitHub. While we could have made IndexedDB work without it, the simplicity of the local forage API would make it both easier to implement and maintain as we have more developers working on DoltHub.

There weren't a ton of resources for adding localForage to a React application specifically, so I thought it could be worth writing this blog about how we made it work.

An example: file upload

In our V1 of file upload, we used one page to navigate through the five steps of aggregating information. While this worked, a few people had requested changing each step to a separate page. Not only does this make the code for each page a little simpler and separated, but it also improves the user experience by letting them use the browser's next and back buttons without losing their work.

We initially passed around state within the file upload components through a React context with a useSetState hook. Because we store about 15 pieces of state to make the upload successful, storing one object within useSetState was a better option than having 15 separate useState hooks. The hardest part of replacing the original logic with storing the state using local forage was maintaining our use of useSetState while asynchronously storing and retrieving partial objects.

First, we created a context that will wrap our parent file upload page component:

// pageComponents/FileUploadPage/fileUploadLocalForageContext.tsx

export const FileUploadLocalForageContext =
  createContext<FileUploadLocalForageContextType>(undefined as any);

export function FileUploadLocalForageProvider(props: Props) {
  const [state, setState] = useState<FileUploadState>(defaultState);
  return (
    <FileUploadLocalForageContext.Provider value={{}}>
      {props.children}
    </FileUploadLocalForageContext.Provider>
  );
}

Next, we need to set up a store with a unique identifier for our local forage instance:

const name = `upload-${props.params.ownerName}-${props.params.repoName}-${props.params.uploadId}`;
const store = localForage.createInstance({ name });

Since all our calls to local forage will be asynchronous, we also added loading and error states:

const [loading, setLoading] = useState(false);
const [error, setError] = useState<Error>();

Then we need a way to store new items in local forage, so we add a function that will handle this:

function setItem<T>(key: keyof FileUploadState, value: T) {
  setLoading(true);
  setState({ [key]: value });
  store
    .setItem(key, value)
    .catch((err) => {
      setError(err);
    })
    .finally(() => {
      setLoading(false);
    });
}

Finally, we need a way to retrieve our stored state from local forage. We used a plugin called localforage-getitems to help and added a useEffect hook that gets the state on mount:

import localForage from "localforage";
import { extendPrototype } from "localforage-getitems";

// Allows usage of `store.getItems()`
extendPrototype(localForage);

useEffect(async () => {
  setLoading(true);
  const data = { subscribed: true };

  try {
    const res = await store.getItems();
    if (data.subscribed) {
      setState({ ...defaultState, ...res });
    }
  } catch (err) {
    if (data.subscribed) setError(err);
  } finally {
    if (data.subscribed) setLoading(false);
  }

  return () => {
    data.subscribed = false;
  };
}, []);

We can now access our local forage state from child components using useContext. Every time we get to a new page in the process, we wait for the local forage state to load, check for errors, and then render the form for the current step. Each step adds new items to local forage, and the process continues at the next step. You can see the local forage storage in your developer tools at every stage:

Once the process is completed or the user navigates away from the file upload process, the current local forage store is cleared using store.dropInstance({ name }) so that you start fresh with new state every time.

Conclusion

Using client-side storage to store temporary data on your website can be beneficial for certain use cases, like storing small strings for comment inputs and storing more complicated state for a multi-page process like file upload. If you have any questions or feedback join us on Discord and find me in the #dolthub channel.