How to Store Unlimited* Data in the Browser with IndexedDB

This article explains the fundamentals of storing data in the browser using the IndexedDB API, which offers a far greater capacity than other client-side mechanisms.

Storing web app data used to be an easy decision. There was no alternative other than sending it to the server, which updated a database. Today, there’s a range of options, and data can be stored on the client.

Why Store Data in the Browser?

It’s practical to store most user-generated data on the server, but there are exceptions:

• device-specific settings such as UI options, light/dark mode, etc.
• short-lived data, such as capturing a range of photographs before choosing one to upload
• offline data for later synchronization, perhaps in areas with limited connectivity
• progressive web apps (PWAs) which operate offline for practical or privacy reasons
• caching assets for improved performance

Three primary browser APIs may be suitable:

1. Web Storage

   Simple synchronous name-value pair storage during or beyond the current session. It’s practical for smaller, less vital data such as user interface preferences. Browsers permit 5MB of Web Storage per domain.

2. Cache API

   Storage for HTTP request and response object pairs. The API is typically used by service workers to cache network responses, so a progressive web app can perform faster and work offline. Browsers vary, but Safari on iOS allocates 50MB.

3. IndexedDB

   A client-side, NoSQL database which can store data, files, and blobs. Browsers vary, but at least 1GB should be available per domain, and it can reach up to 60% of the remaining disk space.

OK, I lied. IndexedDB doesn’t offer unlimited storage, but it’s far less limiting than the other options. It’s the only choice for larger client-side datasets.

IndexedDB Introduction

IndexedDB first appeared in browsers during 2011. The API became a W3C standard in January 2015, and was superseded by API 2.0 in January 2018. API 3.0 is in progress. As such, IndexedDB has good browser support and is available in standard scripts and Web Workers. Masochistic developers can even try it in IE10.

This article references the following database and IndexedDB terms:

• database: the top-level store. Any number of IndexedDB databases can be created, although most apps will define one. Database access is restricted to pages within the same domain; even sub-domains are excluded. Example: you could create a notebook database for your note-taking application.

• object store: a name/value store for related data items, conceptually similar to collections in MongoDB or tables in SQL databases. Your notebook database could have a note object store to hold records, each with an ID, title, body, date, and an array of tags.

• key: a unique name used to reference every record (value) in an object store. It can be automatically generated or set to a value within the record. The ID is ideal to use as the note store’s key.

• autoIncrement: a defined key can have its value auto-incremented every time a record is added to a store.

• index: tells the database how to organize data in an object store. An index must be created to search using that data item as criteria. For example, note dates can be indexed in chronological order so it’s possible to locate notes during a specific period.

• schema: the definition of object stores, keys, and indexes within the database.

• version: a version number (integer) assigned to a schema so a database can be updated when necessary.

• operation: a database activity such as creating, reading, updating, or deleting (CRUD) a record.

• transaction: a wrapper around one or more operations which guarantees data integrity. The database will either run all operations in the transaction or none of them: it won’t run some and fail others.

• cursor: a way to iterate over many records without having to load all into memory at once.

• asynchronous execution: IndexedDB operations run asynchronously. When an operation is started, such as fetching all notes, that activity runs in the background and other JavaScript code continues to run. A function is called when the results are ready.

The examples below store note records — such as the following — in a note object store within a database named notebook:

{ id: 1, title: "My first note", body: "A note about something", date: <Date() object>, tags: ["#first", "#note"] } 

The IndexedDB API is a little dated and relies on events and callbacks. It doesn’t directly support ES6 syntactical loveliness such as Promises and async/await. Wrapper libraries such as idb are available, but this tutorial goes down to the metal.

IndexDB DevTools Debugging

I’m sure your code is perfect, but I make a lot of mistakes. Even the short snippets in this article were refactored many times and I trashed several IndexedDB databases along the way. Browser DevTools were invaluable.

All Chrome-based browsers offer an Application tab where you can examine the storage space, artificially limit the capacity, and wipe all data:

The IndexedDB entry in the Storage tree allows you to examine, update, and delete object stores, indexes, and individual record:

(Firefox has a similar panel named Storage.)

Alternatively, you can run your application in incognito mode so all data is deleted when you close the browser window.

Check for IndexedDB Support

window.indexedDB evaluates true when a browser supports IndexedDB:

if ('indexedDB' in window) { // indexedDB supported } else { console.log('IndexedDB is not supported.'); } 

It’s rare to encounter a browser without IndexedDB support. An app could fall back to slower, server-based storage, but most will suggest the user upgrade their decade-old application!

Check Remaining Storage Space

The Promise-based StorageManager API provides an estimate of space remaining for the current domain:

(async () => { if (!navigator.storage) return; const required = 10, // 10 MB required estimate = await navigator.storage.estimate(), // calculate remaining storage in MB available = Math.floor((estimate.quota - estimate.usage) / 1024 / 1024); if (available >= required) { console.log('Storage is available'); // ...call functions to initialize IndexedDB } })(); 

This API is not supported in IE or Safari (yet), so be wary when navigator.storage can’t returns a falsy value.

Free space approaching 1,000 megabytes is normally available unless the device’s drive is running low. Safari may prompt the user to agree to more, although PWAs are allocated 1GB regardless.

As usage limits are reached, an app could choose to:

• remove older temporary data
• ask the user to delete unnecessary records, or
• transfer less-used information to the server (for truly unlimited storage!)

Open an IndexedDB Connection

An IndexedDB connection is initialized with indexedDB.open(). It is passed:

• the name of the database, and
• an optional version integer

const dbOpen = indexedDB.open('notebook', 1); 

This code can run in any initialization block or function, typically after you’ve checked for IndexedDB support.

When this database is first encountered, all object stores and indexes must be created. An onupgradeneeded event handler function gets the database connection object (dbOpen.result) and runs methods such as createObjectStore() as necessary:

dbOpen.onupgradeneeded = event => { console.log(`upgrading database from ${ event.oldVersion } to ${ event.newVersion }...`); const db = dbOpen.result; switch( event.oldVersion ) { case 0: { const note = db.createObjectStore( 'note', { keyPath: 'id', autoIncrement: true } ); note.createIndex('dateIdx', 'date', { unique: false }); note.createIndex('tagsIdx', 'tags', { unique: false, multiEntry: true }); } } }; 

This example creates a new object store named note. An (optional) second argument states that the id value within each record can be used as the store’s key and it can be auto-incremented whenever a new record is added.

The createIndex() method defines two new indexes for the object store:

1. dateIdx on the date in each record
2. tagsIdx on the tags array in each record (a multiEntry index which expands individual array items into an index)

There’s a possibility we could have two notes with the same dates or tags, so unique is set to false.

Note: this switch statement seems a little strange and unnecessary, but it will become useful when upgrading the schema.

An onerror handler reports any database connectivity errors:

dbOpen.onerror = err => { console.error(`indexedDB error: ${ err.errorCode }`); }; 

Finally, an onsuccess handler runs when the connection is established. The connection (dbOpen.result) is used for all further database operations so it can either be defined as a global variable or passed to other functions (such as main(), shown below):

dbOpen.onsuccess = () => { const db = dbOpen.result; // use IndexedDB connection throughout application // perhaps by passing it to another function, e.g. // main( db ); }; 

Continue reading How to Store Unlimited* Data in the Browser with IndexedDB on SitePoint.