Google Drive datasets

A Google Drive dataset reads its recordings from a Drive folder. Nothing is downloaded when you add it. Sift fetches audio when someone listens, and keeps it in a download folder on that person’s computer, under a size limit they choose.

Each member reads the folder with their own Google account, on their own computer. The hub never holds Google sign-ins and never talks to Google.

flowchart LR
  F["Drive folder<br/>(shared with each member)"] --> L["Sift lists its WAV and FLAC files<br/>and reads their headers"]
  L --> O["You open a recording"]
  O --> W["Sift fetches the stretch you need,<br/>or the whole recording"]
  W --> C["Download folder on this computer<br/>(size limit you choose)"]
  K["Make available offline"] --> P["Whole recordings, kept<br/>and never removed for space"]
  P --> C

Before you start

• Your copy of Sift must be built with Google sign-in. If the Google Drive folder option says “Not set up in this build”, ask whoever builds Sift for your organisation. See Google Drive: one-time setup.
• The folder must be shared with your Google account. Viewer is enough.
• The folder holds .wav or .flac files, directly or in subfolders.

Add a Google Drive dataset

1. On Datasets, click Add dataset, then choose Google Drive folder.
2. Under Folder, paste the Folder link. In Google Drive, use the folder’s ⋮ menu, then Share, then Copy link. Include subfolders is on. Untick it to read only the files directly in the folder. Click Check.
3. Under Google account, choose the account the folder is shared with. If you have not signed in yet, click Sign in to Google. Your browser opens Google’s sign-in page. Each account shows “Can open it”, “Can’t open it” or “Can see it; downloads are off”.
4. Click Look inside. The preview shows the number of recordings, an estimate of the hours, the sample rate, channels and size on Drive. Only file headers are read.
5. Read the Checks: different sample rates or channel counts, FLAC files without a seek table (each is downloaded whole the first time it is opened), and files that cannot be read (they are left out).
6. If the files have more than one channel, the Channels section is required. Choose which channel holds the signal. See Channels and the channel map.
7. Click Continue, choose the labels, then fill in the details. Under On this computer, choose where downloads go if this computer has no download folder yet. You can also add Folders with copies of the files: a disk that already holds the same files, so they are read from disk and never downloaded.
8. Click Create followed by the dataset’s name (for example Create saltmarsh_drive).

What downloads, and when

• Adding or opening the dataset reads Drive’s file list and each new file’s header. No audio.
• Events and playback fetch only the stretch around what you are listening to.
• Recording (whole recordings), applying a model, and Make available offline download whole recordings.
• By default only the dataset’s channel is kept. A download is checked against Drive’s checksum.

Keeping the list up to date

When you open a Drive dataset, Sift lists the folder again if the last listing is more than 10 minutes old. While the dataset is open, it checks for changes every minute and lists the whole folder every 30 minutes. Nothing runs for datasets that are not open.

New files are added as their headers are read, about 500 a minute while the dataset is open. If you can only view the dataset, Sift tells you how many new files are not in it yet. Someone who can label adds them when their copy of Sift refreshes.

To list the folder now, click Refresh from Drive on the Recordings tab. The line next to it says when this computer last read the folder (“Refreshed 5 minutes ago”, or “Not read from Drive on this computer yet”), and what the last refresh found. If a colleague’s app listed it more recently, it adds “listed by a colleague” and when. If you can only view the dataset, it adds “New files are added when someone who labels it opens it”.

When a Drive dataset waits

The hub never reads Google Drive itself: each person’s app lists the folder with their own Google account. So in an organisation, a Drive dataset can wait for someone. The organisation’s Datasets page says what it waits for:

• “Waiting for a manager to open it in the app”, with the managers’ names, and “Until then it’s empty.” The dataset was made in the hub web app, and none of its managers has opened it in the desktop app yet. The first manager who does lists the folder and sets the dataset up for everyone. Ask one of the managers named.
• “New Google Drive files are added when someone who labels it opens it.”, with “Last listed” and when, or “Not listed yet”. New files in the folder reach the dataset only when someone who can label it opens it in the app. If you can only view it, ask someone who labels it to open it.

Status on this computer

On a Drive dataset, the Recordings tab’s This computer column shows each recording’s status:

Status	Meaning
Not downloaded	On Google Drive. Fetched when it is opened.
Partial (N %)	Stretches were fetched for listening. The rest is on Drive.
Downloaded (date)	The recording’s channel is on this computer.
Local copy	Read from a file on your own disk with the same bytes as Drive’s.
Changed on Drive	Changed on Drive since it was downloaded. The copy here is out of date and is not used.
Downloading N %, Queued, Download failed	A download in progress, waiting, or stopped.

Kept offline marks recordings you made available offline.

Next to these, Drive’s own state can show Missing on Drive, In Drive’s trash or Moved out of the folder. Sift never deletes such recordings. Their boxes and marks stay, and what is downloaded still plays. Put the file back and it returns on the next refresh.

A replaced tag means the file was replaced on Drive after it was added, so its boxes were drawn on other audio. They are kept, and a manager decides.

The Show menu filters by these states: On this computer, Kept offline, Partly fetched, Not downloaded, Changed on Drive, No longer in the Drive folder.

Work offline

1. On the Recordings tab, select recordings.
2. Click Make available offline.

They are downloaded whole and kept. Space limits never remove them. To undo, choose Don’t keep offline in a recording’s ⋯ menu, or Stop keeping in Storage settings.

Remove from this computer deletes what was downloaded for the selected recordings. They stay on Drive with their boxes and marks. Your own local copies are never touched.

The dataset’s Drive settings

The dataset’s Settings tab, under Audio, shows the Drive folder:

• which account reads it on this computer, with Switch account if you have several,
• Refresh from Drive and Open in Google Drive,
• the space this dataset’s downloads take, with Remove downloads,
• Keep all channels of downloaded recordings: more space, but no new download if the channel map changes,
• Folders with local copies: click Add a folder with local copies…. Sift compares files by size and checksum. A file named like a recording but with different bytes is reported and not used.

Google accounts

Open them from the sidebar’s account menu: Google accounts….

• Sign in to Google, or Add another account, opens your browser.
• Remove revokes this computer’s access for that account. Downloads stay.

Sign-ins stay in this computer’s keychain. They never reach your organisation.

The first time, Google may warn that the app is unverified. If your organisation told you to expect this, choose Advanced, then go on to the app. Allow See and download all your Google Drive files.

Fix access problems

“Sign in to Google to open this dataset’s folder”: click Sign in to Google and use the account the folder is shared with.

“Sign in to Google again”: Google ended this computer’s sign-in. Click Sign in again.

“… can’t open this folder”: ask whoever shares the folder to add your account (Viewer is enough), or sign in with an account it is already shared with. Google does not say whether a folder exists to accounts that cannot see it, so check the link too. You can also add a folder with local copies.

“Downloads are turned off for this folder”: the folder’s owner turned off downloading for viewers. Ask them to allow it in the folder’s sharing settings.

“Offline, and this recording isn’t on this computer”: connect, or next time make the recordings available offline first.

“Google Drive isn’t set up in this build”: this copy of Sift has no Google sign-in. Ask whoever builds Sift for your organisation.

In every case the recordings, boxes and marks stay, and what is downloaded still plays.