Organisations, the hub and sync

Sift works on its own on one computer. To label with other people, you join an organisation. Each organisation has a hub: a small server that keeps everyone’s apps in step. The hub web app is the hub’s website, where admins manage people and anyone can see projects, requests and activity in a browser.

Members, roles, assignments, requests, disputes between people, “Here now” and “Working now” all need an organisation. A project on this computer only has none of them.

What the hub holds

The hub holds the organisation’s people, its projects and their members, every change made in its projects, and the published versions of its datasets. It also hands out claims, carries requests and shows who is working now.

The hub never receives audio. Each person’s app reads recordings from that person’s own drives, shares or Google Drive. A dataset’s recordings are matched across computers by their audio content, so your copy is found even if it sits in a different folder or on a drive with a different name.

flowchart LR
  subgraph A["Your computer"]
    AppA["Sift app<br/>engine and labeller"]
    AudA[("Audio files<br/>on this computer")]
    AppA --- AudA
  end
  subgraph B["A colleague's computer"]
    AppB["Sift app<br/>engine and labeller"]
    AudB[("Their copy<br/>of the audio")]
    AppB --- AudB
  end
  Hub["Hub<br/>people, projects, changes,<br/>versions, claims, requests,<br/>who is here now"]
  Web["Hub web app<br/>in a browser"]
  AppA <-->|"changes, no audio"| Hub
  AppB <-->|"changes, no audio"| Hub
  Web --> Hub

Every change is an op

Each thing you do in an organisation project is recorded as a small change, called an op: an answer, a box you draw or move, a noise rating, a note, a mark on a recording, a change to a label set. Each op carries a timestamp.

The hub keeps every op and passes it on to everyone else who can see that dataset. It never merges or edits them. Every app applies the same ops in the same way, so everyone ends up with the same state.

People’s answers are never overwritten by each other. Each person has their own answer on each event. When answers differ, the event shows as disputed and every answer stays side by side. See Disputes and settling them.

Every op is also signed by the computer it was made on. The hub keeps each computer’s public key, so any member’s app can carry someone else’s ops to the hub, for example from a site with no network, and the hub still records them as that person’s. See Sites with no network.

Offline by default

You can label without a connection. Your changes are kept on your computer and sent when the hub can be reached again. Nothing is lost if you close the app first.

While you are offline:

• The project home says “Offline” and shows what is on your computer. Who is here, requests for you and recent activity come back when the hub answers.
• Members and inviting need the hub.
• Nobody’s claims apply, so two people may answer the same events. Both answers are kept. The labeller says so.
• Requests you send, replies, and finishing, declining, closing or withdrawing a request are saved on this computer and sent, in order, once the hub answers. See Ask a colleague.

More than one organisation

If you belong to several organisations, only one is open at a time, but your work in the others still goes to their hubs. When you switch to another organisation, or open a project on this computer only, the one you left keeps sending its waiting changes in the background, about every minute and a quarter, and straight away when you leave it with changes waiting. Requests and replies you saved there while offline go too.

• Choose an organisation shows, beside each organisation, what still waits there: “3 changes not sent yet”, and “1 refused” if the hub refused any.
• While another organisation holds changes, the bottom of the sidebar says so, for example “Kerry Whale Watch: 3 changes not sent yet (sending in the background)”.
• If its changes can’t go (a dataset was archived, the app needs updating, the hub refuses them), the note says so. Sift checks every few minutes whether that changed, and sends them as soon as it has.
• An organisation in the background only sends. It fetches the datasets it sends to, so your changes are checked against their latest state, but not the others. Everything else arrives when you open it again.
• Refused changes in another organisation are counted, and listed in its Refused changes once you open it.
• Signing out of a hub stops all of its organisations. If your sign-in to a hub runs out, its organisations stop syncing until you sign in again: the note still lists their waiting changes. Their changes stay on this computer meanwhile. If someone else signs in to the same hub on this computer, your organisations there stop syncing; they never sync as you under someone else’s sign-in.

How two people stay in step

When you start a labelling session in an organisation dataset, your app asks the hub to claim the next events for you. A claim is a short lease: other people’s apps skip those events while you hold them, so you aren’t both given the same ones. Choosing Done in the labeller releases your claims. If you close your laptop instead, they lapse after about 10 minutes.

Your answers go to the hub as ops. The hub tells everyone else’s app at once that something changed, and their apps fetch the new ops.

sequenceDiagram
  autonumber
  participant A as Aoife's app
  participant H as Hub
  participant D as Declan's app
  A->>H: Claim the next events in Harbour survey
  H-->>A: Granted (events Declan holds are left out)
  Note over A: Aoife answers, even offline.<br/>Each answer is an op.
  A->>H: Send ops
  H-->>D: Live event: Harbour survey changed
  D->>H: Fetch new ops
  Note over D: Same ops, same state.<br/>Different answers show as disputed.

What syncs and what stays on this computer

flowchart LR
  subgraph S["Synced to the project's members"]
    direction TB
    s1["Answers, noise ratings, notes"]
    s2["Boxes people draw, and changes to them"]
    s3["Of interest and Nothing here marks"]
    s4["Label sets"]
    s5["Models' registrations (name, version, file hashes)"]
    s6["Assignments, members and roles"]
    s7["Dataset settings such as the channel map"]
    s8["Published versions"]
  end
  subgraph L["Stays on this computer"]
    direction TB
    l1["Audio files"]
    l2["Where each recording lives here"]
    l3["The model's files"]
    l4["The cache of downloaded audio"]
    l5["Google account tokens and<br/>this computer's signing key"]
    l6["Notification settings"]
  end

A model is registered with the project by name and by the hashes of its files. Each computer that applies it points at its own copy of the files.

“Nothing here” marks sync like the others, including the “No calls in this recording” mark you can make while answering a request. Older versions of Sift kept “No calls” marks on your computer only. The first time you open a dataset in the labeller, your old marks there become “Nothing here” marks your team can see, and Sift tells you how many. If you can only view the dataset, your marks stay on your computer until your role lets you label it. They are shared the next time you open the dataset after that.

Here now and Working now

While you are signed in, your app tells the hub what you are doing about once a minute, for example “Labelling Harbour survey” or “Answering a request”. Other people see it in three places:

• Here now on the project home: who else is in this project right now and what they are doing.
• Working on on the project’s Members page.
• Working now on the organisation’s home.

People who can’t see a project only see “Working in a private project”.

The app sends nothing while you aren’t signed in to an organisation. While you are signed in, a project on this computer only doesn’t appear on any project’s Here now, but what you are doing can still appear in the organisation’s Working now, for example “Labelling” and the dataset’s name.

Share a personal project into the organisation

A project you made before joining, or while working without an organisation, lives on your computer only. You can share it into the organisation so others can work in it. Only the organisation’s owners and admins can do this.

1. Sign in to the organisation and open one of its projects.
2. Open the project switcher at the top of the sidebar.
3. Beside your personal project, choose Share…. You can also use Share to organisation… from the project’s card on Home.
4. Read what happens. Leave Share sites coarsely on to round sites to about 11 km before they leave this computer; datasets you add and send later are shared the same way. Then choose Share project.

What sharing does:

• Its datasets sync, with the project’s label sets, models, answers, boxes and published versions.
• Audio stays on this computer. Nothing is uploaded. Members see the datasets and can listen once they have their own copy of the files. See Audio on this computer.
• This copy stays here. The personal project on your computer isn’t changed.
• You manage it. Only you and the organisation’s admins see it until you invite members. See Projects, members and roles.

Sharing is a one-way copy. Later changes to the personal project stay on your computer until you send them.

The copy on this computer, after sharing

The personal project’s home says Shared to organisation. Work in the organisation’s project from now on. You can keep the personal copy for reference, or choose Archive this copy to put it away. Archiving changes nothing in the organisation.

If you change the personal copy after sharing it, for example a label set, a label or an answer, its home says how many changes were made here since. To bring them to the organisation:

1. Sign in to the organisation.
2. Open the personal project’s home, or choose Send changes to organisation… from its card on Home.
3. Choose Send changes. A preview lists every change.
4. Choose Send.

What sending does:

• New and Change items go to the organisation’s project as your own changes, at the time you made them. They reach the hub at the next sync. Detector runs you made here go too, with their detections and your answers.
• Conflict: someone changed the same thing in the organisation after you made your change, or it is gone there. A deletion or a ruling is a conflict if anyone worked on that box there since. It is left out, so sending never overrides newer work.
• Can’t be sent: something you may not do in the organisation (yet), for example a ruling while you aren’t a manager there.
• Not sent: published versions, new recordings (add the audio to the organisation’s project) and changes someone else made.
• Datasets you added to the personal project since are shared, as when you shared it, with the site as precise or as coarse as you chose then. The preview names them.

Nothing left out is dropped. The copy’s home lists the changes not sent last time and why, and the next Send changes offers them again, for example after you become a manager or the conflicting change is gone. Export for a manager… writes them to a file a manager can import; once the manager’s import reaches the organisation, those changes are no longer offered.

Sending needs the hub: Sift fetches the organisation’s latest first, so the preview checks against it. A change goes once, even if the app closed in the middle of a send. If you shared the same project to a second organisation, each gets its own changes.

Check that you are in sync

The project home shows a sync badge next to the project’s name. Synced means your changes have reached the hub and you have everyone else’s. The organisation’s Datasets page (open the organisation’s home from the account menu, then Datasets in its sidebar) shows the same badge, where it reads In sync. Hover over the badge for a line of detail, or click it on the organisation’s home to sync now.

The badge never says Synced while any change is waiting, blocked, refused, set aside, held here or waiting for related changes. What else it can say:

The badge says	What it means	What to do
Syncing…	A sync is running.	Nothing.
N changes waiting	Your changes haven’t reached the hub yet.	Nothing. They go at the next sync.
N to fetch	Others’ changes are on the hub and not here yet.	Nothing. They arrive at the next sync.
Offline	The hub can’t be reached. Your work is kept here.	Check your connection. Keep working if you like.
Hub busy	The hub asked Sift to wait a little.	Nothing. Sift carries on by itself.
Slow connection: syncing in smaller pieces	The hub answers, but slowly.	Nothing. Sift keeps going on its own.
N changes blocked	Changes you made can’t be sent now, for example the dataset was archived.	Open Datasets: each dataset says why, with what you can do. See Blocked changes.
N changes refused	The hub refused changes you made. They are kept.	Open Refused changes in the sidebar. See Refused changes.
N changes set aside	A change from the hub couldn’t be read here and was set aside. Everything else is in sync.	Nothing. The hub’s managers have been told. Note it if you are in a pilot.
Waiting for N related changes	Changes from others that refer to something not here yet.	Nothing. They apply when it arrives.
N changes held here, not on the hub yet	The hub was restored from a backup and lost changes others made, which your computer holds.	Nothing. Keep syncing. See After the hub was restored.
Sync problem	Something went wrong. The detail says what.	Note the detail. Sift tries again by itself.
Signed out	Your sign-in ended.	Sign in again. Nothing is lost.
Update required	This version of Sift is too old for the hub.	Update Sift. Your work is kept until then.
Hub needs updating	The hub is older than this version of Sift. Changes wait here.	Ask your organisation’s admin to update the hub.
Clock ahead, or In sync — clock N min slow (or fast)	Your computer’s clock is off from the hub’s.	Set the date and time automatically in your computer’s settings.
Not syncing	You are in a project on this computer only.	Nothing. Those datasets don’t sync.

Badges combine, for example “3 changes waiting, 2 blocked”.

The Datasets page

The organisation’s Datasets page lists each dataset with one line about it: “In sync”, “N changes waiting”, “N to fetch”, “Not on the hub yet”, or “N changes blocked:” followed by why. After that line it adds what else applies: “, N refused”, “, N set aside”, “, N held here, not on the hub yet”, “, N versions don’t match”, “ (archived)“. Sync now runs a sync straight away. Export unsynced… writes every change not on the hub yet to a file, whatever its state.

A dataset may also say:

• “Waiting for its earlier history from the hub: version checks wait until it arrives.” Sift is still fetching the dataset’s older changes. Labelling works. Checking versions waits until the history is here.
• “Waiting for a manager to open it in the app” (with the managers’ names), “Until then it’s empty.” A Google Drive dataset made on the hub that no manager’s app has set up yet. See Google Drive datasets.
• “New Google Drive files are added when someone who labels it opens it.” with when it was last listed. See the same section.
• The hub was restored from an older backup. A notice with its own actions. See Restored from an older backup.

Blocked changes

A change is blocked when you made it and it can’t reach the hub now. Sift keeps it on this computer and never sends it on its own. On the Datasets page, each blocked dataset (or a project’s label sets and models) says why and offers what you can do:

It says	Why	You can
“A manager archived this dataset.”	The dataset is archived and read only until a manager restores it.	Ask a manager, export, discard
“Its project, name, is archived.”	The project is archived.	Ask a manager, export (and discard, when the dataset is on the hub)
“This dataset is no longer shared with you.”	A manager took the dataset away from you.	Ask a manager, export, discard
“You’re no longer in its project.”	You were removed from the project.	Ask a manager, export, discard
“You can view this dataset but not change it.”	Your role there is view only.	Ask a manager for label access, export, discard
“Only a manager of project can add this dataset to the hub.”	You shared the dataset into a project while offline, and only a manager can put it on the hub.	Ask a manager, move it to a project you manage, export
“Its project, name, was deleted.”	The project was deleted before the dataset reached the hub.	Move it to another project, export
“This project was deleted.” (a project’s label sets and models)	The project was deleted.	Export, discard
A reason from the hub, with “Ask a manager to add it again, or export the changes.”	The hub was restored from a backup and no longer has this dataset or project, and this computer couldn’t add it again.	Ask a manager, export

A project’s own label sets and models can be blocked the same way: “This project is archived.”, “You’re no longer in this project.” or “Only the project’s managers can change its label sets and models.” Which buttons appear depends on the reason and on whether the dataset ever reached the hub.

The buttons:

• Ask a manager says whom to ask and what to ask for. Your changes stay on this computer until then.
• Export changes… writes the changes to a file. Send it to a manager of the project, who can import them as their own (see Import a member’s changes). For a dataset that never reached the hub, the file is a backup only: a manager can’t import it.
• Move to another project… appears for a dataset that isn’t on the hub yet, when you manage another active project. It is added to the hub there with every change made here.
• Discard changes… undoes the changes on this computer and never sends them. They stay in exports of this computer’s unsynced work. It isn’t offered for a dataset that never reached the hub, because its own creation is part of that work.

Prefer Ask a manager or Export changes… to Discard changes… unless you are sure you don’t want the work.

When a manager restores your access, the changes go at the next sync by themselves.

Refused changes

The hub checks every change against what you may do when the change arrives, not when you made it. If a manager lowered your role, narrowed a dataset or archived the project while you were working offline, the hub refuses the changes you made in the meantime. Sift undoes them on your computer, so you see what everyone else sees, but it never throws them away.

• You are told. A notice says how many changes were refused. The page where you made them says so too, for example the Labels page after a label-set save that said “Saved” while you were offline. The sync badge shows “N changes refused” and never “Synced” while any are kept.
• Refused changes appears in the sidebar with a count. (Managers always have it in the sidebar, to import members’ files; the count shows only when something was refused.) It lists each change in words (“Added a box labelled whistle”), grouped by project and dataset, with why it was refused and when.
• Send again. When your access comes back, Sift offers to send the changes again. It never sends them on its own. Choose Send again on a dataset to queue its changes for the next sync. A box and the edits made to it go together, in order. A change keeps the time you made it, so it never overrides work others did since.
• Some changes can’t be sent again as they are, because the hub refused the change itself rather than who made it. The list says so. Export them for a manager to import, or discard them.
• Discard. Choose Discard to stop seeing changes you no longer want. They are kept in exports but never sent.
• Export writes your unsynced and refused changes to a file. Send it to a manager of the project, who can import them.

Managers see how many changes each member’s computer holds as refused, but never the changes themselves.

Import a member’s changes (managers)

A manager (an organisation owner or admin, or a manager of one of its projects) chooses Import a member’s changes… and picks the file a member sent. It is in two places, whether or not any of your own changes were refused:

• Refused changes in the sidebar. For managers it is always there; it shows a count only while the hub refused changes made on this computer.
• The organisation’s Datasets page, next to Sync now.

A preview lists every change:

• Adds and Changes are imported. Each is made again as your own change, noted as imported from the member.
• Conflict: something changed here after the member made the change, or what it changes is gone. It is left out, so an import never overrides newer work.
• Left out: changes still waiting on the member’s computer (they go to the hub by themselves), changes they discarded, your own changes (send them again instead), and rulings, detector runs and versions.

Choose Import to make the changes. They go to the hub at the next sync. Importing needs the hub: Sift syncs first so the preview checks against everyone’s newest work, and an imported change keeps the time the member made it, so it never overrides later work.

After the hub was restored from a backup

If the organisation’s hub is restored from a backup, it may lose changes made after that backup. Most of the time you have nothing to do:

• Sift notices, fetches every dataset and project again (picking up where it stopped if the connection drops, without downloading again what it holds), and sends again any of your changes the restored hub no longer has. The first sync afterwards takes longer than usual.
• “N changes held here, not on the hub yet”. The restored hub no longer has some changes other people made, which your computer holds. They are kept and shown, never deleted. Each change is signed by the computer it was made on, so your next sync sends them back to the hub as their authors’ work. A change made before Sift signed changes waits for its author’s computer instead. Publishing the dataset waits until the hub has them all. Choose held here… on the Datasets page to see them. See Sites with no network.
• A dataset or project the restored hub lost. A manager’s (or, for a project, an organisation admin’s) computer adds it to the hub again by itself. A notice says so. A project made again comes back with only the person whose computer made it as its member: “its members and their roles must be added again” in the hub web app. If its name was taken meanwhile, it comes back as “name (restored)”. A dataset made again comes back restricted to its managers: “its grants and member access must be set again”. If your computer can’t add it again, its changes show as blocked with the hub’s reason: ask a manager, or export them.
• A refused change says “Re-publish needed”. You published a version after the backup the hub was restored from. Publish the dataset again; the version gets a new number.

Restored from an older backup

Rarely, a dataset shows a notice titled “The hub was restored from an older backup”. The hub was restored from a backup older than history this computer had already cleared out (this can only happen when the hub’s admin turned on device pruning). Everything is still on your computer. But some changes the hub refused can’t be undone here, and version checks and compactions are paused for this dataset. The notice is on the Datasets page and on the dataset’s own page.

What to do, in order:

1. Don’t reinstall Sift and don’t delete anything. Tell your manager or the hub’s admin.
2. Choose Export the work only this computer holds… and save the file somewhere safe.
3. Then one of two things:
   • If the notice offers Start over from the hub…, send the file to a manager, who can import it. Then choose Start over from the hub…: the dataset is rebuilt from what the hub holds, and your changes still to send stay and go at the next sync. The button stays greyed out until the export is written (“Start over becomes available once the export is written: starting over removes that work from this computer”). A manager whose own computer shows this notice for the dataset starts over before importing others’ files.
   • If the notice says “This dataset can’t start over from the hub:” followed by why, keep the file: it is a full backup of the dataset. Give it to your hub’s admin. There is no Start over button. Leave Sift’s data on this computer as it is.

Troubleshooting

“Offline” with changes waiting. Your work is kept here and is sent when the hub is reachable. Check your network connection.

“Slow connection: syncing in smaller pieces”. The hub answers, but slowly, so a batch of changes takes too long to arrive. Sift asks for smaller pieces and keeps going on its own; nothing is waiting on you. Syncing a large dataset takes longer on such a link.

“Hub busy”. The hub asked Sift to wait a little before syncing again (it is under load or being maintained). Sift waits as long as it was asked and then carries on by itself. Your work is kept here meanwhile.

“N changes blocked”. Changes you made that can’t be sent now: an archived dataset or project, access you no longer have, a deleted project, a dataset only a manager can add to the hub. They are kept on this computer. The Datasets page says why for each, and lets you ask a manager, export, move or discard them. See Blocked changes.

“Waiting for N related changes”. Changes from others that refer to something that hasn’t reached your computer yet, such as an edit to a box whose creation is still on its way. They apply as soon as it arrives.

“N changes set aside”. A change from the hub couldn’t be read on this computer, probably because a faulty or older app wrote it. It was set aside so everything else keeps syncing, and reported to the hub automatically. Managers see a notice, “N changes from the hub set aside”, with View to list where.

“Sync problem” on one dataset, the others in sync. The hub couldn’t serve that dataset just now. The rest keep syncing, and Sift tries that dataset again on its own, a little later each time.

Changes waiting in another organisation. They go to its hub in the background while you work elsewhere, as long as you are signed in there. Open that organisation to see the details. See More than one organisation.

“Signed out”. Sign in again to sync. Nothing is lost. (A sign-in refresh that was interrupted, for example by a dropped connection, doesn’t sign you out.)

“Update required”. This version of Sift is too old for the hub. The Datasets page says “Update Sift to keep syncing”, with Check for updates… (or Restart to update) where your copy updates itself. Your work is kept on this computer until then.

“Hub needs updating”. This version of Sift is newer than your organisation’s hub, so changes made here wait until the hub is updated. Your work is kept on this computer. Ask your organisation’s admin to update the hub.

“This computer’s copy is newer than this app”. A newer version of Sift updated this computer’s copy of the organisation, and then an older one was opened. Update Sift to keep working. Meanwhile, Export unsynced work… writes your changes that aren’t on the hub yet to a file.

“Clock ahead”, or a clock minutes slow or fast. Your computer’s clock is off from the hub’s. A slow clock can make your changes lose to changes others made at the same time. A fast one has its changes re-timed to the hub’s clock and sent again, so they aren’t lost, but some may still wait. Set the date and time automatically in your computer’s settings.

“N changes refused”. The hub refused changes you made, usually because your role changed while you were offline. They are kept. Open Refused changes in the sidebar to see why, then send them again once your access is back, export them for a manager, or discard them. See Refused changes.

“Send again” is greyed out. You can’t make those changes now, for example you can only view the dataset. Ask a manager for label access, or export the changes and send the file to a manager.

You copied this computer’s data folder, or restored it from a backup. Nothing to do. Sift notices when an organisation’s data is opened from a copy or a backup, and that copy then syncs as a new device of its own, so it never mixes up its changes with the original’s. Changes that were waiting in the copy still go to the hub. A change both copies hold goes up once. If the original is still in use, both keep syncing.

A change waits with “the hub hasn’t this change’s data yet”. Its data didn’t finish uploading. It stays waiting and is sent again at the next sync; nothing is refused.