Loops and parallel steps

Steps are ordinary function calls, so they work in loops, in conditions, and in Promise.all. This page shows how Nelkap finds each one again, and how to handle jobs too big for a single run.

Steps in a loop

Each step in a loop is saved on its own. An import that stops halfway goes on from the last batch that made it:

export async function importContacts(fileId: string) {
  const batches = await nelkap.step.run('batches', () =>
    splitIntoBatches(fileId, 500),
  );

  for (const [index, batchId] of batches.entries()) {
    await nelkap.step.run(`batch ${index}`, () =>
      importRows(batchId),
    );
  }
}

The list of batches comes from a step too, so every call of the run loops over the same list. A name can repeat: steps of the same kind and name are matched in the order your code reaches them. A loop of step.run('send') works, but a name that says which item it handles, like batch 3, is easier to read on the run’s page.

Steps side by side

Steps can run at the same time. Start them together and wait for all of them:

const [profile, orders] = await Promise.all([
  nelkap.step.run('profile', () => getProfile(userId)),
  nelkap.step.run('orders', () => getOrders(userId)),
]);

The two functions run in parallel, and each result is saved as it finishes. They’re saved in the order your code called them, so a run that picks up later finds them the same way.

Steps in conditions

A step can sit behind an if, as long as the condition takes the same path on every call. Base it on the payload or on the result of an earlier step, never on something that can change between calls, like the time:

const user = await nelkap.step.run('user', () =>
  getUser(userId),
);

if (user.plan === 'pro') {
  await nelkap.step.run('pro welcome', () =>
    sendProWelcome(userId),
  );
}

Stay under 100 steps

A run holds up to 100 steps of every kind, and 32 MB of saved steps in all. A run that goes over fails with stepLimitReached. Two ways keep big jobs under the limit:

Put several items in each step. A step that imports 500 rows counts once.
Split the job across runs, as below.

Split a job across runs

For a job of any size, give each part its own run. A first workflow lists the parts and starts a run for each one, then each run does its part:

export async function importFile(fileId: string) {
  const batches = await nelkap.step.run('batches', () =>
    splitIntoBatches(fileId, 500),
  );

  for (const [index, batchId] of batches.entries()) {
    await nelkap.step.run(`start batch ${index}`, () =>
      nelkap.run(importBatch, batchId),
    );
  }
}

export async function importBatch(batchId: string) {
  await nelkap.step.run('import', () =>
    importRows(batchId),
  );
}

Starting each run in a step means a run that picks up later doesn’t start the same batch twice. Give importBatch a concurrency limit to control how many batches go at once. The first workflow still counts one step per batch. For more batches than one run can start, add a level: the first workflow starts a run per chunk of batches, and each of those starts a run per batch. The first run ends once it has started every batch, without waiting for them. To act when the last batch is done, count the batches done in your database, as the large import example does.

Get started

Concepts

Frameworks

Triggers

Write workflows

Deploy

Monitor

Manage

Help

Loops and parallel steps

Steps in a loop

Steps side by side

Steps in conditions

Stay under 100 steps

Split a job across runs

​Steps in a loop

​Steps side by side

​Steps in conditions

​Stay under 100 steps

​Split a job across runs

Steps in a loop

Steps side by side

Steps in conditions

Stay under 100 steps

Split a job across runs