Durable sequential tasks#
TaskRunner runs tasks one at a time in the PythonHere process and persists
their inputs, results, errors, and retry state. Work continues if Jupyter
disconnects. After PythonHere restarts, completed tasks remain complete and an
interrupted task becomes runnable again.
Connect to PythonHere#
%load_ext pythonhere
%connect-there
Submit and start work#
Every task needs a unique nonempty string id. Tasks and returned results must
contain plain JSON-compatible values. If "squares-example" already exists
from an earlier attempt, use a new name or delete the old run with the
cleanup cell at the end.
%%there
import time
from tasks_here import TaskRunner
def execute_task(task, context):
time.sleep(1)
if task["value"] == 5:
raise ValueError("Intentional example failure for value 5")
return {
"value": task["value"],
"square": task["value"] ** 2,
}
runner = TaskRunner.create("squares-example", max_attempts=3)
runner.add([{"id": str(value), "value": value} for value in range(1, 11)])
runner.start(execute_task)
start() returns promptly while tasks run one at a time in a background thread.
The task for value 5 deliberately fails. With max_attempts=3, the runner
tries it three times before marking it failed; the other nine tasks still run.
Including retries, this example takes about twelve seconds. Tasks continue
running between notebook cells, so the run may already be completed when you
inspect it.
Inspect progress and results#
status = %there get runner.status()
status
{'name': 'squares-example',
'state': 'running',
'total': 10,
'pending': 9,
'running': 1,
'completed': 0,
'failed': 0,
'progress': 0.0,
'current': '1',
'pause_requested': False,
'max_attempts': 3,
'exclusive': True,
'created_at': '2026-10-15T11:08:50.392116+00:00',
'updated_at': '2026-10-15T11:08:50.498292+00:00',
'metadata': {},
'worker_error': None,
'path': '/data/user/0/me.herethere.pythonhere/files/task-runs/squares-example'}
You can inspect results before the whole run finishes:
results = %there get runner.results()
results
Show code cell output
[{'id': '1',
'status': 'running',
'task': {'id': '1', 'value': 1},
'result': None,
'result_present': False,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:50.495452+00:00',
'finished_at': None,
'duration': None,
'error': None}],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.455575+00:00',
'completed_at': None},
{'id': '2',
'status': 'pending',
'task': {'id': '2', 'value': 2},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.455746+00:00',
'completed_at': None},
{'id': '3',
'status': 'pending',
'task': {'id': '3', 'value': 3},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.455842+00:00',
'completed_at': None},
{'id': '4',
'status': 'pending',
'task': {'id': '4', 'value': 4},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.455924+00:00',
'completed_at': None},
{'id': '5',
'status': 'pending',
'task': {'id': '5', 'value': 5},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456001+00:00',
'completed_at': None},
{'id': '6',
'status': 'pending',
'task': {'id': '6', 'value': 6},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456079+00:00',
'completed_at': None},
{'id': '7',
'status': 'pending',
'task': {'id': '7', 'value': 7},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456155+00:00',
'completed_at': None},
{'id': '8',
'status': 'pending',
'task': {'id': '8', 'value': 8},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456228+00:00',
'completed_at': None},
{'id': '9',
'status': 'pending',
'task': {'id': '9', 'value': 9},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456301+00:00',
'completed_at': None},
{'id': '10',
'status': 'pending',
'task': {'id': '10', 'value': 10},
'result': None,
'result_present': False,
'attempt_count': 0,
'failure_count': 0,
'attempts': [],
'duration': None,
'total_duration': 0,
'error': None,
'created_at': '2026-10-15T11:08:50.456375+00:00',
'completed_at': None}]
To wait briefly before checking progress again:
%%there --worker
runner.wait(timeout=1)
print(runner.status())
{'name': 'squares-example', 'state': 'running', 'total': 10, 'pending': 8, 'running': 1, 'completed': 1, 'failed': 0, 'progress': 0.1, 'current': '2', 'pause_requested': False, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-10-15T11:08:50.392116+00:00', 'updated_at': '2026-10-15T11:08:51.534568+00:00', 'metadata': {}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere/files/task-runs/squares-example'}
This waits at most one second and returns sooner if the worker has stopped.
Pause and resume (optional)#
While tasks are still running, you can pause them. pause() lets the current
task finish, and wait() waits for the pause to take effect:
%%there --worker
runner.pause()
runner.wait()
print(runner.status())
{'name': 'squares-example', 'state': 'paused', 'total': 10, 'pending': 8, 'running': 0, 'completed': 2, 'failed': 0, 'progress': 0.2, 'current': None, 'pause_requested': True, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-10-15T11:08:50.392116+00:00', 'updated_at': '2026-10-15T11:08:52.566010+00:00', 'metadata': {}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere/files/task-runs/squares-example'}
Resume any pending tasks with the same executor. Completed tasks are kept:
%%there
runner.start(execute_task)
If all tasks have already completed, there is nothing left to pause or resume.
Wait for completion#
Wait for all remaining work, then show the final counts. --worker keeps the
app’s UI responsive while this cell waits:
%%there --worker
runner.wait()
status = runner.status()
print("State:", status["state"])
print("Completed:", status["completed"], "of", status["total"])
print("Failed:", status["failed"])
State: completed
Completed: 9 of 10
Failed: 1
Expected output:
State: completed
Completed: 9 of 10
Failed: 1
wait() returns when the worker stops. A completed state means all tasks
have been processed, including any that exhausted their retries; it does not
mean every task succeeded. Inspect the deliberately failed task:
failures = %there get runner.failed()
failures
[{'id': '5',
'status': 'failed',
'task': {'id': '5', 'value': 5},
'result': None,
'result_present': False,
'attempt_count': 3,
'failure_count': 3,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:54.808851+00:00',
'finished_at': '2026-10-15T11:08:55.818719+00:00',
'duration': 1.0025099229533225,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}},
{'number': 2,
'started_at': '2026-10-15T11:09:01.065718+00:00',
'finished_at': '2026-10-15T11:09:02.087962+00:00',
'duration': 1.0053778459550813,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}},
{'number': 3,
'started_at': '2026-10-15T11:09:02.118377+00:00',
'finished_at': '2026-10-15T11:09:03.130124+00:00',
'duration': 1.0041442309739068,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}}],
'duration': 1.0041442309739068,
'total_duration': 3.0120319998823106,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'},
'created_at': '2026-10-15T11:08:50.456001+00:00',
'completed_at': None}]
The failed record has id: "5", attempt_count: 3, failure_count: 3,
and the intentional ValueError. The other nine tasks have their square results.
Export results#
For larger results, export a JSON snapshot on the device and download the file:
%%there --worker
runner.export_results("task-results.json")
%there download task-results.json ./task-results.json
Load the downloaded file in the notebook:
import json
with open("task-results.json", encoding="utf-8") as f:
results = json.load(f)
The export includes all task records, including pending and failed tasks, in
submission order. Export after completion for final results, or during execution
for a snapshot. Relative paths use the device’s current working directory,
which is PythonHere’s upload directory by default. For large runs, exporting
and downloading a file avoids transferring all results through %there get.
Reopen after PythonHere restarts#
The task list and results are persistent, but Python callables and other runtime objects are not. After restarting PythonHere, reconnect, recreate the executor, open the named run, and start it again. Completed tasks are kept; only unfinished work runs again. If all ten tasks already finished, the run stays completed.
In this walkthrough, all tasks have already finished. Reopening shows that
the results and the failed task survived the restart. The task that exhausted
its retries stays failed; calling start() does not retry it again. To try recovery of unfinished work separately,
restart the app while a fresh run is executing, then reconnect and run this cell:
%%there
import time
from tasks_here import TaskRunner
def execute_task(task, context):
time.sleep(1)
if task["value"] == 5:
raise ValueError("Intentional example failure for value 5")
return {
"value": task["value"],
"square": task["value"] ** 2,
}
runner = TaskRunner.open("squares-example")
runner.start(execute_task)
Wait for any unfinished work, then inspect the saved results:
%%there --worker
runner.wait()
print(runner.status())
{'name': 'squares-example', 'state': 'completed', 'total': 10, 'pending': 0, 'running': 0, 'completed': 9, 'failed': 1, 'progress': 1.0, 'current': None, 'pause_requested': False, 'max_attempts': 3, 'exclusive': True, 'created_at': '2026-10-15T11:08:50.392116+00:00', 'updated_at': '2026-10-15T11:09:03.563013+00:00', 'metadata': {}, 'worker_error': None, 'path': '/data/user/0/me.herethere.pythonhere/files/task-runs/squares-example'}
saved_results = %there get runner.results()
saved_results
Show code cell output
[{'id': '1',
'status': 'done',
'task': {'id': '1', 'value': 1},
'result': {'value': 1, 'square': 1},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:50.495452+00:00',
'finished_at': '2026-10-15T11:08:51.512079+00:00',
'duration': 1.0042130000074394,
'error': None}],
'duration': 1.0042130000074394,
'total_duration': 1.0042130000074394,
'error': None,
'created_at': '2026-10-15T11:08:50.455575+00:00',
'completed_at': '2026-10-15T11:08:51.512215+00:00'},
{'id': '2',
'status': 'done',
'task': {'id': '2', 'value': 2},
'result': {'value': 2, 'square': 4},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:51.532225+00:00',
'finished_at': '2026-10-15T11:08:52.544140+00:00',
'duration': 1.0043134610168636,
'error': None}],
'duration': 1.0043134610168636,
'total_duration': 1.0043134610168636,
'error': None,
'created_at': '2026-10-15T11:08:50.455746+00:00',
'completed_at': '2026-10-15T11:08:52.544266+00:00'},
{'id': '3',
'status': 'done',
'task': {'id': '3', 'value': 3},
'result': {'value': 3, 'square': 9},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:52.716334+00:00',
'finished_at': '2026-10-15T11:08:53.739693+00:00',
'duration': 1.0050933079910465,
'error': None}],
'duration': 1.0050933079910465,
'total_duration': 1.0050933079910465,
'error': None,
'created_at': '2026-10-15T11:08:50.455842+00:00',
'completed_at': '2026-10-15T11:08:53.739822+00:00'},
{'id': '4',
'status': 'done',
'task': {'id': '4', 'value': 4},
'result': {'value': 4, 'square': 16},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:53.768668+00:00',
'finished_at': '2026-10-15T11:08:54.782363+00:00',
'duration': 1.0052369229961187,
'error': None}],
'duration': 1.0052369229961187,
'total_duration': 1.0052369229961187,
'error': None,
'created_at': '2026-10-15T11:08:50.455924+00:00',
'completed_at': '2026-10-15T11:08:54.782494+00:00'},
{'id': '5',
'status': 'failed',
'task': {'id': '5', 'value': 5},
'result': None,
'result_present': False,
'attempt_count': 3,
'failure_count': 3,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:54.808851+00:00',
'finished_at': '2026-10-15T11:08:55.818719+00:00',
'duration': 1.0025099229533225,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}},
{'number': 2,
'started_at': '2026-10-15T11:09:01.065718+00:00',
'finished_at': '2026-10-15T11:09:02.087962+00:00',
'duration': 1.0053778459550813,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}},
{'number': 3,
'started_at': '2026-10-15T11:09:02.118377+00:00',
'finished_at': '2026-10-15T11:09:03.130124+00:00',
'duration': 1.0041442309739068,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'}}],
'duration': 1.0041442309739068,
'total_duration': 3.0120319998823106,
'error': {'type': 'ValueError',
'message': 'Intentional example failure for value 5',
'traceback': 'Traceback (most recent call last):\n File "/home/user/hostcwd/.buildozer/android/app/tasks_here.py", line 943, in _run_loop\n File "<string>", line 10, in execute_task\nValueError: Intentional example failure for value 5\n'},
'created_at': '2026-10-15T11:08:50.456001+00:00',
'completed_at': None},
{'id': '6',
'status': 'done',
'task': {'id': '6', 'value': 6},
'result': {'value': 6, 'square': 36},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:55.851173+00:00',
'finished_at': '2026-10-15T11:08:56.865618+00:00',
'duration': 1.0033425380242988,
'error': None}],
'duration': 1.0033425380242988,
'total_duration': 1.0033425380242988,
'error': None,
'created_at': '2026-10-15T11:08:50.456079+00:00',
'completed_at': '2026-10-15T11:08:56.865746+00:00'},
{'id': '7',
'status': 'done',
'task': {'id': '7', 'value': 7},
'result': {'value': 7, 'square': 49},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:56.889569+00:00',
'finished_at': '2026-10-15T11:08:57.903773+00:00',
'duration': 1.0033163069747388,
'error': None}],
'duration': 1.0033163069747388,
'total_duration': 1.0033163069747388,
'error': None,
'created_at': '2026-10-15T11:08:50.456155+00:00',
'completed_at': '2026-10-15T11:08:57.903901+00:00'},
{'id': '8',
'status': 'done',
'task': {'id': '8', 'value': 8},
'result': {'value': 8, 'square': 64},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:57.933493+00:00',
'finished_at': '2026-10-15T11:08:58.948942+00:00',
'duration': 1.0050470000132918,
'error': None}],
'duration': 1.0050470000132918,
'total_duration': 1.0050470000132918,
'error': None,
'created_at': '2026-10-15T11:08:50.456228+00:00',
'completed_at': '2026-10-15T11:08:58.949102+00:00'},
{'id': '9',
'status': 'done',
'task': {'id': '9', 'value': 9},
'result': {'value': 9, 'square': 81},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:08:58.979740+00:00',
'finished_at': '2026-10-15T11:08:59.994785+00:00',
'duration': 1.0035531540052034,
'error': None}],
'duration': 1.0035531540052034,
'total_duration': 1.0035531540052034,
'error': None,
'created_at': '2026-10-15T11:08:50.456301+00:00',
'completed_at': '2026-10-15T11:08:59.994928+00:00'},
{'id': '10',
'status': 'done',
'task': {'id': '10', 'value': 10},
'result': {'value': 10, 'square': 100},
'result_present': True,
'attempt_count': 1,
'failure_count': 0,
'attempts': [{'number': 1,
'started_at': '2026-10-15T11:09:00.027395+00:00',
'finished_at': '2026-10-15T11:09:01.040481+00:00',
'duration': 1.0060150770004839,
'error': None}],
'duration': 1.0060150770004839,
'total_duration': 1.0060150770004839,
'error': None,
'created_at': '2026-10-15T11:08:50.456375+00:00',
'completed_at': '2026-10-15T11:09:01.040608+00:00'}]
Execution is at least once. If PythonHere stops after an executor begins but before its result is committed, that task can run again. Executors with irreversible side effects should therefore be idempotent or check durable state before applying those effects.
Optional: reuse runtime resources#
The executor always receives a second argument. It is None when start() has
no setup callback. For expensive runtime resources, create them once per worker
session and reuse the returned context for every task and retry:
def setup():
return create_runtime()
def execute_with_runtime(task, runtime):
return runtime.process(task)
runner.start(execute_with_runtime, setup=setup)
Setup runs in the background worker thread, so start() still returns promptly.
While setup executes, runner.status()["state"] is "setting_up"; after setup
succeeds it becomes "running" before the first task is claimed.
If pause is requested while setup is running, setup finishes and the worker exits
before claiming a task. A later start() calls setup again. Context is runtime
state: it is not persisted or included in results, and the runner does not close
it automatically. Applications that require deterministic resource cleanup must
manage that cleanup themselves.
Clean up#
When you are done inspecting the completed run, you can delete it to repeat
this example with the same name. Skip this cell to keep its saved results.
This deletes only "squares-example" and its tasks/results; other runs and
exported JSON files are untouched.
%%there --worker
from tasks_here import TaskRunner, TaskRunNotFoundError
try:
saved_run = TaskRunner.open("squares-example")
except TaskRunNotFoundError:
print("The example run has already been deleted.")
else:
saved_run.delete()
print("Example run deleted. You can now run the example again.")
Example run deleted. You can now run the example again.
A running task run cannot be deleted. Finish or pause it first.