SQLite Background Jobs

ServiceStack.Jobs is our solution for queueing and managing background jobs and scheduled tasks in .NET 10 Apps. It's a easy to use library that seamlessly integrates into existing ServiceStack Apps with a built-in Management UI to provide real-time monitoring, inspection and management of background jobs.

Durable and Infrastructure-Free​

Prior to Background Jobs we've been using Background MQ for executing our background tasks which lets you queue any Request DTO to execute its API in a background worker. It's been our preferred choice as it didn't require any infrastructure dependencies since its concurrent queues are maintained in memory, this also meant they were non-durable that didn't survive across App restarts. Whilst ServiceStack MQ enables an additional endpoint for your APIs our main use-case for using it was for executing background tasks which would be better suited by purpose-specific software designed for the task.

SQLite Persistence​

It uses SQLite as the backing store for its durability since it's low latency, fast disk persistence and embeddable file-based database makes it ideally suited for the task which allows creation of naturally partition-able and archivable monthly databases on-the-fly without any maintenance overhead or infrastructure dependencies making it easy to add to any .NET App without impacting or adding increased load to their existing configured databases.

SQLite Background Jobs runs on a single App Server. To process Jobs on multiple App Servers sharing the same database, use RDBMS Background Jobs with PostgreSQL, SQL Server or MySQL, which uses the same IBackgroundJobs APIs, Admin UI and data models.

Queue APIs or Commands​

For even greater reuse you're able to queue your existing ServiceStack APIs as a Background Job in addition to Commands added in the last v8.3 release for encapsulating units of logic into internal invokable, inspectable and auto-retryable building blocks.

Real Time Admin UI​

The Background Jobs Admin UI provides a real time view into the status of all background jobs including their progress, logs, retries, batches and queues, with controls to cancel, requeue and replay Jobs and pause queues. See Monitoring & Operations for a tour.

Feature Overview​

• No infrastructure dependencies
  • Monthly archivable rolling Databases with full Job Execution History
• Execute existing APIs or versatile Commands
  • Commands auto registered in IOC
• Named queues with their own concurrency, priorities and rate limits
  • Pause, resume and re-throttle queues at runtime
• Serially execute jobs with the same named Worker, or per concurrency key
• Multi-step workflows with Jobs dependent on their parent Job
• Job Batches with live progress and completion callbacks
• Await a Job's result or have it delivered to a webhook or MQ
• Retries with exponential backoff and jitter, with every failed attempt recorded
• Idempotent and singleton Jobs to prevent duplicate work
• Job expiry, timeouts and cancellation
• Durable, time zone aware Recurring Tasks
• Queue Jobs to be executed after a specified Date
• Execute Jobs within the context of an Authenticated User
• Maintain Status, Logs and Progress of Executing Jobs
• Execute transitive (i.e. non-durable) jobs using named workers
• Health checks, OpenTelemetry and Profiling
• Attach optional Tag, TenantId, CreatedBy and Args metadata to Jobs

Please let us know if there are any other missing features you would love to see implemented.

Install​

As it's more versatile and better suited, we've replaced the usage of Background MQ with ServiceStack.Jobs in all .NET 10 Identity Auth Templates for sending Identity Auth Confirmation Emails when SMTP is enabled. So the easiest way to get started with ServiceStack.Jobs is to create a new Identity Auth Project, e.g:

Existing .NET 10 Templates​

Existing .NET 10 Projects can configure their app to use ServiceStack.Jobs by mixing in:

Which adds the Configure.BackgroundJobs.cs Modular Startup configuration and a ServiceStack.Jobs NuGet package reference to your project.

Usage​

SQLite Background Jobs uses the same IBackgroundJobs APIs as RDBMS Background Jobs, where any API, Controller or Minimal API can queue your APIs and Commands, e.g:

class MyService(IBackgroundJobs jobs) : Service 
{
    public object Any(MyOrder request)
    {
        var jobRef = jobs.EnqueueCommand<SendEmailCommand>(new SendEmail {
            To = "my@email.com",
            Subject = $"Received New Order {request.Id}",
        });
        //...
    }
}

See Using Background Jobs for queueing APIs and Commands, all the options available when queueing Jobs, and logging, progress and cancellation from inside a running Job.

Configuration​

The main BackgroundsJobFeature options:

Upgrading from v10.2​

v10.3 upgrades the Background Jobs schema on startup, and clears Jobs that are still queued or running when the upgrade is applied. See Upgrading to v10.3 for what changes and how to prepare.