@setu-ts/storage-plugin

File and object storage. Registers an IStorage under CAPABILITIES.STORAGE ('storage').

Five providers ship: MemoryProvider (zero-dependency default), LocalStorageProvider (over runtime.fs), S3Provider, GcsProvider, and AzureBlobProvider. Backblaze B2 is a first-class 'b2' provider type reusing S3Provider over B2’s S3-compatible endpoint.

Installation

import { StoragePlugin } from '@setu-ts/storage-plugin';

Install the peers for the selected provider when using its built-in lazy loader:

Both S3 packages are loaded at startup, even before requesting a signed URL. A missing peer makes startup reject with OptionalDependencyUnavailableError naming that package. Installing only @aws-sdk/client-s3 is insufficient.

With only the S3 client installed, Node startup reports: OptionalDependencyUnavailableError: Optional dependency '@aws-sdk/s3-request-presigner' could not be loaded by '@setu-ts/storage-plugin'. Install it where '@setu-ts/storage-plugin' is installed, in the version range that package declares for it — in an npm, Yarn or Bun workspace that is usually the workspace root, not the member — or choose an option that does not need it on this runtime.

Usage

In a full-stack starter composition, configure its storage arm in setu.config.ts. The starter owns that registration; adding another app.register(StoragePlugin(...)) can duplicate it. See the starter options. The registration below applies when your application owns its plugin list.

import { createUploadMiddleware, getUploadedFile, StoragePlugin } from '@setu-ts/storage-plugin';
import { CAPABILITIES, type IStorage } from '@setu-ts/common';

app.register(
  StoragePlugin({
    provider: 's3',
    options: { bucket: 'uploads', region: 'us-east-1' },
  }),
);

const storage = app.services.get<IStorage>(CAPABILITIES.STORAGE);

const bytes = new Uint8Array([137, 80, 78, 71]);
await storage.put('avatars/ada.png', bytes, { contentType: 'image/png' });
const data = await storage.get('avatars/ada.png');
const url = await storage.getSignedUrl('avatars/ada.png', { expiresIn: 3600 });

Uploads

import { createUploadMiddleware, getUploadedFile } from '@setu-ts/storage-plugin';
import { CAPABILITIES, type IStorage } from '@setu-ts/common';

const storage = app.services.get<IStorage>(CAPABILITIES.STORAGE);

app.router.post('/upload', {
  middleware: [createUploadMiddleware({ fieldname: 'avatar', maxSize: 5_000_000 })],
  handler: async (ctx) => {
    const file = getUploadedFile(ctx, 'avatar');
    if (!file) return ctx.response.status(400).json({ error: 'No file' });

    await storage.put(`avatars/${file.filename}`, file.data, {
      contentType: file.mimeType,
    });
    return ctx.response.json({ ok: true });
  },
});

Three things this snippet gets right that an earlier version did not, each of which failed at a different point:

• fieldname must name the field you read back. The middleware defaults it to 'file' and FILTERS parts to it, so without this line getUploadedFile(ctx, 'avatar') is guaranteed to return undefined.
• The option is maxSize, not maxFileSize — and the compiler’s suggestion for the misspelling is maxFiles, a file-COUNT cap, so taking it silently changes the meaning.
• getUploadedFile returns UploadedFile | undefined, so it needs the guard before file.filename is read.

Passing contentType is what makes a signed URL render the image rather than download it; the field on UploadedFile is mimeType.

Size limits answer 413; a malformed body, a disallowed MIME type and too many files answer 400. That includes a body the runtime refused before this middleware ran: maxBodyBytes on RuntimePlugin rejects the read with a 413-hinted error, and the middleware serves that hint rather than reporting a malformed multipart.

The multipart parser is implemented in this package — no dependency.

Options

StoragePluginOptions is a union discriminated on provider, so each backend’s options are checked against that backend’s own shape and an unknown key is reported by name. provider is optional only on the memory arm, which is the default.

The 'local' provider’s LocalStorageProviderOptions has one optional field:

Upload middleware options

The effective parse bound is min(maxSize * 2 + framing, maxBodyBytes). It bounds parsing and the per-part copies the parse makes, not the initial read: the HTTP adapter buffers the whole body before any middleware runs and IRequest exposes no body stream, so no middleware can decline to read it.

Object metadata

storage.put(path, data, { contentType, metadata }) records attributes on the stored object. Backends differ in what they can hold, and that is a property of the backend rather than a gap:

Memory and local store bytes only: IStorage.get returns bytes, the local provider’s getSignedUrl rejects, and the memory provider’s URL is synthetic, so nothing could read an attribute back.

Signed URLs

getSignedUrl semantics differ by provider, deliberately:

A presigned URL serves the object under whatever content type it was stored with, which is why put takes one — see Object metadata.

The local provider needs write permission

LocalStorageProvider writes through runtime.fs, so on Deno the process must be started with --allow-write. The generated start task in a scaffolded project requests --allow-net --allow-env --allow-read --allow-sys and not that, so a project switched to provider: 'local' needs the flag added.

The provider proves the root is writable at connect() and refuses to start with the flag named, rather than letting every upload fail against a health check that reports up — a stat probe only proves the root is READABLE.

Streaming

The optional getStream? reads an object as a ReadableStream<Uint8Array>, wired through IResponse.stream() for zero-copy downloads.

Backpressure — per provider

Demand-driven streaming is not a property of getStream? in general. It holds for S3Provider and is measured there; the other providers reach the object differently, and two of them read as fast as the backend will send regardless of what the consumer is doing.

For the two eager providers and the two fallbacks, a large object is resident in memory regardless of how slowly the client reads. Cancelling a GCS or Azure download does now release the upstream stream, but it cannot undo the bytes already drained, and the two fallbacks have no upstream left to release. Only the S3 path should be relied on for a large-object relay today.

Measured for S3Provider against real MinIO: it stays 2-3 MiB ahead of the reader — 3.1 MiB at 1 MiB/s, 1.9-2.5 MiB at 2 MiB/s — and that figure is constant, moving by at most 0.30 MiB over an eight-second window while delivered bytes grow four-fold. A consumer that stops reading altogether stops the wire. test/integration/stream-backpressure-real.test.ts pins all of that at both rates against a byte-counting relay in front of the backend. (The same figure was observed once on a 256 MiB object; that is provenance for the size-independence claim, not something the guard re-checks — it runs against 64 MiB.)

Process RSS does not measure it. A large download raises the allocator high-water mark without retaining anything: forcing GC mid-transfer returns heapUsed and external to baseline while RSS stays up, and importing the AWS SDK alone costs ~29 MiB before any object is touched. A generated ReadableStream is therefore not a like-for-like control for an SDK-backed one, and comparing the two on RSS reports a difference that is mostly the SDK. Count bytes at the socket instead.

Health indicator

Registered under the storage capability. Since M70c it reports two signals: the provider’s lifecycle (isReady()) and its reachability (isHealthy()).

data reports { provider, reachable }, where reachable is true, false, or 'unknown' when the provider has no liveness check.

Diagnostics (M98m)

StoragePlugin({ diagnostics: { enabled: true, alias: 'primary' } }) counts every public call this instance’s own StorageService makes (put, get, delete, exists, getSignedUrl, getStream) — succeeded and failed — for the local diagnostics connector’s GET /v1/storage (@setu-ts/diagnostics-plugin). getSignedUrl mints a URL rather than transferring bytes, so its record carries lastDurationMs: null and lastBytes: null; lastBytes is the byte length of the last settled buffer for put and get (a get that finds nothing throws and settles as a failure), and null for the non-buffered operations; getStream counts the open, not the drain. Object paths, stored bytes, content types, signed URLs and errors are never captured. Without the option the instance registers an inert disabled source and its storage calls are not touched. Results, errors, the getStream? fallback and rejection reasons are identical either way; enabled calls add two monotonic clock reads (every call is timed) and one derived promise each (which re-rejects with the original reason, so an unhandled rejection stays unhandled). Direct provider calls and a service constructed without the collector are outside coverage. Enable only on an approved development dataset: counts aggregate every tenant using the instance.

Exports

Generated from the package barrel by pnpm run docs:exports; pnpm run check:docs fails when it drifts.

Full API

Every export and option is documented in PUBLIC_API.md.

API reference

Storage (@setu-ts/storage-plugin)

Provides file storage abstraction.

Registration

import { createUploadMiddleware, getUploadedFile, StoragePlugin } from '@setu-ts/storage-plugin';
import type { IStorage } from '@setu-ts/common';
import { CAPABILITIES } from '@setu-ts/common';

app.register(
  StoragePlugin({
    provider: 's3',
    options: {
      bucket: config.get('S3_BUCKET'),
      region: config.get('AWS_REGION'),
      accessKeyId: config.get('AWS_ACCESS_KEY_ID'),
      secretAccessKey: config.get('AWS_SECRET_ACCESS_KEY'),
    },
  }),
);

Upload middleware

The upload surface is a free exported middleware factory (createUploadMiddleware), NOT a method on IStorage. Parsed files are exposed through the committed per-request ctx.state bag plus a typed getUploadedFile() helper.

const uploadMw = createUploadMiddleware({
  fieldname: 'file', // parts with any other name are dropped
  maxSize: 10 * 1024 * 1024, // per file; exceeding it answers 413
  allowedMimeTypes: ['image/jpeg', 'image/png'], // optional
  maxFiles: 5, // optional; exceeding it answers 400
});

app.router.post('/upload', {
  middleware: [uploadMw],
  handler: async (ctx) => {
    // `getUploadedFile` returns `UploadedFile | undefined`, and the fieldname
    // must match the middleware's — it filters parts to that name.
    const file = getUploadedFile(ctx, 'file');
    if (!file) return ctx.response.status(400).json({ error: 'No file' });

    const storage = ctx.services.get<IStorage>(CAPABILITIES.STORAGE);
    // `file.name` is the form field name; `file.filename` is the client's original file name.
    const key = `uploads/${file.filename}`;
    // Without `contentType` the object is stored as `application/octet-stream`
    // and the signed URL below downloads it instead of rendering it.
    await storage.put(key, file.data, { contentType: file.mimeType });

    const url = await storage.getSignedUrl(key, { expiresIn: 3600 });
    return ctx.response.json({ url, key });
  },
});

Refusal statuses: a body or file over its limit answers 413; a malformed body, a disallowed MIME type and too many files answer 400. A body refused by RuntimePlugin({ maxBodyBytes }) before the middleware ever sees it also answers 413, served from that refusal’s own status hint — it is not reported as a malformed multipart, which is what a client would otherwise be told to fix.

maxBodyBytes (default 50 MB) caps the body the middleware will PARSE, with the effective bound min(maxSize * 2 + framing, maxBodyBytes). It bounds the parse THIS middleware triggers — which, now that forms are read through the shared memoized accessor (M94b), is not necessarily the request’s first parse — not the initial read: the body is read lazily by the first consumer (M87), and RuntimePlugin({ maxBodyBytes }) (M90a) is the bound that covers every reader, including a CSRF middleware that parsed the same form earlier. IRequest still exposes no body stream, so no middleware can decline to read; capping the read for every reader needs a streaming request body.

Since M94b the middleware parses through the request’s formData accessor (or the same shared parseFormBody when the request omits it), and a part carrying NO filename under the field name is a plain form value, not an upload — the web standard’s answer; see the CHANGELOG migration note.

Usage — buffered download

app.router.get('/files/:key', async (ctx) => {
  const storage = ctx.services.get<IStorage>(CAPABILITIES.STORAGE);
  const file = await storage.get(ctx.params.key);
  return ctx.response.header('content-type', 'application/octet-stream').send(file);
});

Usage — streaming download (getStream?)

app.router.get('/files/stream/:key', async (ctx) => {
  const storage = ctx.services.get<IStorage>(CAPABILITIES.STORAGE);
  const stream = await storage.getStream!(ctx.params.key);
  return ctx.response.header('content-type', 'application/octet-stream').stream(stream);
});

Demand-driven streaming is a property of S3Provider, not of getStream? in general. On that path, serving a large object to a slow client does not pull the object into memory: the provider stays 2-3 MiB ahead of the consumer (measured constant in elapsed time and in bytes delivered), a consumer that stops reading stops the wire, and cancelling releases the upstream connection. On every other provider a large object becomes memory-resident regardless of how slowly the client reads — GcsProvider and AzureBlobProvider stream natively but drain their SDK stream eagerly with no demand signal (cancelling releases the upstream stream, but cannot undo what has already been drained), while MemoryProvider and LocalStorageProvider have no native getStream at all and take StorageService’s buffered fallback. The storage-plugin README carries the per-provider table. Note also that process RSS does not measure any of this — a large download raises the allocator high-water mark while retaining nothing, and importing the AWS SDK costs ~29 MiB by itself.

Providers

The plugin ships five named providers plus a first-class B2 preset that reuses S3 under the hood.

All cloud providers support an injectable client option that bypasses the lazy import, but the three types are NOT alike and the difference decides what you can pass:

IS3Backend was named IAwsS3Client before the alpha.9 release, which promised something it never was: @aws-sdk/client-s3’s surface is send(command), so a real S3Client was refused with Injected S3 client is missing required methods. There is consequently no supported way to configure the underlying SDK client (custom retry policy, timeout, proxy agent). The old name is still exported as a deprecated alias of the same type (AI_GUIDELINES §9.2), so existing imports keep compiling; new code should use IS3Backend.

IStorage methods

Per-provider getSignedUrl behavior

Health status

Since M70c the indicator reports two signals: the provider’s lifecycle (isReady()) and its reachability (isHealthy()).

data reports { provider, reachable }, where reachable is true, false, or 'unknown' when the provider has no liveness check.

Storage diagnostics (M98m)

StoragePlugin({ diagnostics: { enabled: true, alias: 'primary' } }) counts every public call its OWN StorageService makes — put, get, delete, exists, getSignedUrl, getStream — for the local diagnostics connector’s GET /v1/storage. enabled must be the literal true; alias is 1–64 UTF-8 bytes with no control character; any other key is refused. All three refuse when StoragePlugin(...) is called, with fixed messages that never echo a value.

Every instance, opted in or not, registers an IStorageDiagnosticsSource under CAPABILITIES.STORAGE_DIAGNOSTICS with { multi: true } (not in provides); without the option it answers disabled and nothing is attached to the service — no clock read and no extra promise on any storage call. With it, each call reads the runtime monotonic clock twice (at start and at settlement — every call is timed, so ageMs, lastDurationMs and lastBytes describe one settlement) and returns a promise derived from the provider’s that re-rejects with the original reason, so an unhandled provider rejection stays unhandled; results, null semantics, the getStream? fallback and the ORIGINAL rejection reason are unchanged, and a synchronous provider throw reaches the caller as a rejection with the same error, exactly as without the option. A null get is impossible (absent objects throw and settle as failed); a false exists and a false delete are successful calls. getSignedUrl mints a URL, not a data transfer: its record carries lastDurationMs: null and lastBytes: null. lastBytes is the byte length of the last settled buffer for put and get, and null for the non-buffered operations. A getStream counts the open, not the drain. Records expire 60 seconds after their last observation and a snapshot whose freshest record is older than 30 seconds is stale. A clock failure latches collection-failed without changing any application result; closing the plugin detaches the collector. Object paths, stored bytes, content types, signed URLs and error text never reach the collector, the wire, or the client.