Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Media storage and ffmpeg

5. Media storage and ffmpeg

5.1 Local storage

STORAGE_PROVIDER=local
UPLOAD_DIRECTORY=/var/lib/rustiz/uploads
FRONTEND_URL=https://social.example.com
  • Files written by uploads and generated media use <yyyy>/<mm>/<dd>/<32-char-id>.<ext>.
  • The public URL is ${FRONTEND_URL}/uploads/<key>; the same process serves the directory with tower-http’s ServeDir.
  • It must be a persistent volume and must be writable by the service user.
  • With more than one API replica, use a shared volume or object storage. A local directory on one replica is invisible to the others.
  • A reverse proxy must not strip or rewrite /uploads.

5.2 S3 / R2

STORAGE_PROVIDER=cloudflare
CLOUDFLARE_ACCOUNT_ID=...
CLOUDFLARE_ACCESS_KEY=...
CLOUDFLARE_SECRET_ACCESS_KEY=...
CLOUDFLARE_BUCKETNAME=rustiz-media
CLOUDFLARE_BUCKET_URL=https://media.example.com
CLOUDFLARE_REGION=auto

Or:

STORAGE_PROVIDER=s3
AWS_BUCKETNAME=rustiz-media
AWS_REGION=eu-west-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_BUCKET_URL=https://media.example.com
# AWS_ENDPOINT=http://minio:9000 for MinIO

The bucket must be publicly readable over HTTPS (or fronted by a public CDN). Some providers — TikTok and TikTok Business in particular — fetch media by URL (pull_from_url) rather than accepting an upload, so a private or loopback /uploads URL cannot work for them. Public object storage is the robust choice for those providers. The media host must also be allowed in the platform’s developer app where the platform asks for verified domains.

Object keys differ slightly: with object storage, API uploads use <32-char-id>.<ext> while generated media (converted JPEGs, thumbnails) uses <yyyy>/<mm>/<dd>/<id>.<ext>. Both are addressed through CLOUDFLARE_BUCKET_URL / AWS_BUCKET_URL.

R2 lifecycle rules are bucket-wide (optionally prefix-filtered) and cannot ask SQLite which objects a post still references. Therefore:

  1. Always add a rule to abort incomplete multipart uploads, e.g. after 1 day. Rustiz streams uploads through object_store; a crashed process can otherwise leave billable parts behind.
  2. If you need a retention policy, do not put a blanket 30-day expiration on the media prefix. Scheduled and published posts older than 30 days still reference those objects, and deleting them breaks previews, reposts and platform pulls. A 30-day rule is appropriate only for a dedicated scratch or trash prefix.
  3. A safe pattern is: run a reconciliation job that copies objects no longer referenced by Media/Post rows into trash/<yyyy>/<mm>/, then apply a 30-day expiration to trash/ and let R2 delete it. Keep live media in the normal prefix with no expiry.
  4. If a compliance policy requires deleting all media after 30 days, export or archive it first and accept that old posts can no longer be reposted or previewed.

5.4 ffmpeg and video thumbnails

Video thumbnails are generated by shelling out to ffmpeg:

ffmpeg -y -ss <seconds> -i <temp-input> -frames:v 1 -q:v 2 <temp-output.jpg>

The engine first runs ffmpeg -version. If the binary is missing, it logs ffmpeg not available; skipping video thumbnail and marks the media ready without a thumbnail; publishing is not failed. FFMPEG_BINARY can point at an absolute path.

ffmpeg is not needed for PNG→JPEG conversion: providers with convert_to_jpeg (TikTok and TikTok Business) are handled by the image crate at JPEG quality 100. That conversion does need UPLOAD_DIRECTORY or object storage, because the converted bytes are written back through the same storage layer.

The stock Docker image is distroless and has no ffmpeg. Two ways to add it:

Static binary (recommended; keeps distroless):

FROM rust:1-slim AS build
WORKDIR /src
COPY . .
RUN cargo build --release --locked -p rustiz

FROM gcr.io/distroless/cc-debian12
COPY --from=mwader/static-ffmpeg:9.0.2-1 /ffmpeg /usr/local/bin/ffmpeg
COPY --from=build /src/target/release/rustiz /rustiz
ENV DATABASE_URL=sqlite:///data/rustiz.db UPLOAD_DIRECTORY=/data/uploads PORT=5000
VOLUME /data
EXPOSE 5000
ENTRYPOINT ["/rustiz"]
CMD ["serve"]

Pin the mwader/static-ffmpeg tag; the latest image is large (hundreds of MiB). The static binary has no external library dependencies and works on the distroless glibc base.

Debian runtime:

FROM debian:bookworm-slim
RUN apt-get update \
 && apt-get install -y --no-install-recommends ffmpeg ca-certificates \
 && rm -rf /var/lib/apt/lists/*
COPY --from=build /src/target/release/rustiz /usr/local/bin/rustiz
# ... same ENV/ENTRYPOINT as above

This is simpler but larger and gives the container a shell and package manager. Whichever path you choose, keep /data writable and persistent.

5.5 Upload validation rules

POST /public/v1/upload (multipart field file) and POST /public/v1/upload-from-url accept only:

png jpg jpeg gif webp mp4 mov webm m4v pdf
  • The extension comes from the filename, or from the URL path for upload-from-url; a URL without a recognised extension is rejected.
  • The body limit is MAX_UPLOAD_MB (default 1024 MiB). The reverse proxy and Cloudflare have their own, often lower, limits.
  • Remote uploads are fetched through the SSRF-safe client; only http/https is allowed. The guard blocks loopback, private, link-local and metadata addresses unless ALLOW_PRIVATE_NETWORK=true.
  • Local media reads are confined to UPLOAD_DIRECTORY with lexical path normalisation; ../ and sibling prefixes such as uploads-evil are rejected.
  • Remote reads have a 120-second timeout and one retry on timeout. Range requests are required by some providers; S3/R2 serve them natively, and local files are read with seek.
  • Deleting media is a soft delete: the database row is marked deleted but the stored file is kept (a published post may still reference it). Reclaiming storage is an operator task; see §5.3.