S3 and compatible storage

Amazon S3, MinIO, Backblaze B2 and Wasabi.

Who can see it Not yet
Works with Amazon S3 · MinIO · Backblaze B2 · Wasabi
Set up in console

What it scans

Current objects in one bucket, optionally limited to a literal key prefix. Custom HTTPS endpoint_url is supported on customer-run agents, including S3-compatible services such as MinIO, Backblaze B2 and Wasabi. Compatibility requires HeadBucket, ListObjectsV2 and GetObject with range reads; test your service with a small prefix first.

What you need

An agent, the bucket name, region and list/read access. For compatible services, get the provider's S3 API endpoint and S3 key pair. Use that endpoint, not a console URL or a public download URL. Hosted agents accept only approved public AWS endpoints, not custom endpoints.

Step 1: Grant list and read access

In AWS IAM → Policies → Create policy → JSON, paste this policy. Replace example-scan-bucket and reports/ to match your bucket and prefix. Create a policy named MyrikeepReadReports. No write or delete permission is needed.

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketProbeAndList",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-scan-bucket"
    },
    {
      "Sid": "ReadSelectedPrefix",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::example-scan-bucket/reports/*"
    }
  ]
}

This is the list/read policy needed by the current connector. It probes the bucket with HeadBucket before listing. A prefix condition on ListBucket blocks that probe. Listing permission therefore covers object names in the bucket; object content access is restricted to reports/. Set the console prefix as well. For isolation of names and content, scan a dedicated bucket. For a whole bucket, change the object resource to arn:aws:s3:::example-scan-bucket/*. See HeadBucket permissions.

SSE-KMS objects may also require kms:Decrypt on the exact KMS key, with its key policy allowing the scan identity. Do not add broad KMS access. Compatible providers use their own bucket/key permissions; apply equivalent list and read restrictions there.

Step 2: Choose keys or workload identity

For AWS workload identity, attach this policy to the agent's instance/task role in IAM → Roles → select role → Add permissions. Select S3/Azure workload identity in Myrikeep and leave Credential name empty. The agent uses boto3's credential chain on its host. The role must be accessible inside the container. Hosted workload identity needs an operator's explicit opt-in.

For keys, create a dedicated identity in IAM → Users → Create user, without console access. Attach only this policy. Under Security credentials → Create access key, choose the use case that matches your host and copy the pair once. Use a provider-issued S3 key pair for compatible services. Temporary AWS keys also require their session token and expire; workload identity avoids manual renewal.

Step 3: Add and scan

In Connections → Add connection, choose AWS S3 bucket, enter bucket, prefix, region and optional custom endpoint, and assign your agent. Save the files below if using keys. Wait for present, then queue a scan in Scans.

Connection settings table

Console field Example Required or optional
Connection name Reports bucket Required; unique in workspace
Type AWS S3 bucket (s3) Required
Bucket example-scan-bucket Required; no s3://
Prefix reports/ Optional; blank scans whole bucket
Region eu-west-1 Optional; use your bucket/provider region
Endpoint URL https://minio.example.test Optional; HTTPS, no credentials/query/fragment
Agent Storage agent Required; same workspace
Credential mode Local credential catalog or S3/Azure workload identity Required
Credential name S3_REPORTS Required for catalog; empty for workload identity
Exclude patterns (View/edit) *.tmp Optional; one glob per line
Enabled (connection row) Enabled Disable to stop new scans

Secret file

For catalog keys, save /srv/myrikeep/secrets/S3_REPORTS:

json
{"access_key_id": "<S3_ACCESS_KEY_ID>", "secret_access_key": "<S3_SECRET_ACCESS_KEY>"}

For temporary credentials, include "session_token": "<AWS_SESSION_TOKEN>". Merge this catalog entry:

json
{"version": 1, "credentials": {"S3_REPORTS": {"type": "s3", "source": {"kind": "file_json", "path": "/run/secrets/S3_REPORTS"}}}}

Follow Agent credentials for file modes and mounts. Workload identity needs no catalog entry or secret JSON.

Scope, exclusions and schedules

Prefixes are literal and case-sensitive: reports/ differs from reports. Exclusions such as *.tmp, reports/cache/* and */scratch/* match full keys and basenames before content is read. A prefix is scan scope, not an IAM grant. See Getting started for schedule cadence.

"Who can see it" support

Sharing checks are not available for this source yet. Public access, bucket policies, ACLs and cross-account access are not assessed in phase 1. LastModified supports age review; it is not last access. See Who can see it.

Troubleshooting

State or error What to do
unknown Check agent heartbeat, assignment and current connection revision.
missing / credential_missing Match alias/type and mount; for workload identity check the SDK can obtain host credentials.
Invalid / credential_config_invalid Check exact JSON, modes and nonempty fields; temporary keys need a session token.
present Catalog resolved, or workload credentials initialized. Bucket authorization is tested by a scan.
source_error / cannot open S3 client for bucket Check bucket, region, endpoint and HeadBucket/list permission in local agent diagnostics.
AccessDenied / cannot list / cannot read Check bucket policy, prefix read resource, explicit denies and KMS key permissions.
ExpiredToken / InvalidAccessKeyId / SignatureDoesNotMatch Renew keys/token; check provider region, clock and endpoint.
Hosted source or endpoint prohibited Use a customer-run agent for a custom/private S3 endpoint.
Paused / Migration paused / Disabled Resume agent, assign and save the connection, or enable it.
signin_required Applies to personal drives; S3 uses keys or workload identity.

Limits

No bucket discovery, object version history, Glacier restore, policy edits or writes. Folder markers are skipped. Enumeration stops with an error above one million listed objects; narrow the prefix. Agent sampling limits can be lower. Not all bytes are classified. Modern Office/PDF text and common text formats have parsers; other files use strings-only checks. Text in images and scanned PDFs needs OCR, an optional premium feature a workspace owner enables; it runs inside your agent with the OCR agent image. Inspect Coverage.