Skip to content

File Management

The bot provides comprehensive file management capabilities for the IrregularChat community, including file search, automatic organization, and community archive links through the signed-in member portal.

Search for files in the IrregularChat shared archive.

!files <query> - Search filenames for query
!files -g <keyword> - Search file contents (grep)
!files -s <query> - Smart AI-powered search
!files --cat <name> - Browse a specific category
!files --list - List all categories
!files drone - Find files with "drone" in name
!files -g FPV manual - Find content mentioning "FPV manual"
!files -s UAV tactics - Smart search with AI-generated synonyms
!files --cat Tech - Browse files in the Tech category

The file search supports advanced query operators:

OperatorExampleDescription
ORdrone OR uavMatch files with either term
“quotes”"drone strike"Exact phrase match
NOTdrone NOT toyExclude files with “toy”
-termdrone -toyExclude files with “toy”
(default)drone attackFiles must contain both terms

After a search returns results, you can:

  1. Reply with a number to get a direct download link for that file (RustFS presigned URL)
  2. Reply tldr N to get an AI summary of PDF file N

Note: Download links expire after 15 minutes for security. The bot maintains your search session for 10 minutes, allowing quick follow-up interactions.

When files are shared in Signal groups, the bot automatically:

  1. Scans for viruses using ClamAV (optional)
  2. Normalizes filenames with consistent formatting
  3. Categorizes files based on group name and content keywords
  4. Copies to archive in the IrregularChat directory structure
  5. Syncs to pCloud via rclone for cloud backup

Files are organized into categories based on the Signal group they’re shared in:

Group TopicArchive Directory
AI/ML, NLP, Machine LearningAI-ML
Unmanned Systems, Drones, UAV, FPVUnmannedSystems
Counter-UAS, C-UASUnmannedSystems/Counter_Unmanned_Systems
DragonOS, RF, SDRDragonOS
Cyber, Security, InfosecCybersecurity
Red Team, PentestRed Teaming
Blue Team, SOC, DefensiveBlue Team
OSINT, Research, IntelResearch
Influence, IWAR, PSYOPInfluence
SpaceSpace
Tech, TechnologyTech
Business, StartupBusiness
Fabrication, 3D PrintFabrication
LeadershipLeadership
Off Topic, GeneralOff topic

Files are further categorized into subdirectories based on:

Content Keywords:

  • Russia/Russian content -> Adversary/Russia
  • China/Chinese content -> Adversary/China
  • Reports -> Reports
  • Analysis -> Analysis
  • Guides -> Documents/Guides
  • Policy/Doctrine -> Documents/Policy_and_Doctrine

File Types:

  • Videos (.mp4, .mov, .mkv) -> Audio_Video
  • Audio (.mp3, .wav, .m4a) -> Audio_Video
  • Documents (.pdf, .doc, .ppt) -> Documents
  • Images (.png, .jpg, .gif) -> Images

Files are renamed for consistency:

  • Date prefix added: 2024-01-15_original_name.pdf
  • Spaces converted to underscores
  • Special characters removed
  • Length limited to 200 characters

This section describes the technical architecture for file storage, synchronization, and access.

When a user selects a file from !files search results, provide a direct link to that file. This is achieved using MinIO S3-compatible storage with presigned URLs that provide secure, time-limited (15 min) direct downloads.

┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
│ Signal Groups │────▶│ Signal Bot (Docker) │────▶│ Local Archive │
│ (file uploads) │ │ /app/irregularchat │ │ /datadrive/ │
└─────────────────────┘ └──────────────────────┘ │ IrregularChat/ │
└────────┬────────┘
│
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ MinIO S3 │ │ pCloud │ │ Signal Bot │
│ (primary) │ │ (legacy sync) │ │ Direct Access │
│ Presigned URLs │ │ rclone backup │ │ (local files) │
│ s3.irregular │ └─────────────────┘ └─────────────────┘
│ .chat │
└─────────────────┘

MinIO provides S3-compatible object storage for direct file downloads:

  • Public URL: https://s3.irregular.chat
  • Bucket: irregularchat
  • Link Type: Presigned URLs with 24-hour expiration
  • Access: Anyone with the presigned URL can download (time-limited)

Note: RustFS was evaluated in Dec 2025 for its lower memory footprint (~74 MiB vs 16 GiB), but had issues with presigned URL signature validation via Cloudflare Tunnel. MinIO remains the recommended solution.

Advantages over pCloud:

  • Direct file links (not folder navigation)
  • No IP-binding restrictions
  • Self-hosted, full control
  • S3-compatible API

The bot runs in Docker and accesses the file archive via a bind mount:

compose.yml (production):

services:
signal-bot:
volumes:
# Mount the IrregularChat archive into the container
- /datadrive/IrregularChat/Topics:/app/irregularchat
Host PathContainer PathPurpose
/datadrive/IrregularChat/Topics/app/irregularchatRead/write access to file archive

The IRREGULARCHAT_PATH environment variable tells the bot where to find files:

Terminal window
IRREGULARCHAT_PATH=/app/irregularchat

rclone keeps the historical pCloud archive available as a private import source.

rclone remote configuration:

[pcloud]
type = pcloud
token = {"access_token":"...", "token_type":"bearer", "expiry":"0001-01-01T00:00:00Z"}

Sync command:

Terminal window
# Sync local archive to pCloud
rclone sync /datadrive/IrregularChat pcloud:IrregularChat --progress
# One-way sync (local → cloud), preserving cloud files not in local
rclone copy /datadrive/IrregularChat pcloud:IrregularChat --progress

Sync schedule: Runs periodically via cron or systemd timer to keep cloud storage up-to-date.

The bot now links archived files directly to the community file drive. Links to individual files open through the portal’s private Silo gateway; a member must sign in to view or download. Folder results open the archive with that folder path prefilled in search. Approved chat uploads use the portal’s chat-file gateway. Members can star files and copy a sign-in link from the drive.

The pCloud remote remains an archive import source. Public pCloud links are being retired. Earlier chat messages with those links may stop working; search again with !files for a current link.


This section documents attempts to enable direct folder navigation in pCloud public folder links.

When users search for files and request a link, they want to navigate directly to the folder containing the file, not the root of the public folder (IrregularChat/).

Section titled “1. Direct Download Links via getpublinkdownload API”

Approach: Use pCloud’s API to generate direct download links for files.

const response = await fetch(
`https://api.pcloud.com/getpublinkdownload?code=${PUBLIC_CODE}&fileid=${fileId}`
);
const data = await response.json();
// Returns: { hosts: ["..."], path: "/..." }
// Construct URL: https://{host}{path}

Result: FAILED

  • API generates IP-bound download links
  • Link only works from the server’s IP address
  • Users get error: “This link was generated for another IP address”

Why: pCloud’s download API binds the generated URL to the requesting IP for security.

2. Path-Based Folder Navigation (#folder=path)

Section titled “2. Path-Based Folder Navigation (#folder=path)”

Approach: Use hash fragment with folder path string.

const folderUrl = `${PCLOUD_PUBLIC_URL}#folder=${encodeURIComponent(folderPath)}`;
// Example: https://u.pcloud.link/publink/show?code=...#folder=UnmannedSystems/Documents

Result: FAILED

  • URL loads but navigates to root folder (IrregularChat/)
  • Path strings are not recognized by pCloud’s public folder viewer

3. Numeric Folder ID Navigation (#folder=folderid)

Section titled “3. Numeric Folder ID Navigation (#folder=folderid)”

Approach: Use pCloud’s numeric folder IDs instead of path strings.

// Build folder index from pCloud API
const response = await fetch(`https://api.pcloud.com/showpublink?code=${PUBLIC_CODE}`);
// Returns folder structure with numeric folderids
// Example: UnmannedSystems/Documents -> folderid=29123489281
const folderUrl = `${PCLOUD_PUBLIC_URL}#folder=${folderId}`;
// Example: https://u.pcloud.link/publink/show?code=...#folder=29123489281

Result: FAILED

  • Index builds successfully (14,139 files, 1,445 folders)
  • Numeric folderid is retrieved correctly
  • URL still navigates to root folder (IrregularChat/)

1. Query Parameter Instead of Hash (?folder=)

Section titled “1. Query Parameter Instead of Hash (?folder=)”
https://u.pcloud.link/publink/show?code=...&folder=29123489281

Untested - may or may not work with pCloud’s URL parser.

https://u.pcloud.link/publink/show?code=...&path=/UnmannedSystems/Documents
https://e.pcloud.link/publink/show?code=...#folder=29123489281
https://my.pcloud.com/#page=publink&code=...&folder=29123489281

Inspecting how pCloud’s own web interface constructs URLs when browsing folders.

https://u.pcloud.link/publink/show?code=...#file=12345678
https://u.pcloud.link/publink/show?code=...#folder=29123489281&file=filename.pdf
Section titled “6. pCloud’s publink API with Different Parameters”
GET https://api.pcloud.com/showpublink?code=...&forcedownload=0&folderid=29123489281

Instead of one public link for the entire IrregularChat folder, create individual public links for each category folder. This would give direct access but require significant setup.

packages/shared-utils/src/bot-utils/file-search.ts generates signed-in portal links from archive paths. The portal checks each file against the Silo manifest on access. The investigation above is retained as history; its public-link code and folder URL examples are no longer used by the bot.


When enabled, files are scanned with ClamAV before archiving:

  • Clean files are archived normally
  • Infected files are rejected and not stored
  • Scan errors (ClamAV unavailable) allow the file through with a warning

The bot maintains an in-memory index of all files for fast searching:

  • Index refreshes every hour
  • Maximum 50,000 files indexed
  • Stores filename, path, size, modified date, and category

A separate index maps file paths to pCloud file IDs:

  • Refreshes every 30 minutes
  • Stores both file IDs and folder IDs
  • Enables potential future folder navigation URLs

Search results are cached in sessions:

  • 10-minute expiration
  • Links session to group/user context
  • Supports follow-up queries without re-searching

Environment variables for file management:

VariableDefaultDescription
MINIO_ENDPOINT-Internal MinIO API endpoint (e.g., http://minio-irregularchat:9000)
MINIO_ACCESS_KEY-MinIO access key (S3 compatible)
MINIO_SECRET_KEY-MinIO secret key (S3 compatible)
MINIO_BUCKETirregularchatMinIO bucket name
MINIO_REGIONus-east-1MinIO region (required for S3 API)
MINIO_PUBLIC_URL-Public URL for presigned links (e.g., https://s3.irregular.chat)
VariableDefaultDescription
IRREGULARCHAT_PATH/app/irregularchatBase path for file archive
VariableDefaultDescription
The bot no longer needs a public pCloud link to generate file search results.

The 2026-10-16 Silo import inventory contains 15,503 files and 166,872,255,447 bytes.

DirectorySize
UnmannedSystems3.7 GB
Research1.8 GB
Testing1.3 GB
Dragon RF Room229 MB
Influence132 MB
Warfare113 MB
Fabrication100 MB

Recent changes

  1. docs(files): replace archive public links with member routes @sac
  2. refactor(structure): flatten workers into apps/ directory gitayam