File Management

The bot provides comprehensive file management capabilities for the IrregularChat community, including file search, automatic organization, and cloud storage integration with RustFS S3 (primary) and pCloud (legacy fallback).

File Search (!files)

Search for files in the IrregularChat shared archive.

Basic Usage

!files <query>         - Search filenames for query
!files -g <keyword>    - Search file contents (grep)
!files -s <query>      - Smart AI-powered search
!files --cat <name>    - Browse a specific category
!files --list          - List all categories

Examples

!files drone           - Find files with "drone" in name
!files -g FPV manual   - Find content mentioning "FPV manual"
!files -s UAV tactics  - Smart search with AI-generated synonyms
!files --cat Tech      - Browse files in the Tech category

Search Operators

The file search supports advanced query operators:

Operator	Example	Description
OR	drone OR uav	Match files with either term
”quotes”	"drone strike"	Exact phrase match
NOT	drone NOT toy	Exclude files with “toy”
-term	drone -toy	Exclude files with “toy”
(default)	drone attack	Files must contain both terms

Selecting Results

After a search returns results, you can:

1. Reply with a number to get a direct download link for that file (RustFS presigned URL)
2. Reply tldr N to get an AI summary of PDF file N

Note: Download links expire after 15 minutes for security. The bot maintains your search session for 10 minutes, allowing quick follow-up interactions.

Automatic File Organization

When files are shared in Signal groups, the bot automatically:

1. Scans for viruses using ClamAV (optional)
2. Normalizes filenames with consistent formatting
3. Categorizes files based on group name and content keywords
4. Copies to archive in the IrregularChat directory structure
5. Syncs to pCloud via rclone for cloud backup

Directory Structure

Files are organized into categories based on the Signal group they’re shared in:

Group Topic	Archive Directory
AI/ML, NLP, Machine Learning	AI-ML
Unmanned Systems, Drones, UAV, FPV	UnmannedSystems
Counter-UAS, C-UAS	UnmannedSystems/Counter_Unmanned_Systems
DragonOS, RF, SDR	DragonOS
Cyber, Security, Infosec	Cybersecurity
Red Team, Pentest	Red Teaming
Blue Team, SOC, Defensive	Blue Team
OSINT, Research, Intel	Research
Influence, IWAR, PSYOP	Influence
Space	Space
Tech, Technology	Tech
Business, Startup	Business
Fabrication, 3D Print	Fabrication
Leadership	Leadership
Off Topic, General	Off topic

Subdirectory Routing

Files are further categorized into subdirectories based on:

Content Keywords:

• Russia/Russian content -> Adversary/Russia
• China/Chinese content -> Adversary/China
• Reports -> Reports
• Analysis -> Analysis
• Guides -> Documents/Guides
• Policy/Doctrine -> Documents/Policy_and_Doctrine

File Types:

• Videos (.mp4, .mov, .mkv) -> Audio_Video
• Audio (.mp3, .wav, .m4a) -> Audio_Video
• Documents (.pdf, .doc, .ppt) -> Documents
• Images (.png, .jpg, .gif) -> Images

Filename Normalization

Files are renamed for consistency:

• Date prefix added: 2024-01-15_original_name.pdf
• Spaces converted to underscores
• Special characters removed
• Length limited to 200 characters

---

Technical Infrastructure

This section describes the technical architecture for file storage, synchronization, and access.

Goal

When a user selects a file from !files search results, provide a direct link to that file. This is achieved using MinIO S3-compatible storage with presigned URLs that provide secure, time-limited (15 min) direct downloads.

Architecture Overview

┌─────────────────────┐     ┌──────────────────────┐     ┌─────────────────┐
│   Signal Groups     │────▶│  Signal Bot (Docker) │────▶│  Local Archive  │
│  (file uploads)     │     │  /app/irregularchat  │     │  /datadrive/    │
└─────────────────────┘     └──────────────────────┘     │  IrregularChat/ │
                                                          └────────┬────────┘
                                                                   │
                                    ┌──────────────────────────────┼──────────────────────────────┐
                                    │                              │                              │
                                    ▼                              ▼                              ▼
                           ┌─────────────────┐            ┌─────────────────┐           ┌─────────────────┐
                           │     MinIO S3    │            │     pCloud      │           │  Signal Bot     │
                           │   (primary)     │            │  (legacy sync)  │           │  Direct Access  │
                           │  Presigned URLs │            │  rclone backup  │           │  (local files)  │
                           │  s3.irregular   │            └─────────────────┘           └─────────────────┘
                           │    .chat        │
                           └─────────────────┘

MinIO S3 Storage (Primary)

MinIO provides S3-compatible object storage for direct file downloads:

• Public URL: https://s3.irregular.chat
• Bucket: irregularchat
• Link Type: Presigned URLs with 24-hour expiration
• Access: Anyone with the presigned URL can download (time-limited)

Note: RustFS was evaluated in Dec 2025 for its lower memory footprint (~74 MiB vs 16 GiB), but had issues with presigned URL signature validation via Cloudflare Tunnel. MinIO remains the recommended solution.

Advantages over pCloud:

• Direct file links (not folder navigation)
• No IP-binding restrictions
• Self-hosted, full control
• S3-compatible API

Docker Volume Mounting

The bot runs in Docker and accesses the file archive via a bind mount:

compose.yml (production):

services:
  signal-bot:
    volumes:
      # Mount the IrregularChat archive into the container
      - /datadrive/IrregularChat/Topics:/app/irregularchat

Host Path	Container Path	Purpose
/datadrive/IrregularChat/Topics	/app/irregularchat	Read/write access to file archive

The IRREGULARCHAT_PATH environment variable tells the bot where to find files:

IRREGULARCHAT_PATH=/app/irregularchat

rclone Configuration

rclone syncs the local archive to pCloud for cloud backup and public access.

rclone remote configuration:

[pcloud]
type = pcloud
token = {"access_token":"...", "token_type":"bearer", "expiry":"0001-01-01T00:00:00Z"}

Sync command:

# Sync local archive to pCloud
rclone sync /datadrive/IrregularChat pcloud:IrregularChat --progress

# One-way sync (local → cloud), preserving cloud files not in local
rclone copy /datadrive/IrregularChat pcloud:IrregularChat --progress

Sync schedule: Runs periodically via cron or systemd timer to keep cloud storage up-to-date.

pCloud Public Link

The IrregularChat folder on pCloud is shared via a public link:

• URL: https://u.pcloud.link/publink/show?code=kZMdb0VZ...
• Access: Anyone with the link can browse and download
• Limitation: Cannot deep-link to specific folders or files (see investigation below)

Current Limitation: Folder Navigation

Known Issue: pCloud public folder links do not support direct folder navigation via URL parameters. When you request a file link, you’ll receive:

• A link to the main public folder
• The exact filename and category to help you navigate manually

Link Format

Due to pCloud limitations, all links go to the root of the public folder. The response format is:

1. filename.pdf
   Category • Size • Date
   [pCloud URL]
   Look for: filename.pdf

---

pCloud URL Investigation Log

This section documents attempts to enable direct folder navigation in pCloud public folder links.

Problem Statement

When users search for files and request a link, they want to navigate directly to the folder containing the file, not the root of the public folder (IrregularChat/).

What Has Been Tried

1. Direct Download Links via getpublinkdownload API

Approach: Use pCloud’s API to generate direct download links for files.

const response = await fetch(
  `https://api.pcloud.com/getpublinkdownload?code=${PUBLIC_CODE}&fileid=${fileId}`
);
const data = await response.json();
// Returns: { hosts: ["..."], path: "/..." }
// Construct URL: https://{host}{path}

Result: FAILED

• API generates IP-bound download links
• Link only works from the server’s IP address
• Users get error: “This link was generated for another IP address”

Why: pCloud’s download API binds the generated URL to the requesting IP for security.

2. Path-Based Folder Navigation (#folder=path)

Approach: Use hash fragment with folder path string.

const folderUrl = `${PCLOUD_PUBLIC_URL}#folder=${encodeURIComponent(folderPath)}`;
// Example: https://u.pcloud.link/publink/show?code=...#folder=UnmannedSystems/Documents

Result: FAILED

• URL loads but navigates to root folder (IrregularChat/)
• Path strings are not recognized by pCloud’s public folder viewer

3. Numeric Folder ID Navigation (#folder=folderid)

Approach: Use pCloud’s numeric folder IDs instead of path strings.

// Build folder index from pCloud API
const response = await fetch(`https://api.pcloud.com/showpublink?code=${PUBLIC_CODE}`);
// Returns folder structure with numeric folderids
// Example: UnmannedSystems/Documents -> folderid=29123489281

const folderUrl = `${PCLOUD_PUBLIC_URL}#folder=${folderId}`;
// Example: https://u.pcloud.link/publink/show?code=...#folder=29123489281

Result: FAILED

• Index builds successfully (14,139 files, 1,445 folders)
• Numeric folderid is retrieved correctly
• URL still navigates to root folder (IrregularChat/)

What Hasn’t Been Tried

1. Query Parameter Instead of Hash (?folder=)

https://u.pcloud.link/publink/show?code=...&folder=29123489281

Untested - may or may not work with pCloud’s URL parser.

2. Different URL Structures

https://u.pcloud.link/publink/show?code=...&path=/UnmannedSystems/Documents
https://e.pcloud.link/publink/show?code=...#folder=29123489281
https://my.pcloud.com/#page=publink&code=...&folder=29123489281

3. pCloud Web App URL Format

Inspecting how pCloud’s own web interface constructs URLs when browsing folders.

4. File-Specific Links (#file=fileid)

https://u.pcloud.link/publink/show?code=...#file=12345678

5. Combined Folder + File Parameters

https://u.pcloud.link/publink/show?code=...#folder=29123489281&file=filename.pdf

6. pCloud’s publink API with Different Parameters

GET https://api.pcloud.com/showpublink?code=...&forcedownload=0&folderid=29123489281

7. Creating Per-Folder Public Links

Instead of one public link for the entire IrregularChat folder, create individual public links for each category folder. This would give direct access but require significant setup.

Current Implementation

Located in src/src/utils/file-search.ts:

// PCloudFolderIndex stores both files and folders
interface PCloudFolderIndex {
  files: Map<string, PCloudFileEntry>;   // path -> file entry
  folders: Map<string, number>;          // folderPath -> folderid
  lastUpdated: Date;
}

// Index is built from pCloud showpublink API
// Refreshes every 30 minutes
// Currently indexes 14,139 files and 1,445 folders

export async function getDirectDownloadLink(relativePath: string): Promise<{
  success: boolean;
  link?: string;
  filename?: string;
  error?: string;
}> {
  // Looks up numeric folderid from index
  // Generates URL with #folder=folderid
  // Currently doesn't work - navigates to root
}

Architecture Notes

pCloud API Endpoints Used:

• showpublink?code=... - Get public folder structure with all fileids and folderids
• getpublinkdownload?code=...&fileid=... - Get download link (IP-bound)

Index Statistics:

• Total files indexed: 14,139
• Total folders indexed: 1,445
• Index build time: ~1.3 seconds
• Index refresh interval: 30 minutes

Next Steps to Investigate

1. Inspect pCloud web UI - Use browser dev tools to see what URLs pCloud uses internally when navigating folders
2. Test URL variations - Try the untested URL formats listed above
3. Check pCloud documentation - Look for official docs on public folder URL parameters
4. Contact pCloud support - Ask if folder navigation is supported in public links
5. Consider alternatives - If pCloud doesn’t support this, consider alternative hosting with better URL support

---

Virus Scanning

When enabled, files are scanned with ClamAV before archiving:

• Clean files are archived normally
• Infected files are rejected and not stored
• Scan errors (ClamAV unavailable) allow the file through with a warning

Technical Details

File Index

The bot maintains an in-memory index of all files for fast searching:

• Index refreshes every hour
• Maximum 50,000 files indexed
• Stores filename, path, size, modified date, and category

pCloud Index

A separate index maps file paths to pCloud file IDs:

• Refreshes every 30 minutes
• Stores both file IDs and folder IDs
• Enables potential future folder navigation URLs

Search Sessions

Search results are cached in sessions:

• 10-minute expiration
• Links session to group/user context
• Supports follow-up queries without re-searching

Configuration

Environment variables for file management:

MinIO S3 Configuration (Primary)

Variable	Default	Description
MINIO_ENDPOINT	-	Internal MinIO API endpoint (e.g., http://minio-irregularchat:9000)
MINIO_ACCESS_KEY	-	MinIO access key (S3 compatible)
MINIO_SECRET_KEY	-	MinIO secret key (S3 compatible)
MINIO_BUCKET	irregularchat	MinIO bucket name
MINIO_REGION	us-east-1	MinIO region (required for S3 API)
MINIO_PUBLIC_URL	-	Public URL for presigned links (e.g., https://s3.irregular.chat)

File Archive Configuration

Variable	Default	Description
IRREGULARCHAT_PATH	/app/irregularchat	Base path for file archive

pCloud Configuration (Legacy Fallback)

Variable	Default	Description
PCLOUD_PUBLIC_URL	(configured)	pCloud public folder URL
PCLOUD_API_HOST	api.pcloud.com	pCloud API endpoint

Archive Statistics

Current archive size: 7.9 GB across 24 topic directories

Largest Categories

Directory	Size
UnmannedSystems	3.7 GB
Research	1.8 GB
Testing	1.3 GB
Dragon RF Room	229 MB
Influence	132 MB
Warfare	113 MB
Fabrication	100 MB

Related Commands

• All Commands - Complete command reference