Add binary file filtering, prevent AI hallucination, and implement modern web UI

[Copilot]

Overview

This PR addresses three critical improvements to DocSifter:

1. Intelligent file type filtering - Only process text-based documents with AI, not binary files
2. Conservative AI tagging - Prevent hallucination by using strict prompting and validation
3. Modern web interface - Replace tkinter UI with a beautiful Flask-based web application

Problem Statement

The original implementation had several issues:

• AI attempted to tag binary files (images, videos, audio) despite being unable to read their content
• AI would sometimes hallucinate tags based on guesses rather than clear evidence
• The tkinter desktop UI was basic and dated

Solution

1. Binary File Filtering

Added is_text_based_file() method that distinguishes between text-based and binary files:

Text-based files (AI-enabled):

• Documents: PDF, Word, Excel, PowerPoint
• Code: Python, JavaScript, Java, HTML, CSS, etc.
• Data: JSON, YAML, CSV, XML, Markdown

Binary files (heuristic-only):

• Images: JPG, PNG, GIF
• Videos: MP4, MOV
• Audio: MP3, WAV
• Archives: ZIP, RAR

def is_text_based_file(self, file_ext: str) -> bool:
    """Check if file is text-based (not binary like images, videos, audio)"""
    text_extensions = {
        '.pdf', '.doc', '.docx', '.txt', '.rtf', '.odt',
        '.xlsx', '.xls', '.csv', '.py', '.js', '.html', # ...
    }
    return file_ext in text_extensions

Result: Binary files like photo.jpg now only get basic tags ["image", "photo"] instead of hallucinated content descriptions.

2. Conservative AI Prompting

Updated the AI system to be more conservative and evidence-based:

Changes:

• Reduced temperature from 0.7 to 0.3 for more deterministic responses
• Updated system prompt: "You are a conservative file tagger. Only tag based on clear evidence in filename and extension. Never guess or hallucinate."
• Added response validation to filter invalid tags
• Returns empty array if tags cannot be confidently determined

Prompt examples:

Example for "invoice_jan2024.pdf": ["document", "pdf", "financial", "invoice", "2024"]
Example for "document.pdf": ["document", "pdf"]

This ensures the AI only tags what it can clearly infer from metadata, not content it cannot see.

3. Modern Web UI

Created a beautiful, responsive web interface using Flask + HTML/CSS/JS:

Backend (web_ui.py):

• Flask REST API with endpoints for scanning, searching, and status
• Background threading for non-blocking scans
• Real-time progress tracking

Frontend (templates/index.html):

• Modern gradient design with smooth animations
• Real-time scan progress with visual progress bar
• Instant search with 300ms debounce
• Card-based layout with styled tag pills
• Fully responsive for mobile and desktop
• Status badges showing AI availability

API Endpoints:

• GET /api/status - System status and configuration
• POST /api/scan - Start scanning a folder
• GET /api/progress - Get real-time scan progress
• GET /api/search?q=query - Search tagged files
• GET /api/files - Get all tagged files

Screenshots

Initial State

Clean, modern interface with status badges and folder selection.

After Scanning Files

Notice the intelligent tagging:

• Binary files (photo.jpg, video.mp4, screenshot.png) → Only basic tags like "image", "photo", "video"
• Text documents (PDFs, Word files) → Detailed tags like "document", "pdf", "financial", "invoice", "2024"

No hallucinated content for files the AI cannot read!

Search Functionality

Real-time search filters 8 files down to 2 matching "image" tag with instant visual feedback.

Usage

Start the web UI (recommended):

python web_ui.py
# Open http://localhost:5000 in your browser

Or use the desktop UI (legacy):

python docsifter.py

Technical Details

New Files:

• web_ui.py - Flask backend with DocSifterService class
• templates/index.html - Modern single-page web application
• test_web_ui.py - Test suite for web UI and binary file detection
• USAGE.md - Comprehensive usage guide

Modified Files:

• docsifter.py - Added is_text_based_file() method and improved generate_tags_ai() prompt
• requirements.txt - Added flask>=2.3.0 dependency
• README.md - Updated with web UI instructions

Backward Compatibility:
The original tkinter UI is preserved and fully functional for users who prefer desktop applications.

Testing

All tests pass successfully:

• ✅ Original test suite (test_docsifter.py) - 100% pass
• ✅ New web UI tests (test_web_ui.py) - 100% pass
• ✅ Binary file detection validation
• ✅ AI prompt conservativeness validation
• ✅ Manual web UI testing with real files

Example test results:

Testing tag generation for binary files...
  Image tags: ['image', 'photo']
  PDF tags: ['document', 'pdf', 'financial', 'invoice', 'billing', '2024', 'recent']
  Video tags: ['video', 'media']
✓ Binary file tag generation test passed!

Benefits

1. Accuracy: No more hallucinated tags for binary files
2. Trust: Conservative AI only tags what it can confidently determine
3. UX: Modern, responsive web interface that's beautiful and functional
4. Flexibility: Users can choose between web UI or desktop UI
5. Maintainability: Clean separation of concerns with REST API architecture

Original prompt

1. don't parse images for tags. only inspect text based docs not binary
2. also don't hallucinate or guess tags if you cant figure out
3. instead of opening a python UI window which looks ugly, have a local UI open up with HTML/CSS/JS using a latest stable framework

1Instead of opening a

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.