CyberScraper-2077-HotFix

Changelog

v3 - 8/11/24

Added

• Tor support for .onion websites
• TorManager class for handling Tor connections
• TorScraper class for scraping .onion sites
• Tor configuration and setup in Dockerfile
• Health check mechanism for container monitoring

Changed

• Modified WebExtractor to handle both regular and .onion URLs
• Updated Dockerfile with proper Tor configuration
• Improved error handling for different URL types
• Updated proxy handling to avoid affecting non-onion traffic
• Modified startup scripts for better service management

Fixed

• Global socket modification affecting all traffic
• Proxy configuration issues
• Docker networking configuration
• Error handling for failed Tor connections

Known Issues

• Docker container: Regular scraping works but Tor functionality needs fixing

Todo

• Fix Tor connectivity in Docker container
• Improve Docker service startup sequence
• Add better error handling for Tor service failures
• Implement proper circuit isolation for Tor connections
• Add documentation for Tor feature usage
• Improve Docker networking configuration

Developer Notes

The Tor integration is currently working in the native installation but needs fixes in the Docker environment. When using Docker, currently:

• Regular website scraping works normally
• Tor connections need additional configuration
• Service startup sequence needs optimization

Installation Notes

Until the Docker issues are resolved, users are advised to:

1. Use native installation for full functionality
2. If using Docker, be aware that Tor features may not work properly
3. Follow updated installation instructions in README

Please report any issues or bugs on the GitHub issue tracker.

Full Changelog: CyberScraper-2077-v3...CyberScraper-2077-HotFix

CyberScraper-2077-v3

Added Onion Link Scraping Support

Full Changelog: CyberScraper-2077-v2...CyberScraper-2077-v3

CyberScraper-2077-v2

What's Changed

• fix ollama model init error by @scattter in #21

Full Changelog: latest...CyberScraper-2077-v2