fediarchiver

About

• Scans and records fediverse posts
• Scans can be limited to specific users, or be ran in spider mode
• Submits scanned posts to archival services (archive.is, archive.org)
• Web front-end for easy searching available

Installation

1. Create new PostgreSQL user and database for fediarchiver
2. git clone https://git.kiwifarms.net/CrunkLord420/fediarchiver.git && cd fediarchiver
3. go build
4. cp config.json-example config.json and edit it
5. ./fediarchiver

Notes

• UserIDs can be obtained by getting a status eg. 9koPCTgOSOoI3Wcj9k from https://kiwifarms.cc/api/v1/statuses/9mP0jdcQ3ahnpAZqYC
• rate_limit is milliseconds
• http_timeout is seconds
• You will want to increase your File Descriptor Limit if running in spider mode
• Early rapid dev work in progress, expect bugs, non-sane data and DB schema changes. Watch the git log.
• Proxy can be set via environmental variable HTTP_PROXY eg. HTTP_PROXY=socks5://127.0.0.1:9050 ./fediarchiver
• Only tested with Pleroma, Gab-fork and Mastodon. Please report bugs for others.

Requirements

• Golang
• PostgreSQL

License