What is MagnetmeBot?
Direct Answer: MagnetmeBot is a respectful web crawler operated by Magnet.me to discover and update career-related content.
MagnetmeBot is the official web crawler used by Magnet.me to discover and index career-related content, ensuring the platform stays up-to-date with the latest vacancies, events, and employer information. It is designed to crawl websites efficiently and respectfully, respecting the robots.txt protocol and supporting various directives.
User-Agent Identification
The following user-agent strings identify MagnetmeBot in your live traffic data:
Magnet.me-web/1.0 (+https://magnet.me/bot.html)
robots.txt Rules for MagnetmeBot
Respects robots.txt: Yes
Use the following robots.txt rules to control MagnetmeBot access:
# Block MagnetmeBot
User-agent: MagnetmeBot
Disallow: /
# Allow MagnetmeBot
User-agent: MagnetmeBot
Allow: /Robots.txt is a directive, not a barrier
Magnet.me states that MagnetmeBot respects robots.txt. However, configuration mistakes, caching delays, and edge cases mean your directives may not always be followed as expected. Live traffic verification confirms whether MagnetmeBot actually obeys your rules in practice.
Need continuous verification across 500+ bots? Can AI See It automates this.
Crawl Behavior
Frequency:Automatic Crawl Speed Adjustment
Request Pattern:Focuses On The Main Version Of A Page (Canonical Version), Follows Links, And Scrolls
JavaScript Rendering:Yes — this bot can execute JavaScript and render pages.
Official Documentation Quotes
"MagnetmeBot is the official web crawler used by Magnet.me to discover and index career-related content."
Crawl Activity Index
Relative crawl activity for MagnetmeBot over the past 28 days. Higher values indicate increased crawling intensity compared to the period baseline.
View recent activity data (last 7 days)
| Date | Activity Index |
|---|---|
| Mar 28, 2026 | 4.3 |
| Mar 29, 2026 | 3.8 |
| Mar 30, 2026 | 2.0 |
| Mar 31, 2026 | 5.0 |
| Apr 1, 2026 | 8.0 |
| Apr 2, 2026 | 6.7 |
| Apr 3, 2026 | 5.7 |
Source: Cloudflare Radar
Why track MagnetmeBot traffic?
Identify and classify unknown crawler activity. MagnetmeBot may appear in your live traffic data with varying frequency. Tracking its behavior helps you decide whether to allow, throttle, or block it based on actual data.
Protect your crawl budget. Every bot request consumes server resources. Understanding what MagnetmeBot crawls helps you prioritize the crawlers that matter.
Log Verification
To verify MagnetmeBot traffic in your live traffic data:
- Search access logs for the user-agent strings listed above
- Check if the IP addresses match documented ranges (if provided by Magnet.me)
- Verify the crawl pattern matches documented behavior
- Use reverse DNS lookup for additional verification if available
IP Verification: Magnet.me provides official IP verification via Published IP ranges. View verification instructions →
IP addresses are published for verification
Note: Observed behavior in production environments may differ from official documentation. Live traffic monitoring provides the only reliable verification of actual bot behavior.
Undocumented Information
The following information is not officially documented for MagnetmeBot:
- crawl delay configuration
Official Documentation
View Official MagnetmeBot Documentation →
Information sourced from official documentation. Content generated with AI assistance.