### Pages with rec parameter - IAI Recommendation System
User-agent: *
Disallow: /*?rec=*
Disallow: /*&rec=*
Sitemap: https://bolf.lt/sitemap.xml.gz

### Automatically banned scanners and crawlers section ###
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: wotbox
Disallow: /
### Section end ###

User-agent: TwengaBot-Discover
Disallow: /
User-agent: TwengaBot/2.0
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: TwengaBot-2.0
Disallow: /

User-agent: baiduspider
Disallow: /

User-agent: Yandex
Disallow: /
User-agent: *
Disallow: /Disallow: /checkup.php
Disallow: /*?filter_traits 
Disallow: /*?selected_size