Forums

The forums ran from 2008-2020 and are now closed and viewable here as an archive.

Home › Forums › Other › robots.txt

  • This topic is empty.
Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • September 3, 2013 at 4:55 pm #148962
    John
    Participant

    Is this the proper way to set up multiple entries?

    User-agent: aboundex
    Disallow: /
    User-agent: ahrefsbot
    Disallow: /
    User-agent: baiduspider
    Disallow: /
    User-agent: docomo
    Disallow: /
    User-agent: ezooms
    Disallow: /
    User-agent: ichiro
    Disallow: /
    User-agent: libwww-perl
    Disallow: /
    User-agent: linux
    Disallow: /
    User-agent: meanpathbot
    Disallow: /
    User-agent: mj12bot
    Disallow: /
    User-agent: pagesinventory
    Disallow: /
    User-agent: python-urllib
    Disallow: /
    User-agent: sistrix
    Disallow: /

    September 3, 2013 at 5:41 pm #148965
    jurotek
    Participant

    Something like this

    User-agent: *
Disallow: /value
Disallow: /value
and so on
  • Author
    Posts
Viewing 2 posts - 1 through 2 (of 2 total)
  • The forum ‘Other’ is closed to new topics and replies.