Baidu spider

Options
mindlube
mindlube
edited December 2012 in Photon Server
Anyone know how to keep search engines from crawling and sending requests? I guess it's only once per day so can be ignored? Baidu must be crawling DNS entries I guess
2812: 03:14:52.872 - CTCPWebSocketServer::ReadCompleted - Exception - CRequestHeaders::ValidateBasicRequestHeaders() - Handshake headers do not include header: "upgrade" with value: "WebSocket".
"accept" - "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/msword, */*" - 61 63 63 65 70 74 00 - accept. -
"host" - "www.baidu.com" - 68 6F 73 74 00 - host. -
"proxy-connection" - "Close" - 70 72 6F 78 79 2D 63 6F 6E 6E 65 63 74 69 6F 6E 00 - proxy-connection. -
"referer" - "http://www.baidu.com/" - 72 65 66 65 72 65 72 00 - referer. -
"user-agent" - "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)" - 75 73 65 72 2D 61 67 65 6E 74 00 - user-agent. -
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/msword, */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
Referer: http://www.baidu.com/
Host: http://www.baidu.com
Proxy-Connection: Close

1588: 03:14:53.698 - CTCPWebSocketServer::ReadCompleted - Exception - CRequestHeaders::ValidateBasicRequestHeaders() - Handshake headers do not include header: "upgrade" with value: "WebSocket".
"accept" - "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/msword, */*" - 61 63 63 65 70 74 00 - accept. -
"host" - "www.baidu.com" - 68 6F 73 74 00 - host. -
"proxy-connection" - "Close" - 70 72 6F 78 79 2D 63 6F 6E 6E 65 63 74 69 6F 6E 00 - proxy-connection. -
"referer" - "http://www.baidu.com/" - 72 65 66 65 72 65 72 00 - referer. -
"user-agent" - "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)" - 75 73 65 72 2D 61 67 65 6E 74 00 - user-agent. -
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/msword, */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
Referer: http://www.baidu.com/
Host: http://www.baidu.com
Proxy-Connection: Close

1588: 09:06:02.559 - CTCPWebSocketServer::ReadCompleted - Exception - CRequestHeaders::ValidateBasicRequestHeaders() - Handshake headers do not include header: "upgrade" with value: "WebSocket".
"host" - "www.baidu.com" - 68 6F 73 74 00 - host. -
"proxy-connection" - "Keep-Alive" - 70 72 6F 78 79 2D 63 6F 6E 6E 65 63 74 69 6F 6E 00 - proxy-connection. -
Host: http://www.baidu.com
Proxy-Connection: Keep-Alive

Cookies,
anyone?

We use cookies and related technologies to enhance your experience, show you personalised content and analyze performance and traffic on our website. By clicking on the „Accept All“ button you consent to the use of non-functional cookies and the subsequent processing of personal data to optimize our website and services as described in more detail in our Privacy and Cookie Policy.

By clicking on the „Customize or Deny all“ button you can decide otherwise. Clicking on the „Customize“ button will take you to a page where you can configure the usage of non-functional cookies (and related technologies) or deny all of them. You can access these settings at any time and also subsequently deselect cookies at any time in the footer area of our website.

ACCEPT ALL

Cookie Overview

We use the following categories of cookies and related technologies to enhance your experience, show you personalized content, and analyze performance and traffic on our website. We respect your right to privacy and accordingly you can chose to not allow some types of cookies (and related technologies). Click on the different category sliders and change our default settings to manage your cookie settings.

For more information on the specific cookies/related technologies we use and on how we use these, please see section 15 E. of our Privacy and Cookie Policy.

ABSOLUTELY NECESSARY

Authentication cookies we use are required to run our services … Cookies are required: Link

FUNCTIONAL AND MARKETING COOKIES

These cookies collect anonymous data and allow us to optimize our website and user experience. These cookies are listed here:

ANALYTICS

Help us to understand how visitors interact with our services, enables us to analyze and improve our services (also through third party analytics).