sort utility
Combine sort and uniq to get a list of spiders from nginx access log.
cat /var/www/logs/access.log | cut -d ' ' -f 14 | egrep "bot|spider" | sort | uniq
Baiduspider-render/2.0;
Baiduspider/2.0;
Clarabot/1.4;
Googlebot/2.1;
MJ12bot/v1.4.8;
bingbot/2.0;
linkdexbot/2.0;
spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)