What to do about bots looking for your admin, js, css files

Hello,
I've set my customer online list to expire in 24 hours instead of 1 hour, and as I look down the list every day, I see many bots searching for non existent admin pages or js css files which I guess are the default urls that you might find on other e-commerce packages. Some of them look like urls for Wordpress.

Should I be worried about it?
How do you deal with it?

I don't see any harm There's plenty of bots out in the net, and you can't easily stop them.

You can attempt to set no-access with robots.txt by adding this line. (change "default" to your theme name):
If they keep hitting, it could be a sign that they are hotlinking files/images/assets. To block a site from hotlinking, add something like this to your htaccess to deny request coming from a certain referrer. 2 examples here of syntax: So the side effects, this will only work for bots that obey robots or headers. You need to get tricker with user agent whitelist if they continue.

PS: Please don't just start banning IP's looking for your assets unless you know they are a bad crawler. Also keep in mind that bots can spoof referrer to make it seem like they are requesting from site A) when its actually site B) that is using the assets.

Hope that helps tackle the issues.

In the last 4 days I keep getting php errors 4 times a day, 2 errors exactly at the same second:
PHP Notice: Undefined index: email in catalog/controller/account/login.php on line ...

When I check the customer online log at the same time as the error, I get IP address from US, Ukraine, Russia, Germany, and the last page visited and referrer are both http://www.domain.com/index.php

the IP check shows "reported as forum spam source", and a few exist in blacklists.

What does it mean?
How are they getting to login.php from index.php?

2 requests on login at exact same time is classic sign of Xrumur spambot. They are hitting your index first, then they read source and click the first links for login + register + account + forum they can find. 80% of bad bots have this double behavior and you really start to see it looking at any honeypot/tarpit logs. They seem to be networked too, across diverse IP's. You may see bursts (in multiples of 2) hit to avoid being autobanned. Sometimes they will use a proxy service like Packetflip to flood you, you may see 20-50 bots hit within 5 mins from seperate IP's, all using similar behavior and login data.

So about the error: Unless you have a mod that made 1, or a mods ajax, there is no function in account/login that would offer a "portal" besides login itself. Undefined index for email would seem like they are trying to get in with either A) no email, B) corrupty email, C) blank-invalid-or-hidden-newline-type characters.

There are many things you can do to prevent bots at both server level and app level. Its all a learning experience in general but start with getting to know htaccess and signing up with stopforumspam.com. If you have a VPS, get a pot + API key from projecthoneypot.org and use with mod_security. Can also install APF for CpanelHulk+fail2ban, or a similar app for brute force prevent.

Thanks for the advice.
Any tips on vqmod to record the IP addresses that cause the login errors, so I can copy them directly into my htaccess instead of having to go through the customer online list?

Sure man, do you know PHP or comfortable copy paste changing in it? Or just a straight simple vQmod?

If you email dhaupin@creadev.org with store specs and stuff i can see if our quickNdirty test code would work to help you out. Its on calendar to be pimped in early summer with some useful extended log methods...but for now its concept

Thanks for all the info.
I also see a few times, in 1 second I get 8 hits from Amazon AWS and Digital Ocean IP address visiting the home page. It seems to be coordinated. The locations were in US, Brazil, Japan, Singapore, Netherlands, UK. I wonder if this is a response to my blocking of some IP addresses that I saw doing the double hits to the login.

I've also had before several bad requests from IP's from Amazon and Digital Ocean. I wonder if I should block these 8 also.

Yeah AWS and DigitalOcean are the "safe" ones most spammers use. I usually dont block those IP's though since their tactic is to open a VPS, assign an IP, spam until IP is blacklisted (aprx 3 days). After blacklist, they close the VPS, take an image, then re-open a new VPS with new IP. They upload the image and continue where they left off. Generally an infinite stream of fresh IP's. Since someone legit may use the IP in the future i usually leave them be. I have a test DigitalOcean server with 104.x.x.x common spammer range IP for example...but im not spamming Better bet is to report their IP(s) to AWS or DI and get their account totall banned from using the host(s).

The hosts you DO wanna block are these (there are more but these are the worst)
AS4134 - ChinaNet
AS9808 - Guangdong Mobile Com
AS16276 - OVH SAS
AS15003 - Nobis Tech Group
AS36352 - ColoCrossing
AS29761 - QuadraNet
AS15895 - Kyivstar PJSC
AS50915 - S.C. Everhost S.R.L.
AS53889 - Micfo
AS57858 - Fiber Grid OU

You can check the general terribleness of these ASN (hosts) by using something like cleantalk. Here is chinanet for example, at 35% of all their millions of IP being spammers: https://cleantalk.org/blacklists?record=AS4134

You can also block by country, for example China. There is very very little good traffic that comes from China. The traffic that isnt malicious is generally poor quality and bounces anyways. This site will build blacklists for you to enter into htaccess. Just keep in mind, if your htaccess is huge and you have many users, they will get 500 errors until upload is finished. http://www.ip2location.com/blockvisitorsbycountry.aspx

Just keep vigilant, there is no one size fits all answer. Even AWS and DigitalOcean, if they abuse you too much just block them (even if temporary 2 week ban).