Anope IRC Services
Anope Support => 2.0.x/1.9.x Support => Topic started by: Marko Zelenka on November 23, 2016, 12:10:12 AM
-
Issuing a /LOGSEARCH command for some strings makes Services hang, consuming high CPU percentage. The only solution is to forcefully shut Services down using kill -9.
[Nov 23 00:14:24 2016] ADMIN: Mrmot!~Morpheus@XX.XXX.XX.XX (Mrmot) used LOGSEARCH for Ted
[Nov 23 00:34:58 2016] SERVER: services.krstarica.com (SERVICES - Pricaonica Internet Krstarice) has connected to the network (uplinked to no uplink)
01:09:43 -OperServ- Syntax: logsearch [+daysd] [+limitl] pattern
01:09:43 -OperServ-
01:09:43 -OperServ- This command searches the Services logfiles for messages
01:09:43 -OperServ- that match the given pattern. The day and limit argument
01:09:43 -OperServ- may be used to specify how many days of logs to search
01:09:43 -OperServ- and the number of replies to limit to. By default this
01:09:43 -OperServ- command searches one week of logs, and limits replies
01:09:43 -OperServ- to 50.
-
Clearly the time it takes to execute scales linearly with the number of lines to search through. I've never seen it be a real problem though. Get better disks and/or less logs?
-
Such grep takes 10 seconds.
Anope freezed for more than 20 minutes.
kill wasn't enough, had to do kill -9
[anopenew@ircmain old]$ grep Ted * | wc -l
4448
[anopenew@ircmain old]$ cat * | wc -l
4255541
-
That could be due to the relatively inefficient wildcard match it performs for the input string (in this case *Ted*). Compared to sed which is just simply trying to find Ted in each line. By default Anope only searches 7 days of logs, not all of the logs, you could try passing +1d. 4.2 million lines of logs seems rather excessive though, maybe you should look into an alternative solution?
-
You may try https://github.com/anope/anope/commit/437a6dbb2997c51c260115e8514c8cd6150cfaf5.patch which would optimize non-wildcard searches, but I still recommend you look at using logstash+elasticsearch or some other solution made for searching logs of such volume.
-
If it searches 7 days only, then it's a lot less number of lines:
[anopenew@ircmain old]$ cat services.log.20161116 services.log.20161117 services.log.20161118 services.log.20161119 services.log.20161120 services.log.20161121 services.log.20161122 | wc -l
413468
Will try your patch and get back to you.
-
We did apply you patch. We got another problem, we can`t search for nickname Ted:
[anopenew@ircmain logs]$ grep Ted services.log.‎20161123 | wc -l
81
[anopenew@ircmain logs]$ grep Ted services.log.‎20161122 | wc -l
1285
16:16:11 -OperServ- Too many results for Ted.
If we specify /os logsearch +2d Ted we got same resault. Also with /os logsearch +2d +100l Ted
It might be because of connecTED disconnecTED. Is there any solution?
-
You may try https://github.com/anope/anope/commit/437a6dbb2997c51c260115e8514c8cd6150cfaf5.patch which would optimize non-wildcard searches, but I still recommend you look at using logstash+elasticsearch or some other solution made for searching logs of such volume.
-
Is there a Anope module for logstash+elasticsearch?
-
No that wouldn't make sense.
-
Just found that
/os logsearch / Ted/
returns results properly.
Many thanks Adam for your prompt fix and help!