LWP::Simple stopped working, what's the modern alternative?

Forum Moderators: coopster & phranque

Message Too Old, No Replies

LWP::Simple stopped working, what's the modern alternative?

csdude55

11:47 pm on Nov 10, 2019 (gmt 0)

I have some scripts that I wrote for clients a few hundred years ago, that look like:

use LWP::Simple;

# note that it's HTTP, not HTTPS
$getHeader = get('http://www.example.com/header.php');

print "Content-type: text/html\n\n";
print $getHeader;

I don't know when that stopped working, but I just now found it in my error log. These are seldom-used scripts, so it's probably been like that for awhile.

I could rebuild these scripts in PHP, but I doubt it's worth it to them to pay anything.

I tried changing it to LWP::UserAgent, but that didn't show any results, either. Nothing in the error log with that one, though. But I would bet that there's a security setting somewhere that's blocking it for whatever reason.

So before I do anything major, can you guys suggest a simple approach to include the results from a PHP script in the Perl script's output from the same account?

phranque

5:23 am on Nov 11, 2019 (gmt 0)

i don't see any error checking, so not sure how you would find anything in the error log.

csdude55

3:51 pm on Nov 11, 2019 (gmt 0)

It's a fatal error. I'm getting these two lines:

[Fri Nov 08 21:17:09.931673 2019] [cgi:warn] [pid 1983] [client 123.45.67.89:47530] AH01220: Timeout waiting for output from CGI script /home/example/public_html/cgi-bin/script.cgi, referer: http://www.example.com
[Fri Nov 08 21:17:09.932056 2019] [cgi:error] [pid 1983] [client 123.45.67.89:47530] Script timed out before returning headers: script.cgi, referer: http://www.example.com

I had 900 of these in the error log, so it stood out. Then when I went to load the script on their website, sometimes I'll get a 504 Gateway Timeout error, sometimes it will load the content and just not show anything for $getHeader.

I also tried installing LWP::Curl, WWW::Curl, and WWW::Curl::Easy last night, but kept getting errors that it couldn't be installed. There was a note in the error that libcurl may not be installed correctly? I use curl in PHP regularly, though, and it's my understanding that they work together, so I'm not sure.

phranque

8:13 pm on Nov 11, 2019 (gmt 0)

I would start with a simple test script and run it from the command line.

csdude55

11:43 pm on Nov 11, 2019 (gmt 0)

Just tried that... my entire script:

#!/usr/bin/perl 

use CGI::Carp qw(fatalsToBrowser);

# I added qw(!head) just in case, even though I'm not using CGI;
use LWP::Simple qw(!head);

$getHeader = get('http://www.example.com/header.php') or die "get failed: $!";

print "Content-type: text/html\n\n";
print $getHeader;
exit;

The die message:

get failed: Invalid argument

csdude55

1:56 am on Nov 12, 2019 (gmt 0)

Positive update: I disabled the CSF firewall, and now it responds. So it IS a security issue, not a code issue.

No clue how to track it down further, but... there ya go.

phranque

2:04 am on Nov 12, 2019 (gmt 0)

you might get more information if you use LWP::UserAgent instead.

csdude55

3:14 am on Nov 12, 2019 (gmt 0)

I tried it before posting originally, thinking that maybe it was an error with LWP::Simple. It didn't give me any more info, though :-( I don't have any unexpected IPs blocked in CSF, so now I'm guessing that LWP in general uses a specific port that CSF blocks. I'll probably spend the next week going through one setting at a time, trying to track it down :'-(

csdude55

1:44 pm on Nov 12, 2019 (gmt 0)

Second update for future followers:

Note that I disabled CSF, and then the scripts worked fine. So I narrowed the error down to the firewall.

I had 2 IPs in the Permanent Deny list, and none in the Temporary Deny list. Both of the IPs were in the same range and were US addresses, so I was hoping that I might be able to whitelist the range to prevent the problem from happening again.

To narrow it down, I removed both of the IPs in the Permanent Deny list, enabled CSF, and the scripts worked fine.

I added one of the IPs back to the Permanent Deny list, restarted CSF, and the scripts still worked fine. So it's the other IP, right?

Well, I added the second IP back, restarted CSF... and the scripts still worked fine.

So I guess it's not those IPs, after all, but I just needed to restart CSF? That's a major concern because, with no clear cause, there's no way for me to prevent this problem in the future.