Hi,

I've got a series of scripts one of which communicates with an external source via curl.

Let's say it has to download 1200 snippets of data every day. There's a script that populates a database with the 1200 urls that will have to be visited today.

Then a simple curl function loops through those urls and saves data back into the database.

Because there are several services communicating with the data source, I've introduced a 10 second pause between the requests.

Unfortunately, after looping through 70 - 100 urls, the script suddenly stops. It's not the case of the source blocking my connections. So, why is it breaking and how do I make sure it loops through the entire 1200 urls?

<?php
set_time_limit(0);
function get_data($url)
{
 blah blah blah - a simple curl function...
}

$database = "***"; 
mysql_connect("***", "***", "***") or die("Error connecting to database: ".mysql_error());
  mysql_select_db($database) or die(mysql_error());

$sql = mysql_query("SELECT * FROM `list`) or die(mysql_error());


//while loop here
while($row = mysql_fetch_array($sql))
 {//opens while brackets
 $url = $row['url'];
 $id = $row['id'];
 $data = get_data($url);
 $clean_data = mysql_real_escape_string($data);

$sql2 = "UPDATE `list` SET raw='$clean_data' WHERE id='$id'";
$result2 = mysql_query($sql2);
sleep(10); 
flush();
 }
?>

Maybe there's a way to detect the loop has stopped and pick it up from the next empty table row?
Thanks