Forum Moderators: coopster & phranque

Message Too Old, No Replies

Displaying Database Results

anywhere to cut load time?

         

MizzBia

9:56 pm on Oct 18, 2008 (gmt 0)

10+ Year Member




Hello,
I was wondering if anyone new how to cut the load time, or break up the load time, when you are displaying results from a database. My database is huge and I have a listings page that displays "all comments" by a particular username. It takes about 4-5 minutes to actually load the page which shows 10 results per page. Once it loads, if you go to the next page, it takes another 2-3 minutes to show those results. As you can see, this is horrible. The rest of my site goes relatively fast so its not a server issue. Plus, I've seen other sites do the same thing and their page loads lightening fast. Am I missing something?

Anyhow, I was wondering if anyone knew any tips, ideas or techniques that could help with this.
Thanks for reading!

rocknbil

2:06 pm on Oct 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would chase this one by beginning with the queries. Figure out which queries are being made here. If the query is built dynamically, you can have your script output a "comment" into the HTML on this page, example,

<!-- select comments from db_table where user=12345 limit 125,10 order by lastmod desc; -->

Then paste that on a command line, see how long it takes to run. If it's under a second, it's not likely the problem. If it takes time to query, the problem is your DB is not optimized. This can be anything from poor indexing to limitations on your usage or even an overloaded server.

If it runs fast on the command line, it could be networking issues or just an overloaded server.

The next is really left field because it's how your script should be working, and probably is, but just in case because I have seen this. Note my bolded text above. This indexes directly to the next page of results and *should* be relatively fast. If your selects are not using limit and are parsing through the entire table, just grabbing what's needed within the script, this would take forever and needs fixing.

Last is the nature of your environment: is it shared hosting, possibly overloaded? Do you have limitations on your hosting account that are slowing you down?

When I get in these situations, I take the specific code that's giving the grief and put it in a test script to isolate it. Usually this leads to an understanding of exactly what's going on and it can be fixed one way or the other.

phranque

3:08 am on Oct 20, 2008 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



if the query time doesn't answer your question and/or you want to really get into timing tests of your perl code, you could look at the Benchmark perl module [search.cpan.org]:
use Benchmark;
$t0 = new Benchmark;
# ... your code here ...
$t1 = new Benchmark;
$td = timediff($t1, $t0);
print "the code took:",timestr($td),"\n";

MizzBia

3:30 am on Oct 20, 2008 (gmt 0)

10+ Year Member



Hi rocknbil and phranque,
thanks for getting back to me :) I'll take a look at the benchmark module, thanks!

I'm currently on a semi-dedicated server now which actually runs pretty fast. I don't have any trouble on the other areas of my site. My site was built by various different programmers so some areas seem to work better then others which makes me think its coding issues?

It doesn't look to me like a limit is being using:


my $sth7 = $dbh->prepare("select * from comments where member=$userid");

If I were to use a limit, wouldn't it ONLY show that # of comments? I need to show all the users comments... would that still be able to work?

Thanks again for your time, I appreciate it!

rocknbil

4:55 am on Oct 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well I don't think that should slow you down, unless there are a million records by this user and the programming is doing the paging without "limit."

If I were to use a limit, wouldn't it ONLY show that # of comments? I need to show all the users comments....

The way it should work is you select only what you need for this page view. So if you view 10 per page, the first time,

select * from comments where user=$userid limit 0,10;

0 is not necessary the first time but when you loop through to create links, it's useful. Your links below the first 10 query the next section, and so on.

$per_page=10;
select count(*) from comments where userid=$userid;
....
$total_records=$sth->fetchrow_array;

for $i (0..($total_records+=$per_page)) {
$end=$i+$per_page;
$links .= "<a href="script?start=$i&amp;end=$per_page">$i - $end</a>";
}

(Actually this isn't correct, you want the link to say 1-10 not 0-10, for example only)

So by passing your current place in the records, you start at a different place with each page:

select * from comments where user=$userid limit 11,20;
select * from comments where user=$userid limit 21,30;
select * from comments where user=$userid limit 31,40;

---------------------------

Something else came to mind that has nothing to do with your script.

Once it loads, if you go to the next page . . .

Tell me, how many links are at the bottom of the page when you do this? I learned a lesson early on about creating footlinks from huge databases. :-) Is possible there are too many links, choking the browser?

[edited by: rocknbil at 5:07 am (utc) on Oct. 20, 2008]

Swanny007

4:59 am on Oct 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Have you thought of using caching too? I use caching on my largest site's home page due to the constant db queries.

MizzBia

7:39 am on Oct 20, 2008 (gmt 0)

10+ Year Member



Hi Swanny, I have not thought of caching (don't know anything about it). I will look it up if you think it will help. Would caching be considered a big job or is it more like a quick fix?

Thanks rocknbil... you are so helpful :)
The account I usually test the load time has almost 2000 comments so at 10 per page, the "pagination" list of numbers is huge (shows page 1 to 200). You think this is the problem? Thats what I had initially thought... seems to me that the script has to query the entire database in order to show all these pages, no?

You know, you may be on to something because now that I think of it, it takes 5 minutes to load the initial page but even when you go to page 2 or 3... it still takes 3-4 minutes which I'm assuming is because it has to display all those page links each time you go to a new page.

I wanted to implement this other pagination that I found on the net - It allows you to have a nice digg-style pagination system. Do you think this will cause the same issues as I have now? What about if I were to implement that with the code idea you posted above? Would the two work together?

Thanks again, I hope I'm not taking up too much of your time :(

[edited by: phranque at 8:01 am (utc) on Oct. 20, 2008]
[edit reason] No urls, please. See TOS [webmasterworld.com] [/edit]

mark_roach

10:32 am on Oct 20, 2008 (gmt 0)

10+ Year Member



This might be an obvious question, but do you have an index on the member column ?

rocknbil

3:11 pm on Oct 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



....2000 comments so at 10 per page, the "pagination" list of numbers is huge (shows page 1 to 200).

No, 200 links should be fine - UNLESS each link is a query string that is **really** long. Even then, doubtful. But as your database grows, I guarantee you, this WILL be a problem.

... seems to me that the script has to query the entire database in order to show all these pages, no?

See previous examples - absolutely not.

You do a count to see how many records there are.

You do a limit on your selects to select only the data needed for the current page.

You have a routine that, based on the total records you counted, creates the links WITHOUT reading each record.

. . . I wanted to implement this other pagination that I found on the net . . .

Don't know. :-) This is actually kinda typical of project cobbled together from things found free on the net. Any one of them could have problems.

MizzBia

9:09 pm on Oct 20, 2008 (gmt 0)

10+ Year Member



lol... I guess I wasn't allowed to post that link to the digg-style pagination (sorry about that).

Not sure if there is an index Mark, I will have to look into that. My knowledge on all this is very limited so I don't always understand what the script is doing or how the database is setup.

[edited by: phranque at 11:16 pm (utc) on Oct. 20, 2008]