Web service with a script that runs a very long time

[zomgwtfbbq]

I've got an idea for writing a service for my site, I'm not going to tell what exactly it is, don't want ppl to steal the idea.
Anyway the script calculates something using an algorithm and takes a lot of time to complete.. on my new i7 it takes 17 sec per year and I need at least a timespan of 6 years..so I need a settime limit of a couple of minutes, anyone familiar with these kind of things?
Should I limit the calculation to one at a time, or can I have more of these calculations at a time?

When finished it will be accessible for registered members at http://www.tools-and-design.com

Thanks in advance!

[thetan]

First of all, the information economy (the internet) is about execution not about ideas. So get that notion out of your head.

Second of all, the problem is to vague. If you need 17 sec to calc something, then i have a feeling that the solution to the problem can be achieved faster then the current method in which you are implementing. So look at it as an engineering problem rather then a simple scripting problem.

The simple fact that a single script will be running CPU intensive for 17 secs per request is indicative that you're site will die really really fast under heavy concurrent load.

[Avery17]

I ask what language this is written in?

If its in PHP which is what I'm guessing it is in then you should take a look at this.
http://php.net/manual/en/function.set-time-limit.php

If not I just wasted 4 seconds of my life, oh well.

[Assassian360]

First of all, the information economy (the internet) is about execution not about ideas. So get that notion out of your head.

Second of all, the problem is to vague. If you need 17 sec to calc something, then i have a feeling that the solution to the problem can be achieved faster then the current method in which you are implementing. So look at it as an engineering problem rather then a simple scripting problem.

The simple fact that a single script will be running CPU intensive for 17 secs per request is indicative that you're site will die really really fast under heavy concurrent load.

I'm with thetan, your algorithm can probably be simplified.
Either the algorithm complexity itself or perhaps pre-calculation of common values or stages that are never going to change. Even both... In any case, 17 seconds for a web based script is going to make your server go crazy. Any more than a couple of users at a time and instantly users are not going to be very happy.

[tgoe]

Are you being vague because this is really just a cleverly disguised spam?

17s per year per user isn't really much different than a single php generated pageview by a loyal visitor... how exactly are you calculating this 17 seconds?

[thetan]

@OP: http://www.washingtonpost.com/wp-dyn/co ... 04303.html << read this article, it's only pertinent to you're notion of the important of secrecy. In conclusion, there is no point. The information economy is about execution more so then just ideas. Actually, i'm going to quote the article for you

When these advances happen, they happen to many people simultaneously, so many people tend to see the next step forward at the same time. In 2003, we were all social network geniuses, at least compared with everyone in 1993.

Consider CU Community, a Facebook competitor started at Columbia University. Adam Goldberg, its creator, programmed his social network over the summer in 2003. It was more advanced than Facebook, with options for pictures and integrated blogging software, though it did lack the elegant minimalism of Zuckerberg's design. (Disclosure: Washington Post Co. Chairman Donald E. Graham is on Facebook's board, and The Post markets itself on Facebook.)

This phenomenon is age-old: It's called "simultaneous invention." Technology - and the conversation about what can be done with it - advances to the point that the next step is obvious to multiple people at once, and so they all push forward. In the end, one squeezes the others out by landing the patent, or the market share, and becomes synonymous with the invention. That's what happened with Alexander Graham Bell, who in all likelihood invented the telephone after Elisha Gray - and both of them came after Antonio Meucci, who couldn't afford the fee to keep his patent current.

Are you being vague because this is really just a cleverly disguised spam?

yeah i thought the same thing too.

17s per year per user isn't really much different than a single php generated pageview by a loyal visitor... how exactly are you calculating this 17 seconds?

Interesting, i assumed he mean't he was doing some calculation based on years as an input that would take 17 seconds, not a calculation that would take a total of 17 seconds within the time of a year.

page generation for a typical php request should be far lower then 17 seconds (given the programmers aren't retarded), this can otherwise pose that same point of contention that i was talking about earlier with resource consumption and single user DoS

EDIT: look at it this way
if i were to make 250 page requests within that 17 second period:
17 seconds per page request * 250 page requests / 60 second in a minute / 60 minutes in an hour = 1.18 hours
Guess how many more page requests i could make in that hour period?

CPU time aside, (shitty) web servers like apache dedicate a single thread per connection. It's typical for the average system to dedicate 4-8MB of memory to stack space per thread. The fact that the script will keep these threads allocated for a significant amount of time poses yet another point of contention. So just from the first 250 page requests (assuming an 8MB average which is reasonable):

8 MB of stack space per thread(||request) * 250 requests = 2000 MB of memory (RAM)

If you're on a VPS you'd probably be royally fucked by now, however even if you're running on a powerful dedicated box or one you built and is a data center for colocation you have to remember those threads will be held open for roughly an hours time giving me plenty of time to open thousands more.

The threading issue can be addressed non programatically, by ditching apache and using one of the many modern web servers that were created to address the c10k problem. My favorite one that i use everyday and have to recommend would be Nginx.

Most of these c10k web servers work around the same philosophies:

• They're event loop based and use scalable asynchronous IO methods (linux: epoll() && freebsd: kqueue() etc)
• They maintain a small thread pool that gets fed requests and never grows in size (instead of 1 thread per connection)
• For script execution they tend to rely on FastCGI (php-fpm is awesome) or SCGI

[tgoe]

Even if a page requires a db hit or two at say 0.05s to generate, that's about 20s total per year of a single user visiting just once a day and a long running program doing more sleeping (and waiting for data) than anything would get paged out. If it is 17s of CPU time per request and also takes 17s realtime, OP is probably doing popen()s and/or a bad algorithm.

[thetan]

Either way further speculation, without clarity on the problem (source code or at least a more detailed description) or disambiguation on what the OP even mean't, will be entirely moot.