Users rides glitch?

StangFlyer · 09-13-2000, 01:26 AM

Jim, I don't mind your suggestions and I do understand where you're coming from, believe me. Of course, having 100 threads does use resources, there's no doubt about it. However, on a server that has lots of memory, cache, and drive storage, using these resources is not nearly as large of a concern as the amount of CPU time being used. Since only 1 process of "X" application is running at a time really, and the others just waiting, you have one using CPU time and the others are virtually at zero usage. This is visible when monitoring processes/threads on the box in real time and taking note of their CPU usage, along with overall system load. The main box running MustangWorks.com is a Cobalt RaQ3i-512. If you aren't very familiar with it, it's a low cost but high end server with a 550 Mhz Intel based processor with a large cache, 512 Mb Ram, and a 20 Gig Ultra DMA 66 7200 RPM drive that runs the latest version of Red Hat Linux. As far as speed, it blows Win2K away and is EXTREMELY stable. In fact, MW's has gone for over a YEAR in the past on Linux without being rebooted, and was only rebooted because I was forced to after installing OS updates. So I worry less about sucking some memory and more about the load the app puts on the box when each instance does its thing. And, from experience I can tell you that the more file handling you do, the heavier the CPU load and the higher the load.

However, the Perl interpreter is like 100K, and yes it is loaded for every instance. Unless you are using either Mod_Perl on Unix or PerlEX on Windows. Mod_Perl is a module for Apache that is loaded into memory when the web server loads. PerlEX is a service for Windows NT or Win2K that is essentially the Perl interpreter running like Cold Fusion does. By utilizing one of these you eliminated launching the interpreter for each instance and the resource load associated with it. In addition, once a Perl app is loaded, compiled, and run, it is not dumped from memory either. It is kept compiled and reutilized each subsequent time it is invoked. This increasing a CGI's execution time by up to 50%. On MW's main server we DO have Mod_Perl.

Your observations about the scenario that corrupts a data file is probably close to the money. However, it is to be expected, and as I've stated I take precautions so that even in this case no data is lost. The locking system, in combination with my self backup technique, really keeps it from happening on all but the blue moon occasion. However, your suggestion about opening the data files with exclusive locking would be a good idea, but unfortunately is irrelevant because were on Linux. That's a feature of the Windows OS.

Something that I don't get, however, is how will your queuing method get around having to perform the same file locking technique that I am now? Even if you have each instance perform the actual file updates the last instance recorded in the queue file, you still have to make a lock file to protect multiple instances from writing to the queue file now. In the end, you're just doing the same thing, but with more steps and more complexity. And, you still have processes that have to wait in line for their turn.

Really, to do what you're suggesting correctly, you'd really need to create a daemon (like a service on Windows), that would be running in the background all the time. Then an instance of the regular app would just pass string commands to the daemon that contain the update to perform. The daemon would have a queue in memory that it keeps the commands in and just do them one at a time as quickly as possible, removing each command string from memory after completion. Now doing this would both eliminate the file locking system all together and eliminate waiting processes of the Perl app.

However, it is all irrelevant since I'm doing all future apps in CF. And, if I were going to redo any of the apps we have now, I would also do them in CF, as well with a SQL Server back end eliminates all these issues any ways.

------------------
Dan McClain, Editor
The Mustang Works Magazine
1991 Mustang GT - NOVI Supercharged 377 Stroker
1999 Ford Lightning SVT - Supercharged 5.4L Triton

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
User's Rides... Think about it....	Hammer	Modular Madness	2	07-23-2002 06:20 PM
Attn: Dan. "Users Rides" section only shows 2 users.	Fox Body	Blue Oval Lounge	2	02-11-2001 01:11 AM

09-13-2000, 01:26 AM	#13
StangFlyer Founder Join Date: Jun 1995 Location: Michigan Posts: 19,326	Jim, I don't mind your suggestions and I do understand where you're coming from, believe me. Of course, having 100 threads does use resources, there's no doubt about it. However, on a server that has lots of memory, cache, and drive storage, using these resources is not nearly as large of a concern as the amount of CPU time being used. Since only 1 process of "X" application is running at a time really, and the others just waiting, you have one using CPU time and the others are virtually at zero usage. This is visible when monitoring processes/threads on the box in real time and taking note of their CPU usage, along with overall system load. The main box running MustangWorks.com is a Cobalt RaQ3i-512. If you aren't very familiar with it, it's a low cost but high end server with a 550 Mhz Intel based processor with a large cache, 512 Mb Ram, and a 20 Gig Ultra DMA 66 7200 RPM drive that runs the latest version of Red Hat Linux. As far as speed, it blows Win2K away and is EXTREMELY stable. In fact, MW's has gone for over a YEAR in the past on Linux without being rebooted, and was only rebooted because I was forced to after installing OS updates. So I worry less about sucking some memory and more about the load the app puts on the box when each instance does its thing. And, from experience I can tell you that the more file handling you do, the heavier the CPU load and the higher the load. However, the Perl interpreter is like 100K, and yes it is loaded for every instance. Unless you are using either Mod_Perl on Unix or PerlEX on Windows. Mod_Perl is a module for Apache that is loaded into memory when the web server loads. PerlEX is a service for Windows NT or Win2K that is essentially the Perl interpreter running like Cold Fusion does. By utilizing one of these you eliminated launching the interpreter for each instance and the resource load associated with it. In addition, once a Perl app is loaded, compiled, and run, it is not dumped from memory either. It is kept compiled and reutilized each subsequent time it is invoked. This increasing a CGI's execution time by up to 50%. On MW's main server we DO have Mod_Perl. Your observations about the scenario that corrupts a data file is probably close to the money. However, it is to be expected, and as I've stated I take precautions so that even in this case no data is lost. The locking system, in combination with my self backup technique, really keeps it from happening on all but the blue moon occasion. However, your suggestion about opening the data files with exclusive locking would be a good idea, but unfortunately is irrelevant because were on Linux. That's a feature of the Windows OS. Something that I don't get, however, is how will your queuing method get around having to perform the same file locking technique that I am now? Even if you have each instance perform the actual file updates the last instance recorded in the queue file, you still have to make a lock file to protect multiple instances from writing to the queue file now. In the end, you're just doing the same thing, but with more steps and more complexity. And, you still have processes that have to wait in line for their turn. Really, to do what you're suggesting correctly, you'd really need to create a daemon (like a service on Windows), that would be running in the background all the time. Then an instance of the regular app would just pass string commands to the daemon that contain the update to perform. The daemon would have a queue in memory that it keeps the commands in and just do them one at a time as quickly as possible, removing each command string from memory after completion. Now doing this would both eliminate the file locking system all together and eliminate waiting processes of the Perl app. However, it is all irrelevant since I'm doing all future apps in CF. And, if I were going to redo any of the apps we have now, I would also do them in CF, as well with a SQL Server back end eliminates all these issues any ways. ------------------ Dan McClain, Editor The Mustang Works Magazine 1991 Mustang GT - NOVI Supercharged 377 Stroker 1999 Ford Lightning SVT - Supercharged 5.4L Triton