Monday, February 7, 2011

Where does CGI.pm normally create temporary files?

On all my Windows servers, except for one machine, when I execute the following code to allocate a temporary files folder:

use CGI;
my $tmpfile = new CGITempFile(1);
print "tmpfile='", $tmpfile->as_string(), "'\n";

The variable $tmpfile is assigned the value '.\CGItemp1' and this is what I want. But on one of my servers it's incorrectly set to C:\temp\CGItemp1.

All the servers are running Windows 2003 Standard Edition, IIS6 and ActivePerl 5.8.8.822 (upgrading to later version of Perl not an option). The result is always the same when running a script from the command line or in IIS as a CGI script (where scriptmap .pl = c:\perl\bin\perl.exe "%s" %s).

How I can fix this Perl installation and force it to return '.\CGItemp1' by default?

I've even copied the whole Perl folder from one of the working servers to this machine but no joy.

@Hometoast:

I checked the 'TMP' and 'TEMP' environment variables and also $ENV{TMP} and $ENV{TEMP} and they're identical.

From command line they point to the user profile directory, for example:

C:\DOCUME~1\[USERNAME]\LOCALS~1\Temp\1

When run under IIS as a CGI script they both point to:

c:\windows\temp

In registry key HKEY_USERS/.DEFAULT/Environment, both servers have:

%USERPROFILE%\Local Settings\Temp

The ActiveState implementation of CGITempFile() is clearly using an alternative mechanism to determine how it should generate the temporary folder.

@Ranguard:

The real problem is with the CGI.pm module and attachment handling. Whenever a file is uploaded to the site CGI.pm needs to store it somewhere temporary. To do this CGITempFile() is called within CGI.pm to allocate a temporary folder. So unfortunately I can't use File::Temp. Thanks anyway.

@Chris:

That helped a bunch. I did have a quick scan through the CGI.pm source earlier but your suggestion made me go back and look at it more studiously to understand the underlying algorithm. I got things working, but the oddest thing is that there was originally no c:\temp folder on the server.

To obtain a temporary fix I created a c:\temp folder and set the relevant permissions for the website's anonymous user account. But because this is a shared box I couldn't leave things that way, even though the temp files were being deleted. To cut a long story short, I renamed the c:\temp folder to something different and magically the correct '.\' folder path was being returned. I also noticed that the customer had enabled FrontPage extensions on the site, which removes write access for the anonymous user account on the website folders, so this permission needed re-applying. I'm still at a loss as to why at the start of this issue CGITempFile() was returning c:\temp, even though that folder didn't exist, and why it magically started working again.

Thanks in advance
Kevin

  • If you're running this script as you, check the %TEMP% environment variable to see if if it differs.

    If IIS is executing, check the values in registry for TMP and TEMP under HKEY_USERS/.DEFAULT/Environment

    From hometoast
  • The name of the temporary directory is held in $CGITempFile::TMPDIRECTORY and initialised in the find_tempdir function in CGI.pm. The algorithm for choosing the temporary directory is described in the CGI.pm documentation (search for -private_tempfiles). IIUC, if a C:\Temp folder exists on the server, CGI.pm will use it. If none of the directories checked in find_tempdir exist, then the current directory "." is used.

    I hope this helps.

    From ChrisN
  • Not the direct answer to your question, but have you tried using File::Temp?

    It is specifically designed to work on any OS.

    From Ranguard

How to best implement simple crash / error reporting?

What would be the best way to implement a simple crash / error reporting mechanism?

Details: my app is cross-platform (mac/windows/linux) and written in Python, so I just need something that will send me a small amount of text, e.g. just a timestamp and a traceback (which I already generate and show in my error dialog).

It would be fine if it could simply email it, but I can't think of a way to do this without including a username and password for the smtp server in the application... Should I implement a simple web service on the server side and have my app send it an HTTP request with the info? Any better ideas?

Thanks!

  • The web hit is the way to go, but make sure you pick a good URL - your app will be hitting it for years to come.

    From pjz
  • PyCrash?

    From Allan Wind
  • Whether you use SMTP or HTTP to send the data, you need to have a username/password in the application to prevent just anyone from sending random data to you.

    With that in mind, I suspect it would be easier to use SMTP rather than HTTP to send the data.

    From Branan
  • The web service is the best way, but there are some caveats:

    1. You should always ask the user if it is ok to send error feedback information.
    2. You should be prepared to fail gracefully if there are network errors. Don't let a failure to report a crash impede recovery!
    3. You should avoid including user identifying or sensitive information unless the user knows (see #1) and you should either use SSL or otherwise protect it. Some jurisdictions impose burdens on you that you might not want to deal with, so it's best to simply not save such information.
    4. Like any web service, make sure your service is not exploitable by miscreants.
    From nsayer
  • Some kind of simple web service would suffice. You would have to consider security so not just anyone could make requests to your service..

    On a larger scale we considered a JMS messaging system. Put a serialized object of data containing the traceback/error message into a queue and consume it every x minutes generating reports/alerts from that data.

  • I can't think of a way to do this without including a username and password for the smtp server in the application...

    You only need a username and password for authenticating yourself to a smarthost. You don't need it to send mail directly, you need it to send mail through a relay, e.g. your ISP's mail server. It's perfectly possible to send email without authentication - that's why spam is so hard to stop.

    Having said that, some ISPs block outbound traffic on port 25, so the most robust alternative is an HTTP POST, which is unlikely to be blocked by anything. Be sure to pick a URL that you won't feel restricted by later on, or better yet, have the application periodically check for updates, so if you decide to change domains or something, you can push an update in advance.

    Security isn't really an issue. You can fairly easily discard junk data, so all that really concerns you is whether or not somebody would go to the trouble of constructing fake tracebacks to mess with you, and that's a very unlikely situation.

    As for the payload, PyCrash can help you with that.

    From Jim

FollowSymLinks - how to setup

I am trying to have Apache follow a symlink to a raid array server that will contain some large data files. I have tried modifying httpd.conf to have an entry like this

Options FollowSymLinks AllowOverride all Order allow,deny Allow from all

to have Apache follow any sym link in the Sites folder.

I keep getting an error return that seems to indicate I don't have any permissions to access the files. The error is: Forbidden

You don't have permission to access /~imagine/imageLibraryTest/videoClips/imageLibraryVideos/imageLibraryVideos/Data13/0002RT-1.mov on this server.

the sys link file is the last "imageLibraryVideos" in the line with the Data13 being the sub dir on the server containing the file.

The 0002RT-1.mov file hase these permissions: -rwxrwxrwx 1 imagine staff 1138757 Sep 15 17:01 0002RT-1.mov and is in this path: cd /Volumes/ImagineProducts-1/Users/imagine/Sites/imageLibraryVideos/Data13

the link points to: lrwxr-xr-x 1 imagine staff 65 Sep 15 16:40 imageLibraryVideos -> /Volumes/ImagineProducts-1/Users/imagine/Sites/imageLibraryVideos

  • Look in the enclosing directories. They need to be at least mode 711. (drwx--x--x)

    Also, look in /var/log/apache2/error_log (Or whatever the concatenation of ServerRoot and ErrorLog is from the httpd.conf) for a possibly more-detailed error message.

    Finally, ensure you restart apache after messing with httpd.conf.

    From Pi
  • This is a permissions problem where the user that your web server is running under does not have read and/or execute permissions to the necessary directories in the symbolic link path. The quick and easy way to check is to su - web-user (where web-user is the user account that the web server is running under) and then try to cd into the path and view the file. When you come across a directory that you don't have permission to enter, you'll have to change the permissions and/or ownership to make it accessible by the web server user account.

  • Thanks,

    I've set the permissions to all the enclosing directories to 755
    but Apache cstill cannot access the file.

  • You should also look at bind mounts rather than symlinks - that would allow you to remount a given path at a new point. The following is an example:

    mount --rbind /path/to/current/location/somewhere/else /new/mount/point
    

    You can also edit your fstab to do this at boot:

    /path/to/original /new/path bind defaults,bind 0 0
    
    From warren

Best binary XML format for JavaME

Can anyone recommend a good binary XML format? It's for a JavaME application, so it needs to be a) Easy to implement on the server, and b) Easy to write a low-footprint parser for on a low-end JavaME client device.

And it goes without saying that it needs to be smaller than XML, and faster to parse.

  • What kind of data are you planning to use? I would say, that if the server is also done in Java, easiest way for small footprint is to send/receive binary data in predefined format. Just write everything in known order into DataOutputStream.

    But it would really depend, what what kind of data are you working on and can you define the format.

    Actually you should evaluate, if this kind of optimization is even needed. Maybe you target devices are not so limited :)

    /JaanusSiim

    From JaanusSiim
  • You might want to take a look at wbxml (Wireless Binary XML) it is optimized for size, and often used on mobile phones, but it is not optimized for parsing speed.

    From Pat
  • The data would be something akin to SVG.

    From izb
  • Check out the following: SAXProject, XMLPull and kXML. kXML I have found to be pretty good. Regards philip.ie

    From Philip.ie
  • It very much depends on the target device. If you have JSR172 available, then you are done with the parsing, the runtime does it for you. And XML is mainly about making your own format. As was alredy stated if your goal is performance, than XML is probably not the best way to go and you will end up doing some binary stuff.

    From Honza
  • Hessian might be an alternative worth looking at. It is a small protocol, well-suited for Java ME applications.

    "Hessian is a binary web service protocol that makes web services usable without requiring a large framework, and without learning a new set of protocols. Because it is a binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments."

    More links:

    http://hessian.caucho.com/

    http://www.nubius.be/info-software/hessian4j.html

    Wilfred Springer : We used that at TomTom as well. There's one gotcha though: the ME implementation of Hessian is not the same as the SE/EE implementation of Hessian. There are quite a few things missing, so don't expect binary compatibility in all cases.
    From darius

How do I enable more than 8-bit colors in Terminal.app?

In the Vim and Emacs terminal apps, the color schemes look horrid. How do I enable the colors to be as vibrant as the GUI version (or more than 8 colors for that matter)?

Should I just give up, and move over to their respective GUI applications? And if so, which?

  • You can't have more colors in the terminal, because there are only ANSI codes for 8 colors (16 if you count bold/light). If you want to customize the colors, you can use the TerminalColours plugin from http://ciaranwal.sh/2007/11/01/customising-colours-in-leopard-terminal

    Personally, I prefer to use the so-called Carbon Emacs on my Mac. There are several builds available; Google is your friend. I get mine from http://www.porkrind.org/emacs/

    From masto
  • You are possibly being inhibited by the fact you are running in a VT100/WhateverItsCalled compatible terminal and it just doesn't have more than about 16 foreground and 16 background colours. If it did, the lovely library "CaCa" ( ColourAsciiColourArt ) would be much more pleasing to watch than it is.

    If you want more colours, you simply have to use more modern technology, and this generally means using X ( unless your loving pain and want to use directfb/framebuffer/svgalib ).

    For vim, there is GVim ( GTK+Vim ). Emacs GUI version however displeases me much, but I'm not an emacs user.

  • Aquamacs and MacVim

    From Hank Gay
  • Sounds like you want MacVim

    From camflan

How do I disable a button cell in a WinForms DataGrid?

I have a WinForms application with a DataGridView control and a column of DataGridViewButtonCell cells within that. When I click on one of these buttons, it starts a background task, and I'd like to disable the buttons until that task completes.

I can disable the DataGridView control, but it gives no visual indication that the buttons are disabled. I want the user to see that the buttons are disabled, and to notice that the task has finished when the buttons are enabled again.

Bonus points for a method that allows me to disable the buttons individually, so I can leave one of the buttons enabled while the task runs. (Note that I can't actually give out bonus points.)

  • Here's the best solution I've found so far. This MSDN article gives the source code for a cell class that adds an Enabled property.

    It works reasonably well, but there are two gotchas:

    1. You have to invalidate the grid after setting the Enabled property on any cells. It shows that in the sample code, but I missed it.
    2. It's only a visual change, setting the Enabled property doesn't actually enable or disable the button. The user can still click on it. I could check the enabled property before executing the click event, but it also seemed to be messing up the appearance when the user clicked on it. Instead, I just disabled the entire grid. That works alright for me, but I'd prefer a method that allows me to disable some buttons without disabling the entire grid.
    From Don Kirkby
  • Another example using the same concept as the MSDN article linked by Don.

    From hometoast
  • You could give this a try:

    When you click on the cell...

    1. Check to see if the process with the current row identifier is running from a class-level list; if so, exit the cell click event.
    2. Store the row identifier in the class-level list of running processes.
    3. Change the button text to "Running..." or something appropriate.
    4. Attach a basic RunWorkerCompleted event handler to your process (explained shortly).
    5. Call backgroundWorker.RunWorkerAsync(rowIdentifier).

    In the DoWork event handler...

    1. Set e.Result = e.Argument (or create an object that will return both the argument and your desired result)

    In the RunWorkerCompleted event hanlder...

    1. Remove the row identifier from the running processes list (e.Result is the identifier).
    2. Change the button text from "Running..." to "Ready"
    Don Kirkby : Thanks for an interesting alternative, but I'd rather stick with the standard visual cue of graying out the buttons.

Strange characters in PHP

This is driving me crazy.

I have this one php file on a test server at work which does not work.. I kept deleting stuff from it till it became

<?
print 'Hello';
?>

it outputs

Hello

if I create a new file and copy / paste the same script to it it works! Why does this one file give me the strange characters all the time?

  • That's the BOM (Byte Order Mark) you are seeing.

    In your editor, there should be a way to force saving without BOM which will remove the problem.

    From Pat
  • Is the document saved in an encoding such as UTF-8/16? What encoding does the html-page specify?

    From Patrik
  • Wow that was fast guys !

    Thanks I will check it out.

  • We had the same thing happen once upon a time it drove the dev crazy for a while too :-)

    Someone had opened the file and accidentally saved it with BOM and from there on the editor would preserve the BOM while saving the file.

    In your editor, there should be a way to force saving without BOM which will remove the problem.

    From Pat
  • Found it, file -> encoding -> UTF8 with BOM , changed to to UTF :-)

    I should ahve asked before wasing time trying to figure it out :-)

  • Just in case, here is a list of bytes for BOM

    Encoding    Representation (hexadecimal)
    UTF-8   EF BB BF
    UTF-16 (BE) FE FF
    UTF-16 (LE) FF FE
    UTF-32 (BE) 00 00 FE FF
    UTF-32 (LE) FF FE 00 00
    UTF-7   2B 2F 76, and one of the following bytes: [ 38 | 39 | 2B | 2F ]†
    UTF-1   F7 64 4C
    UTF-EBCDIC  DD 73 66 73
    SCSU    0E FE FF
    BOCU-1  FB EE 28 optionally followed by FF†
    

Sending Excel to user through ASP.NET

I have a web application that is able to open an excel template, push data into a worksheet and send the file to a user. When the file is opened a VBA Macro will refresh a pivot table based on the data that was pushed into the template.

The user receives the standard File Open / Save dialog.

In Internet Explorer (version 6), if the user chooses to save the file, when the file is opened the VBA code runs as expected, however if the user chooses 'Open' then the VBA fails with:

Run-Time error 1004: Cannot open Pivot Table source file.

In all other browsers both open and save work as expected.

It is not in my power to upgrade to a newer version of IE (corporate bureaucracy); Is there anything Ican do to allow the users to open without first saving?

  • The option to open or save is a browser selection item, as far as I know it is not possible to override this behavior.

  • Newer versions of Excel really don't like running macros automatically. If you really want to generate an Excel file and send that to your users, build the full file on the server using the COM interface to Excel, or some libraries that can read/write XLS files, and then send the completed file.

    From thebigjc
  • If I had to guess, I'd say it has to do with what zone the file is currently in. Its probably still considered in the "internet zone" when you click Open. VBA shouldn't be running within that zone.

    Have a user mark the server website as safe (Control Panel -> Internet Options -> Security -> Trusted Sites -> Sites) and see if that helps.

    From Will

Change default port when registering a new SQL 2000 server

Hi All,

I'm trying to register an externally hosted SQL 2000 server through Enterprise Manager which isn't on the default port and I can't see anywhere to change it within Enterprise Manager.

So, the question is, how do I connect to the database if:

I.P Address is 123.456.789 (example)

Port is 1334

Thanks in advance

EDIT: Both Rob and Denny Were correct however I still couldn't access the SQL 2000 Server as we had to open port 1334 on our firewall to allow access to the external server.

  • I found this via Google:

    You add a comma and the port number to the end of the server name.

    So if you want to connect to MySqlServer.MyDomain.com on port 3821, you type...

    MySqlServer.MyDomain.com,3821

    From Rob Allen
  • Rob is correct - I have a SQL 2000 server running on the non-default port on a different instance name and the way I access it is like this:

    [ip or dns name]\[instance], [port]

    example:

    my.server.com\MSSQLSERVER2, 12345

    You don't need \[instance] if you used the default sql server instance when you installed.

Where can I find a good introduction to maven?

Hi folks,

I have a sufficient amount of experience with ant so far, but I am interested in learning more about maven. It seems really opaque. Can you recommend a good introduction to maven -- I'm looking for as much clarity and simplicity as possible.

I suppose I'm looking specifically for information on maven 2 if it comes down to it.

  • I found the book "Better Builds with Maven" to be a pretty good introduction. You can download it for free: http://www.exist.com/better-build-maven

    Brian Fox : This book is very old now. Some content is still relevant, but the Maven: The Definitive Guide is much more recent, and is constantly updated.
    From nafisto
  • The book Maven: The Definitive Guide provides an introduction to Maven and covers many of the problems that arise when using Maven.

    The Maven documentation is a bit vague and outdated in places, but the usage examples for plugins are usually good enough to get you going with what you need to do. Also find projects that are utilizing Maven and look into their pom.xml configurations learn more about what is possible with Maven.

  • Don't waste your time, just look at what other people are saying and do some more research. Maven is not worth it.

PHP DOMDocument stripping HTML tags

I'm working on a small templating engine, and I'm using DOMDocument to parse the pages. My test page so far looks like this:

<block name="content">

 <?php echo 'this is some rendered PHP! <br />' ?>

    <p>Main column of <span>content</span></p>

</block>

And part of my class looks like this:

private function parse($tag, $attr = 'name')
{
 $strict = 0;
 /*** the array to return ***/
    $out = array();
 if($this->totalBlocks() > 0)
 {
     /*** a new dom object ***/
     $dom = new domDocument;
  /*** discard white space ***/
     $dom->preserveWhiteSpace = false;

     /*** load the html into the object ***/
     if($strict==1)
     {
         $dom->loadXML($this->file_contents);
     }
     else
     {
         $dom->loadHTML($this->file_contents);
     }

     /*** the tag by its tag name ***/
     $content = $dom->getElementsByTagname($tag);

  $i = 0;
     foreach ($content as $item)
     {
         /*** add node value to the out array ***/
   $out[$i]['name'] = $item->getAttribute($attr);
         $out[$i]['value'] = $item->nodeValue;
   $i++;
     }
 }

 return $out;
}

I have it working the way I want in that it grabs each <block> on the page and injects it's contents into my template, however, it is stripping the HTML tags within the <block>, thus returning the following without the <p> or <span> tags:

this is some rendered PHP! Main column of content

What am I doing wrong here? :) Thanks

  • Nothing: nodeValue is the concatenation of the value portion of the tree, and will never have tags.

    What I would do to make an HTML fragment of the tree under $node is this:

    
    $doc = new DOMDocument();
    foreach($node->childNodes as $child) {
        $doc->appendChild($doc->importNode($child, true));
    }
    return $doc->saveHTML();
    

    HTML "fragments" are actually more problematic than you'd think at first, because they tend to lack things like doctypes and character sets, which makes it hard to deterministically go back and forth between portions of a DOM tree and HTML fragments.

  • Thanks for the ridiculously quick response :)

    the new foreach loop does what I'm looking for.

    foreach ($content as $item)
         {
             /*** add node value to the out array ***/
       $out[$i]['name'] = $item->getAttribute($attr);
    
       $doc = new DOMDocument();
       foreach($item->childNodes as $child) {
           $doc->appendChild($doc->importNode($child, true));
       }
       $out[$i]['value'] = $doc->saveHTML();
       $i++;
         }
    
    From litzinger

ASPNET user does not have write access to Temporary ASP.NET Files

When I run my Visual Studio 2008 ASP.NET project (start without Debugging) on my XP Professional box, I get the following error:

System.Web.HttpException: The current identity (machinename\ASPNET) does not have write access to 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files'.

How do I resolve this?

  • Have you tried, the aspnet_regiis exe in the framework folder?

    y0mbo : Rookie mistake... forgot about that and did a Windows repair the other day without rerunning.
  • Make sure the ASPNET user has permission to write to that folder. Right click on the folder, Properties, Security tab.

    From ddc0660
  • Either grant that user the level of access to that directory, or change the identity that the application's application pool runs under - in IIS Manager, determine what App Pool is used to run your application, then in the App Pool section of IIS Manager, look at the properties for that pool - the tab you want is "Identity" I think (this is off the top of my head).

    You can set it to another user account - for example, Crystal Reports .Net requires update and delete access to C:\Temp - so we have a "webmaster" user, with administrator access, and use that identity for those applications.

    From Ken Ray
  • You can try to fix it using the automated regiis utility aspnet_regiis.ext available in c:\windows\microsoft.net\framework\v2.0.50727

    Otherwise just manually add the needed file permissions as noted in the error.

  • you can right click the Visual Studio & select run as administrator.

How to Autogenerate multiple getters/setters or accessors in Visual Studio

Before I start, I know there is this post and it doesn't answer my question: http://stackoverflow.com/questions/3017/how-to-generate-getters-and-setters-in-visual-studio

In Visual Studio 2008 there is the ability to auto generate getters and setters (accessors) by right clicking on a private variable -> Refactor -> Encapsulate Field...

This is great for a class that has 2 or 3 methods, but come on MS! When have you ever worked with a class that has a few accessors?

I am looking for a way to generate ALL with a few clicks (Eclipse folks out there will know what I am talking about - you can right click a class and select 'generate accessors'. DONE.). I really don't like spending 20 minutes a class clicking through wizards. I used to have some .NET 1.0 code that would generate classes, but it is long gone and this feature should really be standard for the IDE.

UPDATE: I might mention that I have found Linq to Entities and SQLMetal to be really cool ideas, and way beyond my simple request in the paragraph above.

  • Why not crank out a quick macro to do it?

    From Danimal
  • I have an "info class generator" application that you can use an excel sheet and it will generate the private members and the public get/set methods.

    You can download it for free from my website.

  • Possibly a macro. There are also addins (like ReSharper, which is great but commercial) capable of doing that quickly.

    From petr k.
  • Sorry, you really need to install Resharper to get approximately the same amount of refactoring support as you are used to in Eclipse.

    However, Resharper gives you a dialog very similar to the one you are used to in Eclipse: Screenshot

  • In 2008 I don't bother with Encapsulate Field. I use the new syntax for properties:

    public string SomeString { get; set; }
    
    From Eddie Deyo

How do I speed up data retrieval from .NET AD within ColdFusion

How can I optimize the following code, which currently takes over 2 minutes to retrieve and loop through 800+ records from a pool of over 100K records, returning 6 fields per record (adds approximately 20 seconds per additional field):

<cfset dllPath="C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\System.DirectoryServices.dll" />
<cfset LDAPPath="LDAP://" & arguments.searchPath />
<cfset theLookUp=CreateObject(".NET","System.DirectoryServices.DirectoryEntry", dllPath).init(LDAPPath) />
<cfset theSearch=CreateObject(".NET","System.DirectoryServices.DirectorySearcher", dllPath).init(theLookUp) />
<cfset theSearch.Set_Filter(arguments.theFilter) />
<cfset theObject = theSearch.FindAll() />

<cfloop index="row" from="#startRow#" to="#endRow#">
   <cfset QueryAddRow(theQuery) />
   <cfloop list="#columnList#" index="col">
     <cfloop from="0" to="#theObject.Get_Item(row).Get_Properties().Get_Item(col).Get_Count()-1#" index="item">
       <cftry>
         <cfset theQuery[col][theQuery.recordCount]=ListAppend(theQuery[col][theQuery.recordCount],theObject.Get_Item(row).Get_Properties().Get_Item(col).Get_Item(item),"|") />
         <cfcatch type="any">
         </cfcatch>
        </cftry>
      </cfloop>
    </cfloop>
  </cfloop>
  • It's been a long time since I touched CF, but I can give some hints in pseudo-code. For one thing, this expression is extremely inefficent:

    #theObject.Get_Item(row).Get_Properties().Get_Item(col).Get_Count()-1#

    Take the first part for example, Get_Item(row) - your code causes CF to go retrieve the row and its properties for each iteration of the #columnList# loop; and to top it all, you're doing that TWICE per iteration of columnlist (once for loop and again for the inner cfset). If you think about it, it only needs to retrieve the row for each iteration of the outer loop (from #sfstart# to #cfend). So, in pseudo-code do this:

    for each row between start and end

    cfset props = #theobject.get_item(row).get_properties()#

    for each col in #columnlist#

    cfset currentcol = #props.getitem(col)#

    cfset count = #currentcol.getcount() - 1#

    foreach item from 0 to #count#

    cfset #currentcol.getItem(item)# etc...

    Make sense? Every time you enter a loop, cache objects that will be reused in that scope (or child scopes) in a variable. That means you are only grabbing the column object once per iteration of the column loop. All variables defined in outer scopes are available in the inner scopes, as you can see in what I've done above. I know its tempting to cut and paste from previous lines, but don't. It only hurts you in the end.

    hope this helps,

    Oisin

    Peter Boughton : The count variable/caching is not necessary - CF will cache the from/to values for cfloop. (though not for a loop statement within cfscript)
    x0n : i would have thought so, but I was trying to demonstrate a point.
    From x0n
  • Additionally, using a cftry block in each loop is likely slowing this down quite a bit. Unless you are expecting individual rows to fail (and you need to continue from that point), I would suggest a single try/catch block for the entire process. Try/catch is an expensive operation.

    Brian : Yeah, but in this instance I expect bad or missing data...so that hit I have to live with. Question: Given the other suggestions, I assume that I could bypass that by checking the itemCount at the lowest scope
    x0n : again, I can't talk for CF, but in any other language with try/catch and exceptions it's not the try/catch that is expensive, it's the creation of the exception. This is because an exception contains a stack trace - that is to say, a list of all the methods that were called up until this point.
    From Ben Doom
  • How large is the list of items for the inner loop?

    Switching to an array might be faster if there is a significantly large number of items.

    I have implemented this alongside x0n's suggestions...

    <cfset dllPath="C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\System.DirectoryServices.dll" />
    <cfset LDAPPath="LDAP://" & arguments.searchPath />
    <cfset theLookUp=CreateObject(".NET","System.DirectoryServices.DirectoryEntry", dllPath).init(LDAPPath) />
    <cfset theSearch=CreateObject(".NET","System.DirectoryServices.DirectorySearcher", dllPath).init(theLookUp) />
    <cfset theSearch.Set_Filter(arguments.theFilter) />
    <cfset theObject = theSearch.FindAll() />
    
    <cfloop index="row" from="#startRow#" to="#endRow#">
    
        <cfset Props = theObject.get_item(row).get_properties() />
    
        <cfset QueryAddRow(theQuery) />
    
        <cfloop list="#columnList#" index="col">
    
         <cfset CurrentCol = Props.getItem(col) />
    
         <cfset ItemArray = ArrayNew(1)/>
         <cfloop from="0" to="#CurrentCol.getcount() - 1#" index="item">
          <cftry>
           <cfset ArrayAppend( ItemArray , CurrentCol.Get_Item(item) )/>
           <cfcatch type="any">
           </cfcatch>
          </cftry>
         </cfloop>
         <cfset theQuery[col][theQuery.recordCount] = ArrayToList( ItemArray , '|' )/>
    
        </cfloop>
    
    </cfloop>
    
    Brian : Okay, so I tested this out...and I see about a 30% reduction in the time it takes to run...which seems to make senes, because We've effectively reduced one third of the calls for items that are singular. I'm definitely interested in any othe optimzations that you can think of...
  • I would think that you'd want to stop doing so many evaluations inside of your loops and instead use variables to hold counts, pointers to the col object and to hold your pipe-delim string until you're ready to commit to the query object. If I've done the refactoring correctly, you should notice an improvement if you use the code below:

    <cfloop index="row" from="#startRow#" to="#endRow#">
    <cfset QueryAddRow(theQuery) />
    <cfloop list="#columnList#" index="col">
     <cfset PipedVals = "">
     <cfset theItem = theObject.Get_Item(row).Get_Properties().Get_Item(col)>
     <cfset ColCount = theItem.Get_Count()-1>
     <cfloop from="0" to="#ColCount#" index="item">
      <cftry>
      <cfset PipedVals = ListAppend(PipedVals,theItem.Get_Item(item),"|")>
      <cfcatch type="any"></cfcatch>
      </cftry>
     </cfloop>
     <cfset QuerySetCell(theQuery,col) = PipedVals>
    </cfloop>
    

    Brian : Okay, so let me digest a little...That call through the object is not akin to an array, but rather a set of chained calls? [slaps forehead] No wonder it's so heavy... I'm gonna go run this code back through...I'll bring some results back in about an hour
    kooshmoose : I think that's correct since you're instantiating a .NET object you should be able to set a variable to point to an object farther down the chain thus reducing the number of times you'd have to dig through the first object, especially if all you're looking for is a count of objects.
    kooshmoose : Did you compare against my slightly different solution? I doubt you'd see much of a performance difference, but it would be interesting to get some metrics on the difference between storing iterations in an array versus a string.
    Brian : It works out about the same...There are only two multi-value fields (memberOf and proxyAddresses), and so the load seems to spread out better... I suspect if I just retrieved the multi-values, it'd be quicker (I saw recently that array only had perf boost when randomly searching for values)
    kooshmoose : Ah, then you might receive a little more optimization by using a switch on the ColCount variable and doing the loop only when the value is NEQ 0. Otherwise, just do a QuerySetCell without first going to a string via the loop.
    kooshmoose : Also, if you know with 100% certainty the names of all the multi-value fields, then you could skip setting the count and looping over items for all single item columns.
    Brian : Good Point about the count test...
    From kooshmoose

Trying to sell Agile Development.

My work place is very archaic when it comes to software development. Essentially all projects are forced to use a waterfall approach requiring immense paperwork.

I've always been a fan of agile development and recently was approach by my manager asking for me to help him put together a proposal to some of the managers/directors to try to preach the benefits of Agile.

Like stated the target audience is the managers and directors where I work. Many of them hold to the mantra of "If it's not broke don't fix it".

I've found numerous case studies, instructional pages and all sorts of resources online however there are a few things I'm missing.

First of all is there a good resource for the type of documentation that is typically done with an agile process? Our QA manager is a stickler on documentation and unfortunately is likely to ignore the whole concept if he's not happy with the quality/amount of paperwork that is done.

Secondly are there any large companies (fortune 500 type size) that have publicly available information on their software development methodologies (specifically if such companies are using agile)?

  • This question raised some very good points on how introduce it, and also sell the idea to the relevant people.

    In terms of documentation - we document everything on a wiki on an "as needed basis". Anything that will be helpful to other developers (and other people in the company) goes there. We try and cut down on unnecessary documentation that never gets read, or gets out of sync with the code too easily.

    If the requirements in your company are different however - I don't see any harm in enforcing rules to say that everything needs to be documented to a particular level. Just remember - it's your software that makes money and you can't sell your documentation!

    From John Sibly
  • At my company we went through the book, Implementing Lean Software Development: From Concept to Cash, by Tom and Mary Poppendiek. It helped with selling to upper management. It is fairly light on what to do with documentation but that is largely the point of Agile and Lean development, to get rid of extra information that you never use. We took a look back and 90% of the documentation we were keeping was never used even though we did keep it up to date.

    What we do now for QA is to bring them in while we are developing and we all work very closely together so we don't need a pile of documentation to give to QA they know what we are working on and the tests are written while we are coding. It is a big transition but it so far it is working very well. There are less bugs that make it through the development cycle and it takes less time to fix them as they are typically found within a day of when they were introduced.

    From
  • Regarding documentation: What I've seen work well in Agile teams/departments/companies is setting up some simple criteria for deciding if something will be documented. Documentation for the sake of documentation just leads to massive amounts of wasted work resulting in binders of docs that no one ever reads.
    Here are some good questions to ask to help you decide if you should document something: Is the information redundant (is it already documented in the code or the bug database)? Does this information need to live on past your decision making process. Really? Check again.
    Does the information need to be scaled to people outside of your immediate geographic or temporal vicinity?

    Once you've decided you need this information to stay around, equally important is choosing the appropriate form of documentation. Email, bug report, Unit Test, wiki entry, ratified document signed in triplicate? Here are some questions to ask: Who is the intended audience? How long will the information be valid? The only thing worse than no documentation is misleading documentation. Can the "documentation" be automated as a unit test or automation test? Who is going to receive this documentation, and how do they prefer to receive it? How often will the documentation be checked?

    As for agile examples in actual companies, check out Ken Schwaber's two scrum books. They have many real world examples and are both very short and easy reads.

  • Agile Development can't be sold using documentations.

    Instead, ask your management to let you be Agile on a small pilot projet. For this pilot project, you must: - find a motivated end user to help you. - be free not to use the standard developping processes of your company.

    Then, succeed !!!

    From
  • Is this an internal project or is it for a product that you are selling or delivering to a third party? There is a big difference, but ultimately the user should determine what docs they need, not the devs.

    Inform the QA person that one of the key tenets of agile development is to test early and continuously. The tests then become the reference docs, instead of a spec. The conceptual docs can then be handled by a wiki or something similar.

    From
  • An agile process typically produces only one kind of non-code documentation: scenarios and use case descriptions. Implementing the scenarios and use cases is the base of the iterative process. Since one of the point of Agile is to get rid of pointless paperwork, you cannot sell it to an upper management with a paperwork fetish.

    "If it's not broken don't fix it" does not work for project management. If your project management merely looks non-broken, it's likely mediocre. You probably need to start by convincing your management that the process just might be improved. Once you are there, you can suggest experimenting with different approaches. Good places to start might be a small self-contained project, or a single bugfix cycle for a big project.

    One large company which openly endorses agile development is Google. Nowadays, you would be hard-pressed to find any other fortune-500 company that does not just suck at software development. I do have any reference handy, but you might start by the LEAN talk by Mary Poppendieck at Google HQ.

    From ddaa
  • Well, if it ain't broke, then don't fix it.

    If your development team is perfectly aligned with your IT department and management, producing value at a quality level and a rate that management is enthusiastic about, then they aren't going to be amenable to change.

    If, on the other hand, what you mean is that your audience is either:

    • Skeptical about yet another cure for a chronic problem, or
    • Believes that software development is the sound of keyboards clacking

    You've got a chance. The #1 thing that resonates with management is shorter delivery cycles. The #1 thing that resonates with IT is lowered integration drama.

    Your QA manager has reason to be concerned; your aim is to no longer produce those great comfortable reams of plans that gather dust so comfortingly in his/her office and which provide all the finger-pointing ammunition necessary when things go wrong. Hopefully he/she is not purely obstructionist and will be amenable to your arguments that agile processes centralize quality in the development process; instead of fighting for a scrap of attention and resources, quality becomes the driver. "No one knows or cares more about quality than you, Bob, and that's why this initiative is REALLY all about taking the concerns that you've been pushing for years and years, and FINALLY listening to what you've been telling us all along about how important QA is."

    Vis a vis "quantity of documentation" : you can dress up tests and user stories and make all the lovely bulleted Word sections and PowerPoint presentations that you want. The classic text is David Parnas' A Rational Design Process: How and Why to Fake It

  • Regarding selling agile, one angle of approach is proposing agile techniques as a way of reducing the "project crunch". By this I mean the overwhelming list of deferred issues, "99% complete" code and oversights that accumulate towards the end of the project. This normally causes a lot of stress and compromises in order to meet the deadline. If a development organisation has never experienced project crunch, it would be a big surprise to me. Therefore I think this is a neat way of locking into a some shared pain and suggesting agile techniques as a load-levelling technique should provoke some interest that can lead onto a more complete discussion of the methodology(s).

  • Not exactly an answer to your question, but I have to say this.

    There is a lot of room to improve waterfall process, and agile development is not necessarily the best choice. While agile is favored by current fashion, it's not a silver bullet. It might be worthwhile to consider all possibilities and present your management with a study of several possible processes with technical and business analysis. If agile comes out as best in comparision, it would be a very good selling point.

    From ima
  • I agree with a previous poster that agile is not a silver bullet. For me, it simply helped make some sense of the chaos, and gave me common-sense principles to get behind.

    The first step is to understand the key reasons WHY you might want to consider another methodology. Once you have that, then you can start to review your practices with that in mind and make informed decisions about the right thing to do.

    Now, it might be that the right thing to do is to get permission from up high and change everything. But remember, you can adopt any number of agile philosophies without ever saying you "do agile" or having to sell this to upper management. You might still follow a Gantt chart and plan a number of months in advance etc, but you can still pair program or write unit tests first or tackle one complete feature at a time providing you hit your time commitments. And changing your approach internally might just make selling the idea later on easier...

  • Just to chime in with some additional information about my specific situation.

    The applications in question are all internal, used to help with various business processes. My job is a government job, so politics are rampant, money is wasted. Essentially every stereotype that plagues government jobs is probably in effect.

    The development we have been doing lately is small projects; small teams (3-6 people) and should only take about 6 months of work.

    Unfortunately our "one size fits all" SDLC easily takes a year to get all the necessary paper work done. It destroys progress.

    Our biggest problem is the mountains of paperwork. When documentation exceeds source code by a order of magnitude there is a problem. Especially when no one bothers to read or update the documentation.

    Unfortunately the people in charge of the SDLC are rather set in their ways, they have been doing it this way for longer than I have been alive. They want to do it this way until they retire (so they don't have to learn something new).

    What my manager (and myself) want to do is to try to "sell" agile as a new and better approach for these small projects. We want to show the managers that other companies are using it and are having great success with it. We essentially want to sell them on the buzz word despite it being unlikely that they'll understand it.

    As mentioned our biggest stumbling block is the guy who oversees the whole SDLC.

    ima : If that description is correct, the only practical way is to establish some kind of personal contact with people in power and convince them that the new process would not by any way affect them or give them more work.
    From Erratic
  • That does sound like a real challenge to overcome. Documentation tends to be quite a painful subject.

    Have you tried to use prior projects and their artifacts as "evidence" of something not being right, in order to "buy" the opportunity to try something different on a small project? Showing someone reams of outdated and useless documentation should allow you to at least ask the question "what could we do differently to solve this" ?

    Does the effect of this get felt by your customers? Could you perhaps approach them to partner up on this change? I feel sure you can create more value for your customer a lot sooner, maybe they'd be good advocates for change?