What methods of caching, other than to file or database, are available?

Currently I know of only two ways to cache data (I use PHP but I assume that the same will apply to most languages).

1. Save the cache to a file
2. Save the cache to a large DB field

Are there any other (perhaps better) ways of caching or is it really just this simple?

From stackoverflow Teifion

• You can also cache in memory which is much more efficient. Try memcached.

  From Sebastjan Trepča

• Maybe you want to explicit more precisely what you want to cache. You have all this opportunities to cache:

  • Accessing the Data Base where you cache the data first correctly tuning your RDBMS, then using a layer to delegate the decision to detect multiple queries for the same data (with AdoDB for example.)
  • Extracting calculations from loops in the code so you don't compute the same value multiple times. Here your third way: storing results in the session for the user.
  • Precompiling the PHP code with an extension like APC Cache. This way you don't have to compile the same PHP code for every request.
  • The page sent to the user making sure you're setting the right META tags (do a good thing for the world and don't use ETL at least absolutly necessary); or maybe making dynamic pages completely static (having a batch process that generates .html pages); or by using a proxy cache like Squid.
  • Prefetching and by this I refer all those opportunities you have to improve the user experience just by doing things while the user don't look your way. For example, preloading IMG tags in the HTML file, tunning the RDBMS for prefectching, precomputing results storing complex computations in the database, etc.

  From my experience, I'd bet you that your code can be improved a lot before we start to talk about caching things. Consider, for example, how well structured is the navigation of your site and how well you control the user experience. Then check your code with a tool like XDebug.

  Verify also how well are you making your SQL queries and how well are you indexing your tables. Then check your code again to look for opportunities to apply the rule "read many times but write just once"

  Use a simple tool like YSlow to hint other simple things to improve. Check your code again looking for opportunities to put logic in the browser (via JavaScript)

  From ggasp

• Seconding memcached, does the simple stuff well and can go distributive and all that jazz if you need it too

  From Kevin Sheffield

• If you're using Apache, you can use mod_rewrite to statically cache your web pages. Lets say you're using PHP, and you have a request for "/somepage.php". In your .htaccess file you put the following:

  RewriteEngine on
  RewriteCond %{QUERY_STRING} ^$ # let's not cache urls with queries
  RewriteCond %{REQUEST_METHOD} ^GET$ # or POST/PUT/DELETE requests
  RewriteCond static_cache/%{REQUEST_URI} -s # Check that this file exists and is > 0 bytes
  RewriteRule (^.*$) static_cache$1 [L] # If all the conditions are met, we rewrite this request to hit the static cache instead
  

  If your cache turns up empty, the request is handled by your php script as usual, so now it's simply a matter of making your php script store the resulting html in the cache. The simplest way to do this is using another htaccess rule to prepend end append a couple of php files to all your php requests (this might or might not be a good idea, depending on your application):

  php_value auto_prepend_file "pre_cache.php"
  php_value auto_append_file "post_cache.php"
  

  Then you'd do something like this:

  pre_cache.php:

  ob_start();
  

  post_cache.php:

  $result = ob_get_flush();
  if(!$_SERVER['QUERY_STRING']) { # Again, we're not caching query string requests
    file_put_contents("static_cache/" + __FILE__, $result);
  }
  

  With some additional regular expressions in the .htaccess file we could probably start caching query string requests as well, but I'll leave that as an exercise for the reader :)

  From Erlend Halvorsen

ConfigurationManager.AppSettings Performance Concerns

I plan to be storing all my config settings in my application's app.config section (using the ConfigurationManager.AppSettings class). As the user changes settings using the app's UI (clicking checkboxes, choosing radio buttons, etc.), I plan to be writing those changes out to the AppSettings. At the same time, while the program is running I plan to be accessing the AppSettings constantly from a process that will be constantly processing data. Changes to settings via the UI need to affect the data processing in real-time, which is why the process will be accessing the AppSettings constantly.

Is this a good idea with regard to performance? Using AppSettings is supposed to be "the right way" to store and access configuration settings when writing .Net apps, but I worry that this method wasn't intended for a constant load (at least in terms of settings being constantly read).

If anyone has experience with this, I would greatly appreciate the input.

Update: I should probably clarify a few points.

This is not a web application, so connecting a database to the application might be overkill simply for storing configuration settings. This is a Windows Forms application.

According to the MSDN documention, the ConfigurationManager is for storing not just application level settings, but user settings as well. (Especially important if, for instance, the application is installed as a partial-trust application.)

Update 2: I accepted lomaxx's answer because Properties does indeed look like a good solution, without having to add any additional layers to my application (such as a database). When using Properties, it already does all the caching that others suggested. This means any changes and subsequent reads are all done in memory, making it extremely fast. Properties only writes the changes to disk when you explicitly tell it to. This means I can make changes to the config settings on-the-fly at run time and then only do a final save out to disk when the program exits.

Just to verify it would actually be able to handle the load I need, I did some testing on my laptop and was able to do 750,000 reads and 7,500 writes per second using Properties. That is so far above and beyond what my application will ever even come close to needing that I feel quite safe in using Properties without impacting performance.

Thank you, everyone, for your helpful answers.

From stackoverflow Dylan Bennett

• Someone correct me if I'm wrong, but I don't think that AppSettings is typically meant to be used for these type of configuration settings. Normally you would only put in settings that remain fairly static (database connection strings, file paths, etc.). If you want to store customizable user settings, it would be better to create a separate preferences file, or ideally store those settings in a database.

  From bcwood

• Could I ask why you're not saving the user's settings in a database?

  Generally, I save application settings that are changed very infrequently in the appSettings section (the default email address error logs are sent to, the number of minutes after which you are automatically logged out, etc.) The scope of this really is at the application, not at the user, and is generally used for deployment settings.

  From Shawn Simon

• one thing I would look at doing is caching the appsettings on a read, then flushing the settings from the cache on the write which should minimize the amount of actual load the server has to deal with for processing the appSettings.

  Also, if possible, look at breaking the appSettings up into configSections so you can read write and cache related settings.

  Having said all that, I would seriously consider looking at storing these values in a database as you seem to actually be storing user preferences, and not application settings.

  From lomaxx

• I should probably clarify a few points.

  This is not a web application, so connecting a database to the application might be overkill simply for storing configuration settings. This is a Windows Forms application.

  According to the MSDN documention, the ConfigurationManager is for storing not just application level settings, but user settings as well. (Especially important if, for instance, the application is installed as a partial-trust application.)

  From Dylan Bennett

• Check out SQLite, it seems like a good option for this particular scenario.

  From Shawn Simon

• I would not use config files for storing user data. Use a db.

  From Kevin Sheffield

• since you're using a winforms app, if it's in .net 2.0 there's actually a user settings system (called Properties) that is designed for this purpose. This article on MSDN has a pretty good introduction into this

  If you're still worried about performance then take a look at SQL Compact Edition which is similar to SQLite but is the Microsoft offering which I've found plays very nicely with winforms and there's even the ability to make it work with Linq

  From lomaxx

• Dylan,

  Don't use the application config file for this purpose, use a SQL DB (SQLite, MySQL, MSSQL, whatever) because you'll have to worry less about concurrency issues during reads and writes to the config file.

  You'll also have better flexibility in the type of data you want to store. The appSettings section is just a key/value list which you may outgrow as time passes and as the app matures. You could use custom config sections but then you're into a new problem area when it comes to the design.

  From Kev

• The appSettings isn't really meant for what you are trying to do.

  When your .NET application starts, it reads in the app.config file, and caches its contents in memory. For that reason, after you write to the app.config file, you'll have to somehow force the runtime to re-parse the app.config file so it can cache the settings again. This is unnecessary

  The best approach would be to use a database to store your configuration settings.

  Barring the use of a database, you could easily setup an external XML configuration file. When your application starts, you could cache its contents in a NameValueCollection object or HashTable object. As you change/add settings, you would do it to that cached copy. When your application shuts down, or at an appropriate time interval, you can write the cache contents back out to file.

  From Terrapin

How to learn ADO.NET

I need to learn ADO.NET to build applications based on MS Office. I have read a good deal about ADO.NET in the MSDN Library, but everything seems rather messy to me.

What's the basics one must figure out when using ADO.NET? I think a few key words will suffice to let me organize my learning.

From stackoverflow $pageUsers[$entry.posts[0].author].name

• There are three key components (assuming ur using SQL server):

  SQLConnection, SqlCommand, SqlDataReader

  (if you're using something else, replace "Sql" with "Something", like "MySqlConnection", "OracleCommand"

  Everything else is just built on top of that Example 1:

  using (SqlConnection connection = new SqlConnection("CONNECTION STRING"))
  using (SqlCommand command = new SqlCommand())
  {
    command.commandText = "SELECT Name FROM Users WHERE Status = @OnlineStatus";
    command.Connection = connection;
    command.Parameters.Add("@OnlineStatus", SqlDbType.Int).Value = 1; //replace with enum
    connection.Open();
    using (SqlDataReader dr = command.ExecuteReader))
    {
        List<string> onlineUsers = new List<string>();
        while (dr.Read())
        {
           onlineUsers.Add(dr.GetString(0));
        }
    }
  }
  

  Example 2:

  using (SqlConnection connection = new SqlConnection("CONNECTION STRING"))
  using (SqlCommand command = new SqlCommand())
  {
    command.commandText = "DELETE FROM Users where Email = @Email";
    command.Connection = connection;
    command.Parameters.Add("@Email", SqlDbType.VarChar, 100).Value = "user@host.com";
    connection.Open();
    command.ExecuteNonQuery();
  }
  

  From Karl Seguin

• Another way of gettinga command object is to call "connection.CreateCommand()". That way you shouldn't have to set the Commection property on the command object.

  From Mark Cidade

Passing multidimensional arrays as function arguments in C

In C can I pass a multidimensional array to a function as a single argument when I don't know what the dimensions of the array are going to be ?

In addition my multidimensional array may contain types other than strings.

From stackoverflow David

• Yes. Take a look at this page.

  From Jeremy Privett

• Simply put:

  An array is a pointer to a vector in memory. So, an array of arrays is a pointer to a pointer to a vector in memory.

  So:

  /**
  In a near Galaxy you have
  **/
  char a[100][100];
  giveMeTheAnswer( a ); 
  
  /**
  The event horizon must accept that multiverse
  **/
  void giveMeTheAnswer( char** pMatrix ) {
          pMatrix[0][0] = '4';
          pMatrix[0][1] = '2';
  };
  

  Johannes Schaub - litb : yeah that is totally wrong :) try pMatrix[1][0] = '4'; :)

  newacct : Wow that is so wrong. You completely confused arrays of arrays and arrays of pointers. They are completely different.

  John Bode : The type of pMatrix is `char (*)[100]` (pointer to 100-element array of char), not `char **`. Secondly, *arrays are not pointers*. Arrays and pointers are *different* objects that, in some contexts, can be used interchangeably.
  From ggasp

• You can do this with any data type. Simply make it a double pointer:

  typedef struct {
    int myint;
    char* mystring;
  } data;
  
  data** array;
  

  But don't forget you still have to malloc the variable, and it does get a bit complex:

  //initialize
  int x,y,w,h;
  w = 10; //width of array
  h = 20; //height of array
  
  //malloc the 'y' dimension
  array = malloc(sizeof(data*) * h);
  
  //iterate over 'y' dimension
  for(y=0;y<h;y++){
    //malloc the 'x' dimension
    array[y] = malloc(sizeof(data) * w);
  
    //iterate over the 'x' dimension
    for(x=0;x<w;x++){
      //malloc the string in the data structure
      array[y][x].mystring = malloc(sizeof(char) * 50); //50 chars
  
      //initialize
      array[y][x].myint = 6;
      array[y][x].mystring = "w00t";
    }
  }
  

  The code to deallocate the structure looks similar - don't forget to call free() on everything you malloced! (Also, in robust applications you should check the return of malloc().)

  Now let's say you want to pass this to a function. You can still use the double pointer, because you probably want to do manipulations on the data structure, not the pointer to pointers of data structures:

  int whatsMyInt(data** arrayPtr, int x, int y){
    return arrayPtr[y][x].myint;
  }
  

  Call this function with:

  printf("My int is %d.\n", whatsMyInt(array, 2, 4));
  

  Output:

  My int is 6.
  

  From superjoe30

• Pass an explicit pointer to the first element with the array dimensions as separate parameters. For example, to handle arbitrarily sized 2-d arrays of int:

  void func_2d(int *p, size_t M, size_t N)
  {
    size_t i, j;
    ...
    p[i*N+j] = ...;
  }
  

  which would be called as

  ...
  int arr1[10][20];
  int arr2[5][80];
  ...
  func_2d(&arr1[0][0], 10, 20);
  func_2d(&arr2[0][0], 5, 80);
  

  Same principle applies for higher-dimension arrays:

  func_3d(int *p, size_t X, size_t Y, size_t Z)
  {
    size_t i, j, k;
    ...
    p[i*Y+j*Z+k] = ...;
    ...
  }
  ...
  arr2[10][20][30];
  ...
  func_3d(&arr[0][0][0], 10, 20, 30);
  

  From John Bode

How can I tell if a web client is blocking ads?

I'd like to get some statistics on how many people coming to my site have set their browser to block ads. Any tips on the best way to do this?

From stackoverflow Mark Harrison

• I suppose you could compare the ad prints with the page views on your website (which you can get from your analytics software).

  From Vaibhav

• Since programs like AdBlock actually never request the advert, you would have to look the server logs to see if the same user accessed a webpage but didn't access an advert. This is assuming the advert is on the same server.

  If your adverts are on a separate server, then I would suggest it's impossible to do so.

  The best way to stop users from blocking adverts, is to have inline text adverts which are generated by the server and dished up inside your html.

  From GateKiller

• Add the user id to the request for the ad:

  <img src="./ads/viagra.jpg?{user.id}"/>
  

  that way you can check what ads are seen by which users.

  From Jon Works

• You need to think about the different ways that ads are blocked. The first thing to look at is whether they are running noscript, so you could add a script that would check for that.

  The next thing is to see if they are blocking flash, a small movie should do that.

  If you look at the adblock site, there is some indication of how it does blocking: http://adblockplus.org/en/faq_internal#elemhide

  If you look further down that page, you will see that conventional chrome probing will not work, so you need to try and parse the altered DOM.

  From UberAlex

• AdBlock forum says this is used to detect AdBlock. After some tweaking you could use this to gather some statistics.

  <script language="JavaScript" type="text/JavaScript">
  
  setTimeout('detect_abp()', 10000);
  var isFF = (navigator.userAgent.indexOf("Firefox") > -1) ? true : false;
  var hasABP = false;
  
  function detect_abp()
  {
     if(isFF)
     {
  
           if(Components.interfaces.nsIAdblockPlus != undefined)
          {
             hasABP = true;
          }
          else
          {
            var AbpImage = document.createElement("IMG");
            AbpImage.id = 'abp_detector';
            AbpImage.src = '/textlink-ads.jpg';
              AbpImage.style.width = '0px';
            AbpImage.style.height = '0px';
            AbpImage.style.top = '-1000px';
            AbpImage.style.left = '-1000px';
            document.body.appendChild(AbpImage);
            hasABP = (document.getElementById('abp_detector').style.display == 'none');
  
               var e = document.getElementsByTagName("iframe");
              for (var i = 0; i < e.length; i++)
              {
                 if(e[i].clientHeight == 0)
                  {
                      hasABP = true;
                  }
              }
                if(hasABP == true)
                {
                 history.go(1);
                  location = "http://www.tweaktown.com/supportus.html";
                window.location(location);       
               }
          }
        }
      }
      </script> 
  

  From Michal Sznajder

graph serialization

I'm looking for a simple algorithm to 'serialize' a directed graph. In particular I've got a set of files with interdependencies on their execution order, and I want to find the correct order at compile time. I know it must be a fairly common thing to do - compilers do it all the time - but my google-fu has been weak today. What's the 'go-to' algorithm for this?

From stackoverflow Kieron

• I would expect tools that need this simply walk the tree in a depth-first manner and when they hit a leaf, just process it (e.g. compile) and remove it from the graph (or mark it as processed, and treat nodes with all leaves processed as leaves).

  As long as it's a DAG, this simple stack-based walk should be trivial.

  sleske : Yes, that's how you do it. It's called a depth-first search (DFS). And unless you are certain you have a DAG, you mustcheck for back edges, otherwise a cycle will send you into an infinite loop.
  From Kevin Ballard

• I've come up with a fairly naive recursive algorithm (pseudocode):

  Map<Object, List<Object>> source; // map of each object to its dependency list
  List<Object> dest; // destination list
  
  function resolve(a):
      if (dest.contains(a)) return;
      foreach (b in source[a]):
          resolve(b);
      dest.add(a);
  
  foreach (a in source):
      resolve(a);
  

  The biggest problem with this is that it has no ability to detect cyclic dependencies - it can go into infinite recursion (ie stack overflow ;-p). The only way around that that I can see would be to flip the recursive algorithm into an interative one with a manual stack, and manually check the stack for repeated elements.

  Anyone have something better?

  From Kieron

• Topological Sort. From Wikipedia:

  In graph theory, a topological sort or topological ordering of a directed acyclic graph (DAG) is a linear ordering of its nodes in which each node comes before all nodes to which it has outbound edges. Every DAG has one or more topological sorts.

  Pseudo code:

  L ← Empty list where we put the sorted elements
  Q ← Set of all nodes with no incoming edges
  while Q is non-empty do
      remove a node n from Q
      insert n into L
      for each node m with an edge e from n to m do
          remove edge e from the graph
          if m has no other incoming edges then
              insert m into Q
  if graph has edges then
      output error message (graph has a cycle)
  else 
      output message (proposed topologically sorted order: L)
  

  Benjol : Eh... copied directly off wikipedia?

  Jason S : yes, please cite sources
  From Andrew Peters

• If the graph contains cycles, how can there exist allowed execution orders for your files? It seems to me that if the graph contains cycles, then you have no solution, and this is reported correctly by the above algorithm.

  sleske : Yes, a topological sort is not possible if a graph contains cycles. This corresponds to the real world: If I ask you to do A before B, *and* B before A, there's no way you're gonna satisfy me ;-).
  From $pageUsers[$post.author].name

Best way to cache data in .NET

I am in the process of figuring out a cache strategy for our current setup, currently have multiple web servers and wanted to know what is the best way to cache data in this environment. I have done research about MemCache and the native asp.net caching but wanted to get some feedback first. Should I go with a Linux box if I use MemCache or a win32 port of MemCache. Thanks!

From stackoverflow Omar Ramos

• What about checking out Microsoft Velocity (http://code.msdn.microsoft.com/velocity)? Another option if you don't want to start using Microsoft CTP-ware is to check out Nache which allows distributed cache/session state management (http://www.alachisoft.com/ncache/)

  From Jonas Follesø

• http://www.danga.com/memcached/

  worked awesome for me and have heard nothing but goodness about it

  From Kevin Sheffield

• Dare Obasanjo has a pretty good blog post about this topic. You really need to assess what it is you're caching, why you're caching it and what your needs are before you can make a decision on a caching strategy.

  From lomaxx