Home » Caching

Category Archives: Caching

Abstract Out Sitecore FileWatcher Logic Which Monitors Rendering Files on the File System

While digging through code of Sitecore.IO.FileWatcher subclasses this weekend, I noticed a lot of code similarities between the LayoutWatcher and XslWatcher classes, and thought it might be a good idea to abstract this logic out into a new base abstract class so that future FileWatchers which monitor other renderings on the file system can easily be added without having to write much logic.

Before I move forward on how I did this, let me explain what the LayoutWatcher FileWatcher does. The LayoutWatcher FileWatcher clears the html cache of all websites defined in Sitecore when it determines that a layout file (a.k.a .aspx) or sublayout file (a.k.a .ascx) has been changed, deleted, renamed or added.

Likewise, the XslWatcher FileWatcher does the same thing for XSLT renderings but also clears the XSL cache along with the html cache.

Ok, now back to the abstraction. I came up with the following class to serve as the base class for any FileWatcher that will monitor renderings that live on the file system:

using System;
using System.Collections.Generic;
using System.Linq;

using Sitecore.Configuration;
using Sitecore.Diagnostics;
using Sitecore.IO;
using Sitecore.Web;

namespace Sitecore.Sandbox.IO.Watchers
{
    public abstract class RenderingFileWatcher : FileWatcher
    {
        private string RenderingFileModifiedMessage { get; set; }

        private string RenderingFileDeletedMessage { get; set; }

        private string RenderingFileRenamedMessage { get; set; }

        private string FileWatcherErrorMessage { get; set; }

        private object Owner { get; set; }

        public RenderingFileWatcher(string configPath)
            : base(configPath)
        {
            SetMessages();
        }

        private void SetMessages()
        {
            string modifiedMessage = GetRenderingFileModifiedMessage();
            AssertNotNullOrWhiteSpace(modifiedMessage, "modifiedMessage", "GetRenderingFileModifiedMessage() cannot return null, empty or whitespace!");
            RenderingFileModifiedMessage = modifiedMessage;

            string deletedMessage = GetRenderingFileDeletedMessage();
            AssertNotNullOrWhiteSpace(deletedMessage, "deletedMessage", "GetRenderingFileDeletedMessage() cannot return null, empty or whitespace!");
            RenderingFileDeletedMessage = deletedMessage;

            string renamedMessage = GetRenderingFileRenamedMessage();
            AssertNotNullOrWhiteSpace(renamedMessage, "renamedMessage", "GetRenderingFileRenamedMessage() cannot return null, empty or whitespace!");
            RenderingFileRenamedMessage = renamedMessage;

            string errorMessage = GetFileWatcherErrorMessage();
            AssertNotNullOrWhiteSpace(errorMessage, "errorMessage", "GetFileWatcherErrorMessage() cannot return null, empty or whitespace!");
            FileWatcherErrorMessage = errorMessage;

            object owner = GetOwner();
            Assert.IsNotNull(owner, "GetOwner() cannot return null!");
            Owner = owner;
        }

        protected abstract string GetRenderingFileModifiedMessage();

        protected abstract string GetRenderingFileDeletedMessage();

        protected abstract string GetRenderingFileRenamedMessage();

        protected abstract string GetFileWatcherErrorMessage();

        protected abstract object GetOwner();

        private void AssertNotNullOrWhiteSpace(string argument, string argumentName, string errorMessage)
        {
            Assert.ArgumentCondition(!string.IsNullOrWhiteSpace(argument), argumentName, errorMessage);
        }

        protected override void Created(string fullPath)
        {
            try
            {
                Log.Info(string.Format("{0}: {1}", RenderingFileModifiedMessage, fullPath), Owner);
                ClearCaches();
            }
            catch (Exception ex)
            {
                Log.Error(FileWatcherErrorMessage, ex, Owner);
            }
        }

        protected override void Deleted(string filePath)
        {
            try
            {
                Log.Info(string.Format("{0}: {1}", RenderingFileDeletedMessage, filePath), Owner);
                ClearCaches();
            }
            catch (Exception ex)
            {
                Log.Error(FileWatcherErrorMessage, ex, Owner);
            }
        }

        protected override void Renamed(string filePath, string oldFilePath)
        {
            try
            {
                Log.Info(string.Format("{0}: {1}. Old path: {2}", RenderingFileRenamedMessage, filePath, oldFilePath), Owner);
                ClearCaches();
            }
            catch (Exception ex)
            {
                Log.Error(FileWatcherErrorMessage, ex, this);
            }
        }

        protected virtual void ClearCaches()
        {
            ClearHtmlCaches();
        }

        protected virtual void ClearHtmlCaches()
        {
            IEnumerable<SiteInfo> siteInfos = GetSiteInfos();
            if (IsNullOrEmpty(siteInfos))
            {
                return;
            }

            foreach(SiteInfo siteInfo in siteInfos)
            {
                if (siteInfo.HtmlCache != null)
                {
                    Log.Info(string.Format("Clearing Html Cache for site: {0}", siteInfo.Name), Owner);
                    siteInfo.HtmlCache.Clear();
                }
            }
        }

        protected virtual IEnumerable<SiteInfo> GetSiteInfos()
        {
            IEnumerable<string> siteNames = GetSiteNames();
            if (IsNullOrEmpty(siteNames))
            {
                return Enumerable.Empty<SiteInfo>();
            }

            IList<SiteInfo> siteInfos = new List<SiteInfo>();
            foreach(string siteName in siteNames)
            {
                SiteInfo siteInfo = Factory.GetSiteInfo(siteName);
                if(siteInfo != null)
                {
                    siteInfos.Add(siteInfo);
                }
            }

            return siteInfos;
        }

        protected virtual IEnumerable<string> GetSiteNames()
        {
            IEnumerable<string> siteNames = Factory.GetSiteNames();
            if(IsNullOrEmpty(siteNames))
            {
                return Enumerable.Empty<string>();
            }

            return siteNames;
        }

        protected virtual bool IsNullOrEmpty<T>(IEnumerable<T> collection)
        {
            if (collection == null || !collection.Any())
            {
                return true;
            }

            return false;
        }
    }
}

The above class defines five abstract methods which subclasses must implement. Data returned by these methods are used when logging information or errors in the Sitecore log. The SetMessages() method vets whether subclass returned these objects correctly, and sets them in private properties which are used in the Created(), Deleted() and Renamed() methods.

The Created(), Deleted() and Renamed() methods aren’t really doing anything different from each other — they are all clearing the html cache for each Sitecore.Web.SiteInfo instance returned by the GetSiteInfos() method, though I do want to point out that each of these methods are defined on the Sitecore.IO.FileWatcher base class and serve as event handlers for file system file actions:

  • Created() is invoked when a new file is dropped in a directory or subdirectory being monitored, or when a targeted file is changed.
  • Deleted() is invoked when a targeted file is deleted.
  • Renamed() is invoked when a targeted file is renamed.

In order to ascertain whether the code above works, I need a subclass whose instance will serve as the actual FileWatcher. I decided to build the following subclass which will monitor Razor files under the /Views directory of my Sitecore website root:

namespace Sitecore.Sandbox.IO.Watchers
{
    public class RazorViewFileWatcher : RenderingFileWatcher
    {
        public RazorViewFileWatcher()
            : base("watchers/view")
        {
        }
        
        protected override string GetRenderingFileModifiedMessage()
        {
            return "Razor View modified";
        }

        protected override string GetRenderingFileDeletedMessage()
        {
            return "Razor View deleted";
        }

        protected override string GetRenderingFileRenamedMessage()
        {
            return "Razor View renamed";
        }

        protected override string GetFileWatcherErrorMessage()
        {
            return "Error in RazorViewFileWatcher";
        }

        protected override object GetOwner()
        {
            return this;
        }
    }
}

I’m sure Sitecore has another way of clearing/replenishing the html cache for Sitecore MVC View and Controller renderings — if you know how this works in Sitecore, please share in a comment, or better yet: please share in a blog post — but I went with this just for testing.

There’s not much going on in the above class. It’s just defining the needed methods for its base class to work its magic.

I then had to add the configuration needed by the RazorViewFileWatcher above in the following patch include configuration file:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="ViewsFolder" value="/Views" />
    </settings>
    <watchers>
      <view>
        <folder ref="settings/setting[@name='ViewsFolder']/@value"/>
        <filter>*.cshtml</filter>
      </view>
    </watchers>
  </sitecore>
</configuration>

As I’ve discussed in this post, FileWatchers in Sitecore are Http Modules — these must be registered under <modules> of <system.webServer> of your Web.config:


<!-- lots of stuff up here -->

<system.webServer>
    <modules runAllManagedModulesForAllRequests="true">

		<!-- stuff here -->
      
		<add type="Sitecore.Sandbox.IO.Watchers.RazorViewFileWatcher, Sitecore.Sandbox" name="SitecoreRazorViewFileWatcher" />

		<!-- stuff here as well -->

	</modules>

	<!-- more stuff down here -->

</system.webServer>

<!-- even more stuff down here -->

Let’s see if this works.

I decided to choose the following Razor file which comes with Web Forms For Marketers 8.1 Update-2:

razor-view-file-watcher-cshtml-1

I first performed a copy and paste of the Razor file into a new file:

razor-view-file-watcher-cshtml-2

I then deleted the new Razor file:

razor-view-file-watcher-cshtml-3

Next, I renamed the Razor file:

razor-view-file-watcher-cshtml-4

When I looked in my Sitecore log file, I saw that all operations were executed:

razor-view-file-watcher-log

If you have any thoughts on this, please drop a comment.

Yet Another Way to Store Data Outside of the Sitecore Experience Platform

Last February, Sitecore MVP Nick Wesselman shared an awesome blog post on storing data outside of the Sitecore® Experience Platform™ using the NHibernate ORM framework — if you haven’t had a chance to read this, I strongly recommend that you do — which is complete magic, and a simple solution where you don’t have to worry about spinning up your own database tables for storing information.

But, let’s suppose you aren’t allowed to use an ORM like NHibernate in your solution for some reason — I won’t go into potential reasons but let’s make pretend there is one — and you have to find a way to store Sitecore specific information but don’t want to go through the trouble of spinning up a new Sitecore database due to the overhead involved. What can you do?

Well, you can still store information in a non-Sitecore database using the Sitecore API. The following “proof of concept” does this, and is basically modeled after how Sitecore manages data stored in the IDTable and Links Database.

The code in the following “proof of concept” adds/retrieves/deletes alternative URLs for Sitecore Items in the following custom database table:

itemurls-sql-table

I’m not going to talk much about the SQL table or SQL statements used in this “proof of concept” since it’s beyond the scope of this post.

Of course we all love things that are performant — and our clients love when we make things performant — so I decided to start off my solution using the following adapter — this includes the interface and concrete class — for an instance of Sitecore.Caching.Cache (this lives in Sitecore.Kernel.dll):

using System;
using System.Collections;

using Sitecore.Caching;
using Sitecore.Data;
using Sitecore.Diagnostics.PerformanceCounters;

namespace Sitecore.Sandbox.Caching
{
    public interface ICacheProvider
    {
        bool CacheWriteEnabled { get; set; }

        int Count { get; }

        CachePriority DefaultPriority { get; set; }

        bool Enabled { get; set; }

        AmountPerSecondCounter ExternalCacheClearingsCounter { get; set; }

        ID Id { get; }

        long MaxSize { get; set; }

        string Name { get; }

        long RemainingSpace { get; }

        bool Scavengable { get; set; }

        long Size { get; }

        object SyncRoot { get; }

        object this[object key] { get; }

        void Add(ID key, ICacheable data);

        void Add(ID key, string value);

        void Add(string key, ICacheable data);

        void Add(string key, ID value);

        Cache.CacheEntry Add(string key, string data);

        void Add(ID key, object data, long dataLength);

        void Add(object key, object data, long dataLength);

        void Add(string key, object data, long dataLength);

        void Add(object key, object data, long dataLength, DateTime absoluteExpiration);

        void Add(object key, object data, long dataLength, TimeSpan slidingExpiration);

        Cache.CacheEntry Add(string key, object data, long dataLength, DateTime absoluteExpiration);

        void Add(string key, object data, long dataLength, EventHandler<EntryRemovedEventArgs> removedHandler);

        Cache.CacheEntry Add(string key, object data, long dataLength, TimeSpan slidingExpiration);

        void Add(object key, object data, long dataLength, TimeSpan slidingExpiration, DateTime absoluteExpiration);

        void Clear();

        bool ContainsKey(ID key);

        bool ContainsKey(object key);

        ArrayList GetCacheKeys();

        ArrayList GetCacheKeys(string keyPrefix);

        Cache.CacheEntry GetEntry(object key, bool updateAccessed);

        object GetValue(object key);

        void Remove(object key);

        void Remove<TKey>(Predicate<TKey> predicate);

        void RemoveKeysContaining(string value);

        void RemovePrefix(string keyPrefix);

        void Scavenge();
    }
}
using System;
using System.Collections;

using Sitecore.Caching;
using Sitecore.Data;
using Sitecore.Diagnostics;
using Sitecore.Diagnostics.PerformanceCounters;

namespace Sitecore.Sandbox.Caching
{
    public class CacheProvider : ICacheProvider
    {
        private Cache Cache { get; set; }

        public CacheProvider(string cacheName, string cacheSize)
        {
            Assert.ArgumentNotNullOrEmpty(cacheName, "cacheName");
            Assert.ArgumentNotNullOrEmpty(cacheSize, "cacheSize");
            Cache = new Cache(cacheName, StringUtil.ParseSizeString(cacheSize));
        }

        public bool CacheWriteEnabled 
        {
            get
            {
                return Cache.CacheWriteEnabled;
            }
            set
            {
                Cache.CacheWriteEnabled = value;
            }
        }

        public int Count 
        {
            get
            {
                return Cache.Count;
            }
        }

        public CachePriority DefaultPriority 
        {
            get
            {
                return Cache.DefaultPriority;
            }
            set
            {
                Cache.DefaultPriority = value;
            }
        }

        public bool Enabled 
        {
            get
            {
                return Cache.Enabled;
            }
            set
            {
                Cache.Enabled = value;
            }
        }

        public AmountPerSecondCounter ExternalCacheClearingsCounter 
        {
            get
            {
                return Cache.ExternalCacheClearingsCounter;
            }
            set
            {
                Cache.ExternalCacheClearingsCounter = value;
            }
        }

        public ID Id 
        {
            get
            {
                return Cache.Id;
            }
        }

        public long MaxSize 
        {
            get
            {
                return Cache.MaxSize;
            }
            set
            {
                Cache.MaxSize = value;
            }
        }

        public string Name 
        {
            get
            {
                return Cache.Name;
            }
        }

        public long RemainingSpace
        {
            get
            {
                return Cache.RemainingSpace;
            }
        }

        public bool Scavengable 
        {
            get
            {
                return Cache.Scavengable;
            }
            set
            {
                Cache.Scavengable = value;
            }
        }

        public long Size
        {
            get
            {
                return Cache.Size;
            }
        }

        public object SyncRoot
        {
            get
            {
                return Cache.SyncRoot;
            }
        }

        public object this[object key] 
        { 
            get
            {
                return Cache[key];
            } 
        }

        public void Add(ID key, ICacheable data)
        {
            Cache.Add(key, data);
        }

        public void Add(ID key, string value)
        {
            Cache.Add(key, value);
        }

        public void Add(string key, ICacheable data)
        {
            Cache.Add(key, data);
        }

        public void Add(string key, ID value)
        {
            Cache.Add(key, value);
        }

        public Cache.CacheEntry Add(string key, string data)
        {
            return Cache.Add(key, data);
        }

        public void Add(ID key, object data, long dataLength)
        {
            Cache.Add(key, data, dataLength);
        }

        public void Add(object key, object data, long dataLength)
        {
            Cache.Add(key, data, dataLength);
        }

        public void Add(string key, object data, long dataLength)
        {
            Cache.Add(key, data, dataLength);
        }

        public void Add(object key, object data, long dataLength, DateTime absoluteExpiration)
        {
            Cache.Add(key, data, dataLength, absoluteExpiration);
        }

        public void Add(object key, object data, long dataLength, TimeSpan slidingExpiration)
        {
            Cache.Add(key, data, dataLength, slidingExpiration);
        }

        public Cache.CacheEntry Add(string key, object data, long dataLength, DateTime absoluteExpiration)
        {
            return Cache.Add(key, data, dataLength, absoluteExpiration);
        }

        public void Add(string key, object data, long dataLength, EventHandler<EntryRemovedEventArgs> removedHandler)
        {
            Cache.Add(key, data, dataLength, removedHandler);
        }

        public Cache.CacheEntry Add(string key, object data, long dataLength, TimeSpan slidingExpiration)
        {
            return Cache.Add(key, data, dataLength, slidingExpiration);
        }

        public void Add(object key, object data, long dataLength, TimeSpan slidingExpiration, DateTime absoluteExpiration)
        {
            Cache.Add(key, data, dataLength, slidingExpiration, absoluteExpiration);
        }

        public void Clear()
        {
            Cache.Clear();
        }

        public bool ContainsKey(ID key)
        {
            return Cache.ContainsKey(key);
        }

        public bool ContainsKey(object key)
        {
            return Cache.ContainsKey(key);
        }

        public ArrayList GetCacheKeys()
        {
            return Cache.GetCacheKeys();
        }

        public ArrayList GetCacheKeys(string keyPrefix)
        {
            return Cache.GetCacheKeys(keyPrefix);
        }

        public Cache.CacheEntry GetEntry(object key, bool updateAccessed)
        {
            return Cache.GetEntry(key, updateAccessed);
        }
        
        public object GetValue(object key)
        {
            return Cache.GetValue(key);
        }

        public void Remove(object key)
        {
            Cache.Remove(key);
        }

        public void Remove<TKey>(Predicate<TKey> predicate)
        {
            Cache.Remove<TKey>(predicate);
        }

        public void RemoveKeysContaining(string value)
        {
            Cache.RemoveKeysContaining(value);
        }

        public void RemovePrefix(string keyPrefix)
        {
            Cache.RemovePrefix(keyPrefix);
        }

        public void Scavenge()
        {
            Cache.Scavenge();
        }
    }
}

I’m not going to talk about the above interface or class since it just wraps Sitecore.Caching.Cache, and there isn’t much to talk about here.

Next, I spun up the following class that represents an entry in our custom SQL table:

using System;

using Sitecore.Caching;
using Sitecore.Data;
using Sitecore.Reflection;

using Newtonsoft.Json;

namespace Sitecore.Sandbox.Data.Providers.ItemUrls
{
    public class ItemUrlEntry : ICacheable, ICloneable
    {
        public ID ItemID { get; set; }

        public string Site { get; set; }
        
        public string Database { get; set; }
        
        public string Url { get; set; }

        bool cacheable;
        bool ICacheable.Cacheable
        {
            get
            {
                return cacheable;
            }
            set
            {
                cacheable = value;
            }
        }

        bool ICacheable.Immutable
        {
            get
            {
                return true;
            }
        }

        event DataLengthChangedDelegate ICacheable.DataLengthChanged
        {
            add
            {
            }
            remove
            {
            }
        }

        long ICacheable.GetDataLength()
        {
            return TypeUtil.SizeOfID()
                    + TypeUtil.SizeOfString(Site) 
                    + TypeUtil.SizeOfString(Database) 
                    + TypeUtil.SizeOfString(Url);
        }

        public object Clone()
        {
            return new ItemUrlEntry { ItemID = ItemID, Site = Site, Database = Database, Url = Url };
        }

        public override string ToString()
        {
            return JsonConvert.SerializeObject(this);
        }
    }
}

Entries can contain the ID of the Sitecore Item; the specific site we are storing this url for; and the target Database.

You’ll notice I’ve implemented the Sitecore.Caching.ICacheable interface. I’ve done this so I can store entries in cache for performance. I’m not going to go much into the details of how this works since there isn’t much to point out.

I also override the ToString() method for testing purposes. You’ll see this in action later on when we test this together.

Next, we need some sort of provider to manage these entries. I’ve defined the following interface for such a provider:

using System.Collections.Generic;

using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Sites;

namespace Sitecore.Sandbox.Data.Providers.ItemUrls
{
    public interface IItemUrlsProvider
    {
        void AddEntry(ItemUrlEntry entry);
        
        void RemoveEntry(ItemUrlEntry entry);

        Item GetItem(ItemUrlEntry entry);

        ItemUrlEntry GetEntry(ItemUrlEntry entry);

        IEnumerable<ItemUrlEntry> GetAllEntries();
    }
}

IItemUrlsProviders should have the ability to add/remove/retrieve entries. They should also offer the ability to get all entries — I need this for testing later on in this post.

Plus, as a “nice to have”, these providers should return a Sitecore Item for a given entry. Such would be useful when retrieving and setting the context Sitecore Item via a custom Item Resolver (you would typically have an <httpRequestBegin> pipeline processor that does this).

I then created the following class that implements the IItemUrlsProvider interface defined above. This class is specific to adding/removing/retrieving entries from a custom SQL database:

using System;
using System.Collections.Generic;

using Sitecore.Caching;
using Sitecore.Configuration;
using Sitecore.Data;
using Sitecore.Data.DataProviders.Sql;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;

using Sitecore.Sandbox.Caching;

namespace Sitecore.Sandbox.Data.Providers.ItemUrls.SqlServer
{
    public class SqlServerItemUrlsProvider : IItemUrlsProvider
    {
        private SqlDataApi SqlDataApi { get; set; }

        protected ICacheProvider CacheProvider { get; private set; }

        protected string CachePrefix { get; private set; }

        public SqlServerItemUrlsProvider(SqlDataApi sqlDataApi, ICacheProvider cacheProvider, string cachePrefix)
        {
            Assert.ArgumentNotNull(sqlDataApi, "sqlDataApi");
            Assert.ArgumentNotNull(cacheProvider, "cacheProvider");
            Assert.ArgumentNotNullOrEmpty(cachePrefix, "cachePrefix");
            SqlDataApi = sqlDataApi;
            CacheProvider = cacheProvider;
            CachePrefix = cachePrefix;
        }

        public void AddEntry(ItemUrlEntry entry)
        {
            Assert.ArgumentNotNull(entry, "entry");
            Assert.ArgumentCondition(!ID.IsNullOrEmpty(entry.ItemID), "entry.ItemID", "entry.ItemID cannot be null or empty");
            Assert.ArgumentNotNullOrEmpty(entry.Site, "entry.Site");
            Assert.ArgumentNotNullOrEmpty(entry.Database, "entry.Database");
            Assert.ArgumentNotNullOrEmpty(entry.Url, "entry.Url");
            const string addEntrySql = "INSERT INTO {0}ItemUrls{1} ( {0}ItemID{1}, {0}Site{1}, {0}Database{1}, {0}Url{1} ) VALUES ( {2}itemID{3}, {2}site{3}, {2}database{3}, {2}url{3} )";
            var success = Factory.GetRetryer().Execute(() =>
            {
                object[] parameters = new object[] { "itemID", entry.ItemID, "site", entry.Site, "database", entry.Database, "url", entry.Url };
                return SqlDataApi.Execute(addEntrySql, parameters) > 0;
            });

            if (success)
            {
                AddToCache(entry);
            }
        }

        public void RemoveEntry(ItemUrlEntry entry)
        {
            const string deleteEntrySql = "DELETE FROM {0}ItemUrls{1} WHERE {0}Site{1} = {2}site{3} AND {0}Database{1} = {2}database{3} AND {0}Url{1} = {2}url{3}";
            var success = Factory.GetRetryer().Execute(() =>
            {
                object[] parameters = new object[] { "site", entry.Site, "database", entry.Database, "url", entry.Url };
                return SqlDataApi.Execute(deleteEntrySql, parameters) > 0;
            });

            if (success)
            {
                RemoveFromCache(entry);
            }
        }

        public Item GetItem(ItemUrlEntry entry)
        {
            ItemUrlEntry foundEntry = GetEntry(entry);
            if(foundEntry == null)
            {
                return null;
            }

            Database database = Factory.GetDatabase(foundEntry.Database);
            if(database == null)
            {
                return null;
            }

            try
            {
                return database.Items[foundEntry.ItemID];
            }
            catch(Exception ex)
            {
                Log.Error(ToString(), ex, this);
            }

            return null;
        }

        public ItemUrlEntry GetEntry(ItemUrlEntry entry)
        {
            ItemUrlEntry foundEntry = GetFromCache(entry);
            if (foundEntry != null)
            {
                return foundEntry;
            }

            const string getEntrySql = "SELECT {0}ItemID{1} FROM {0}ItemUrls{1} WHERE {2}Site = {2}site{3} AND {2}Database{3} = {2}database{3} AND {0}Url{1} = {2}url{3}";
            object[] parameters = new object[] { "site", entry.Site, "database", entry.Database, "url", entry.Url };
            using (DataProviderReader reader = SqlDataApi.CreateReader(getEntrySql, parameters))
            {
                if (!reader.Read())
                {
                    return null;
                }

                ID itemID = ID.Parse(SqlDataApi.GetGuid(0, reader));
                if (ID.IsNullOrEmpty(itemID))
                {
                    return null;
                }

                foundEntry = entry.Clone() as ItemUrlEntry;
                foundEntry.ItemID = itemID;
                AddToCache(entry);
                return foundEntry;
            }
        }

        public IEnumerable<ItemUrlEntry> GetAllEntries()
        {
            const string getAllEntriesSql = "SELECT {0}ItemID{1}, {0}Site{1}, {0}Database{1}, {0}Url{1} FROM {0}ItemUrls{1}";
            IList<ItemUrlEntry> entries = new List<ItemUrlEntry>();
            using (DataProviderReader reader = SqlDataApi.CreateReader(getAllEntriesSql, new object[0]))
            {
                while(reader.Read())
                {
                    ID itemID = ID.Parse(SqlDataApi.GetGuid(0, reader));
                    if (!ID.IsNullOrEmpty(itemID))
                    {
                        entries.Add
                        (
                            new ItemUrlEntry 
                            {
                                ItemID = itemID, 
                                Site = SqlDataApi.GetString(1, reader), 
                                Database = SqlDataApi.GetString(2, reader), 
                                Url =  SqlDataApi.GetString(3, reader)
                            }
                        );
                    } 
                }
            }

            return entries;
        }

        protected virtual void AddToCache(ItemUrlEntry entry)
        {
            CacheProvider.Add(GetCacheKey(entry), entry);
        }

        protected virtual void RemoveFromCache(ItemUrlEntry entry)
        {
            CacheProvider.Remove(GetCacheKey(entry));
        }

        protected virtual ItemUrlEntry GetFromCache(ItemUrlEntry entry)
        {
            return CacheProvider[GetCacheKey(entry)] as ItemUrlEntry;
        }

        protected virtual string GetCacheKey(ItemUrlEntry entry)
        {
            Assert.ArgumentNotNull(entry, "entry");
            Assert.ArgumentNotNull(entry.Site, "entry.Site");
            Assert.ArgumentNotNull(entry.Database, "entry.Database");
            Assert.ArgumentNotNull(entry.Url, "entry.Url");
            return string.Join("#", CachePrefix, entry.Site, entry.Database, entry.Url);
        }
    }
}

Sitecore.Data.DataProviders.Sql.SqlDataApi and ICacheProvider instances along with a cache prefix are injected into the class instance’s constructor using the Sitecore Configuration Factory (you’ll get a better idea of how this happens when you have a look at the patch configuration file towards the bottom of this post). These are saved to properties on the class instance so they can be leveraged by the methods on the class.

One thing I would like to point out is the Sitecore.Data.DataProviders.Sql.SqlDataApi class is an abstraction — it’s an abstract class that is subclassed by Sitecore.Data.SqlServer.SqlServerDataApi in Sitecore.Kernel.dll. This concrete class does most of the leg work on talking to the SQL Server database, and we just utilize methods on it for adding/deleting/removing entries.

The AddEntry() method delegates the database saving operation to the Sitecore.Data.DataProviders.Sql.SqlDataApi instance, and then uses the ICacheProvider instance for storing the entry in cache.

The RemoveEntry() method also leverages the Sitecore.Data.DataProviders.Sql.SqlDataApi instance for deleting the entry from the database, and then removes the entry from cache via the ICacheProvider instance.

The GetEntry() method does exactly what you think it does. It tries to get the entry first from cache via the ICacheProvider instance and then the database via the Sitecore.Data.DataProviders.Sql.SqlDataApi instance if the entry was not found in cache. If the Item was not in cache but was in the database, the GetEntry() method then saves the entry to cache.

I then created the following Singleton for testing:

using System;
using System.Collections.Generic;

using Sitecore.Configuration;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;

namespace Sitecore.Sandbox.Data.Providers.ItemUrls
{
    public class ItemUrlsProvider : IItemUrlsProvider
    {
        private static readonly Lazy<IItemUrlsProvider> lazyInstance = new Lazy<IItemUrlsProvider>(() => new ItemUrlsProvider());

        public static IItemUrlsProvider Current { get { return lazyInstance.Value; } }

        private IItemUrlsProvider InnerProvider { get; set; }

        private ItemUrlsProvider()
        {
            InnerProvider = GetInnerProvider();
        }

        public void AddEntry(ItemUrlEntry entry)
        {
            InnerProvider.AddEntry(entry);
        }

        public void RemoveEntry(ItemUrlEntry entry)
        {
            InnerProvider.RemoveEntry(entry);
        }

        public Item GetItem(ItemUrlEntry entry)
        {
            return InnerProvider.GetItem(entry);
        }

        public ItemUrlEntry GetEntry(ItemUrlEntry entry)
        {
            return InnerProvider.GetEntry(entry);
        }

        public IEnumerable<ItemUrlEntry> GetAllEntries()
        {
            return InnerProvider.GetAllEntries();
        }

        protected virtual IItemUrlsProvider GetInnerProvider()
        {
            IItemUrlsProvider provider = Factory.CreateObject("itemUrlsProvider", true) as IItemUrlsProvider;
            Assert.IsNotNull(provider, "itemUrlsProvider must be set in configuration!");
            return provider;
        }
    }
}

The Singleton above basically decorates the IItemUrlsProvider instance defined in Sitecore configuration — see the configuration file below — and delegates method calls to it.

I then wired everything together using the following patch configuration file:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <itemUrlsProvider id="custom" type="Sitecore.Sandbox.Data.Providers.ItemUrls.$(database).$(database)ItemUrlsProvider, Sitecore.Sandbox" singleInstance="true">
      <param type="Sitecore.Data.$(database).$(database)DataApi, Sitecore.Kernel" desc="sqlDataApi">
        <param connectionStringName="$(id)"/>
      </param>
      <param type="Sitecore.Sandbox.Caching.CacheProvider, Sitecore.Sandbox" desc="cacheProvider">
        <param desc="cacheName">[ItemUrls]</param>
        <param desc="cacheSize">500KB</param>
      </param>
      <param desc="cachePrefix">ItemUrlsEntry</param>
    </itemUrlsProvider>
  </sitecore>
</configuration>

For testing, I whipped up a standalone ASP.NET Web Form (yes, there are more elegant ways to do this but it’s Sunday so cut me some slack 😉 ):

using System;
using System.Text;
using System.Collections.Generic;
using System.Linq;

using Sitecore.Data.Items;

using Sitecore.Sandbox.Data.Providers.ItemUrls;

namespace Sitecore.Sandbox.Web.tests
{
    public partial class ItemUrlsProviderTest : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            IItemUrlsProvider provider = ItemUrlsProvider.Current;
            Item home = Sitecore.Context.Database.GetItem("/sitecore/content/home");
            StringBuilder output = new StringBuilder();
            
            ItemUrlEntry firstEntry = new ItemUrlEntry { ItemID = home.ID, Site = Sitecore.Context.Site.Name, Database = Sitecore.Context.Database.Name, Url = "/this/does/not/exist" };
            output.AppendFormat("Adding {0} as an entry.<br />", firstEntry);
            provider.AddEntry(firstEntry);

            ItemUrlEntry secondEntry = new ItemUrlEntry { ItemID = home.ID, Site = Sitecore.Context.Site.Name, Database = Sitecore.Context.Database.Name, Url = "/fake/url" };
            output.AppendFormat("Adding {0} as an entry.<br />", secondEntry);
            provider.AddEntry(secondEntry);

            ItemUrlEntry thirdEntry = new ItemUrlEntry { ItemID = home.ID, Site = Sitecore.Context.Site.Name, Database = Sitecore.Context.Database.Name, Url = "/another/fake/url" };
            output.AppendFormat("Adding {0} as an entry.<hr />", thirdEntry);
            provider.AddEntry(thirdEntry);

            ItemUrlEntry fourthEntry = new ItemUrlEntry { ItemID = home.ID, Site = Sitecore.Context.Site.Name, Database = Sitecore.Context.Database.Name, Url = "/blah/blah/blah" };
            output.AppendFormat("Adding {0} as an entry.<hr />", fourthEntry);
            provider.AddEntry(fourthEntry);

            ItemUrlEntry fifthEntry = new ItemUrlEntry { ItemID = home.ID, Site = Sitecore.Context.Site.Name, Database = Sitecore.Context.Database.Name, Url = "/i/am/a/url" };
            output.AppendFormat("Adding {0} as an entry.<hr />", fifthEntry);
            provider.AddEntry(fifthEntry);

            output.AppendFormat("Current saved entries:<br /><br />{0}<hr />", string.Join("<br />", provider.GetAllEntries().Select(entry => entry.ToString())));

            output.AppendFormat("Removing entry {0}.<br /><br />", firstEntry.ToString());
            provider.RemoveEntry(firstEntry);

            output.AppendFormat("Current saved entries:<br /><br />{0}", string.Join("<br />", provider.GetAllEntries().Select(entry => entry.ToString())));
            litResults.Text = output.ToString();
        }
    }
}

The test above adds five entries, and then deletes one. It also outputs what’s in the database after specific operations.

After doing a build, I pulled up the above Web Form in my browser and saw this once the page was done rendering:

ItemUrlsProviderTest

If you have any thoughts on this, please share in a comment.

Until next time, have a Sitecoredatalicious day!

Embedded Tweets in Sitecore: A Proof of Concept

In a previous post, I showcased a “proof of concept” for shortcodes in Sitecore — this is a shorthand notation for embedding things like YouTube videos in your webpages without having to type up a bunch of HTML — and felt I should follow up with another “proof of concept” around incorporating Embedded Tweets in Sitecore.

You might be asking “what’s an Embedded Tweet?” An Embedded Tweet is basically the process of pasting a Tweet URL from Twitter into an editable content area of your website/blog/whatever (think Rich Text field in Sitecore), and let the code that builds the HTML for your site figure out how to display it.

For example, I had used an Embedded Tweet in a recent post:

tweet-url-wordpress

This is what is seen on the rendered page:

tweet-embedded

While doing some research via Google on how to do this in Sitecore, I found this page from Twitter that discusses how you could go about accomplishing this, and discovered how to get JSON containing information about a Tweet — including its HTML — using one of Twitter’s API URLs:

tweet-api-json

The JSON above drove me to build the following POCO class to represent data returned by that URL:

using System.Runtime.Serialization;

namespace Sitecore.Sandbox.Pipelines.RenderField.Tweets
{
    public class Tweet
    {
        [DataMember(Name = "cache_age")]
        public int CacheAgeMilliseconds { get; set; }

        [DataMember(Name = "url")]
        public string Url { get; set; }

        [DataMember(Name = "html")]
        public string Html { get; set; }
    }
}

I decided to omit some of the JSON properties returned by the Twitter URL from my class above — width and height are examples — since I felt I did not need to use them for this “proof of concept”.

I then leveraged the class above in the following class that will serve as a <renderField> pipeline processor to embed Tweets:

using System;
using System.IO;
using System.Net;
using System.Text.RegularExpressions;
using System.Web;

using Sitecore.Caching;
using Sitecore.Diagnostics;
using Sitecore.Pipelines.RenderField;

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace Sitecore.Sandbox.Pipelines.RenderField.Tweets
{
    public class ExpandTweets
    {
        private string TwitterWidgetScriptTag {get ; set; }

        private string TwitterApiUrlFormat { get; set; }

        private string _TweetPattern;
        private string TweetPattern 
        {
            get
            {
                return _TweetPattern;
            }
            set
            {
                _TweetPattern = value;
                if (!string.IsNullOrWhiteSpace(_TweetPattern))
                {
                    _TweetPattern = HttpUtility.HtmlDecode(_TweetPattern);
                }
            }
        }

        private HtmlCache _HtmlCache;
        private HtmlCache HtmlCache
        {
            get
            {
                if (_HtmlCache == null)
                {
                    _HtmlCache = CacheManager.GetHtmlCache(Context.Site);
                }

                return _HtmlCache;
            }
        }

        public void Process(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull(args, "args");
            AssertRequired();
            if(!ShouldFieldBeProcessed(args))
            {
                return;
            }

            args.Result.FirstPart = ExpandTweetUrls(args.Result.FirstPart);
        }

        private static bool ShouldFieldBeProcessed(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull(args, "args");
            Assert.ArgumentNotNull(args.FieldTypeKey, "args.FieldTypeKey");
            string fieldTypeKey = args.FieldTypeKey.ToLower();
            return fieldTypeKey == "text"
                    || fieldTypeKey == "rich text"
                    || fieldTypeKey == "single-line text"
                    || fieldTypeKey == "multi-line text";
        }

        private void AssertRequired()
        {
            Assert.IsNotNullOrEmpty(TwitterWidgetScriptTag, "TwitterWidgetScriptTag must be set! Check your configuration!");
            Assert.IsNotNullOrEmpty(TwitterApiUrlFormat, "TwitterApiUrlFormat must be set! Check your configuration!");
            Assert.IsNotNullOrEmpty(TweetPattern, "TweetPattern must be set! Check your configuration!");
        }

        protected virtual string ExpandTweetUrls(string html)
        {
            string htmlExpanded = html;
            MatchCollection matches = Regex.Matches(htmlExpanded, TweetPattern, RegexOptions.Compiled | RegexOptions.IgnoreCase);
            foreach (Match match in matches)
            {
                string tweetHtml = GetTweetHtml(match.Groups["id"].Value);
                if (!string.IsNullOrWhiteSpace(tweetHtml))
                {
                    htmlExpanded = htmlExpanded.Replace(match.Value, tweetHtml);
                }
            }

            if (matches.Count > 0)
            {
                htmlExpanded = string.Concat(htmlExpanded, TwitterWidgetScriptTag);
            }

            return htmlExpanded;
        }

        protected virtual string GetTweetHtml(string id)
        {
            string html = GetTweetHtmlFromCache(id);
            if (!string.IsNullOrWhiteSpace(html))
            {
                return html;
            }

            Tweet tweet = GetTweetFromApi(id);
            AddTweetHtmlToCache(id, tweet);
            return tweet.Html;
        }

        private string GetTweetHtmlFromCache(string id)
        {
            return HtmlCache.GetHtml(id);
        }

        private void AddTweetHtmlToCache(string id, Tweet tweet)
        {
            if (string.IsNullOrWhiteSpace(tweet.Html))
            {
                return;
            }

            if (tweet.CacheAgeMilliseconds > 0)
            {
                HtmlCache.SetHtml(id, tweet.Html, DateTime.Now.AddMilliseconds(tweet.CacheAgeMilliseconds));
                return;
            }

            HtmlCache.SetHtml(id, tweet.Html);
        }

        protected virtual Tweet GetTweetFromApi(string id)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(string.Format(TwitterApiUrlFormat, id));
            try
            {
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                using (StreamReader reader = new StreamReader(response.GetResponseStream()))
                {
                    var result = reader.ReadToEnd();
                    JObject jObject = JObject.Parse(result);
                    return JsonConvert.DeserializeObject<Tweet>(jObject.ToString());
                }
            }
            catch (Exception ex)
            {
                Log.Error(this.ToString(), ex, this);
            }

            return new Tweet { Html = string.Empty };
        }
    }
}

Methods in the class above find all Tweet URLs in the Rich Text, Single-Line Text, or Multi-Line Text field being processed — the code determines if it’s a Tweet URL based on a pattern that is supplied by a configuration setting (you will see this below in this post); extract Tweets’ Twitter identifiers (these are located at the end of the Tweet URLs); and attempt to find the Tweets’ HTML in Sitecore’s HTML cache.

If the HTML is found in cache for a Tweet, we return it. Otherwise, we make a request to Twitter’s API to get it, put it in cache one we have it (it is set to expire after a specified number of milliseconds from the time it was retrieved: Twitter returns the number of milliseconds in one full year by default), and then we return it.

If the returned HTML is not empty, we replace it in the field’s value for display.

If the HTML returned is empty — this could happen when an exception is encountered during the Twitter API call (of course we log the exception in the Sitecore log when this happens 😉 ) — we don’t touch the Tweet URL in the field’s value.

Once all Tweet URLs have been processed, we append a script tag referencing Twitter’s widget.js file — this is supplied through a configuration setting, and it does the heavy lifting on making the Tweet HTML look Twitterific 😉 — to the field’s rendered HTML.

I then tied everything together using the following patch configuration file:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <renderField>
        <processor type="Sitecore.Sandbox.Pipelines.RenderField.Tweets.ExpandTweets, Sitecore.Sandbox"
					patch:after="processor[@type='Sitecore.Pipelines.RenderField.GetTextFieldValue, Sitecore.Kernel']">
          <TwitterWidgetScriptTag>&lt;script async src="//platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt;</TwitterWidgetScriptTag>
          <TwitterApiUrlFormat>https://api.twitter.com/1/statuses/oembed.json?id={0}&amp;omit_script=true</TwitterApiUrlFormat>
          <TweetPattern>https://twitter.com/.+/status/(?&lt;id&gt;\d*)</TweetPattern>
        </processor>
      </renderField>
    </pipelines>
  </sitecore>
</configuration>

Let’s see this in action!

I created a test Item, and added some legitimate and bogus Tweet URLs into one of its Rich Text fields (please pardon the grammatical issues in the following screenshots :-/):

tweets-rich-text

This isn’t the most aesthetically pleasing HTML, but it will serve its purpose for testing:

tweets-rich-text-html

After saving and publishing, I navigated to my test Item’s page, and saw this:

tweets-front-end

If you have any suggestions on making this better, or have other ideas for embedding Tweets in Sitecore, please share in a comment.

Honey, I Shrunk the Content: Experiments with a Custom Sitecore Cache and Compression

A few days ago, I pondered whether there would be any utility in creating a custom Sitecore cache that compresses data before it’s stored and decompresses data upon retrieval. I wondered whether having such a cache would facilitate in conserving memory resources, thus curtailing the need to beef up servers from a memory perspective.

You’re probably thinking “Mike, who cares? Memory is cheaper today than ever before!” That thought is definitely valid.

However, I would argue we owe it to our clients and to ourselves as developers to push the envelope as much as possible by architecting our solutions to be as efficient and resource conscious as possible.

Plus, I was curious over how expensive compress/decompress operations would be for real-time requests.

The following code showcases my ventures into trying to answer these questions.

First, I defined an interface for compressors. Compressors must define methods to compress and decompress data (duh :)). I also added an additional Decompress method to cast the data object to a particular type — all for the purpose of saving client code the trouble of having to do their own type casting (yeah right, I really did it to be fancy).

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors.Base
{
    public interface ICompressor
    {
        string Name { get; }

        byte[] Compress(object uncompressedData);

        T Decompress<T>(byte[] compressedData) where T : class;

        object Decompress(byte[] compressedData);

        long GetDataSize(object data);
    }
}

I decided to use two compression algorithms available in the System.IO.Compression namespace — Deflate and GZip. Since both algorithms within this namespace implement System.IO.Stream, I found an opportunity to use the Template method pattern.

In the spirit of this design pattern, I created an abstract class — see the CompressionStreamCompressor class below — which contains shared logic for compressing/decompressing data using methods defined by the Stream class. Subclasses only have to “fill in the blanks” by implementing the abstract method CreateNewCompressionStream — a method that returns a new instance of the compression stream represented by the subclass.

CompressionStreamCompressor:

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;

using Sitecore.Diagnostics;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors.Base
{
    public abstract class CompressionStreamCompressor : ICompressor
    {
        protected CompressionStreamCompressor()
        {
        }

        public virtual byte[] Compress(object uncompressedData)
        {
            Assert.ArgumentNotNull(uncompressedData, "uncompressedData");
            
            byte[] uncompressedBytes = ConvertObjectToBytes(uncompressedData);
            return Compress(uncompressedBytes);
        }

        private byte[] Compress(byte[] uncompressedData)
        {
            Assert.ArgumentNotNull(uncompressedData, "uncompressedData");

            using (MemoryStream memoryStream = new MemoryStream())
            {
                using (Stream compressionStream = CreateNewCompressionStream(memoryStream, CompressionMode.Compress))
                {
                    compressionStream.Write(uncompressedData, 0, uncompressedData.Length);
                }

                return memoryStream.ToArray();
            }
        }

        public virtual T Decompress<T>(byte[] compressedData) where T : class
        {
            object decompressedData = Decompress(compressedData);
            return decompressedData as T;
        }

        public virtual object Decompress(byte[] compressedBytes)
        {
            Assert.ArgumentNotNull(compressedBytes, "compressedBytes");

            using (MemoryStream inputMemoryStream = new MemoryStream(compressedBytes))
            {
                using (Stream compressionStream = CreateNewCompressionStream(inputMemoryStream, CompressionMode.Decompress))
                {
                    using (MemoryStream outputMemoryStream = new MemoryStream())
                    {
                        compressionStream.CopyTo(outputMemoryStream);
                        return ConvertBytesToObject(outputMemoryStream.ToArray());
                    }
                }
            }
        }

        protected abstract Stream CreateNewCompressionStream(Stream stream, CompressionMode compressionMode);

        public long GetDataSize(object data)
        {
            if (data == null)
                return 0;

            IFormatter formatter = new BinaryFormatter();
            long size = 0;

            using (MemoryStream memoryStream = new MemoryStream())
            {
                formatter.Serialize(memoryStream, data);
                size = memoryStream.Length;
            }

            return size;
        }

        protected static byte[] ConvertObjectToBytes(object data)
        {
            if (data == null)
                return null;

            byte[] bytes = null;
            IFormatter formatter = new BinaryFormatter();

            using (MemoryStream memoryStream = new MemoryStream())
            {
                formatter.Serialize(memoryStream, data);
                bytes = memoryStream.ToArray();
            }
            
            return bytes;
        }

        protected static object ConvertBytesToObject(byte[] bytes)
        {
            if (bytes == null)
                return null;

            object deserialized = null;
            
            using (MemoryStream memoryStream = new MemoryStream(bytes))
            {
                IFormatter formatter = new BinaryFormatter();
                memoryStream.Position = 0;
                deserialized = formatter.Deserialize(memoryStream);
            }

            return deserialized;
        }
    }
}

DeflateCompressor:

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;

using Sitecore.Diagnostics;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Compression
{
    public class DeflateCompressor : CompressionStreamCompressor
    {
        private DeflateCompressor()
        {
        }

        protected override Stream CreateNewCompressionStream(Stream stream, CompressionMode compressionMode)
        {
            return new DeflateStream(stream, compressionMode, false);
        }

        public static ICompressor CreateNewCompressor()
        {
            return new DeflateCompressor();
        }
    }
}

GZipCompressor:

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;

using Sitecore.Diagnostics;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Compression
{
    public class GZipCompressor : CompressionStreamCompressor
    {
        private GZipCompressor()
        {
        }

        protected override Stream CreateNewCompressionStream(Stream stream, CompressionMode compressionMode)
        {
            return new GZipStream(stream, compressionMode, false);
        }

        public static ICompressor CreateNewCompressor()
        {
            return new GZipCompressor();
        }
    }
}

If Microsoft ever decides to augment their arsenal of compression streams in System.IO.Compression, we could easily add new Compressor classes for these via the template method paradigm above — as long as these new compression streams implement System.IO.Stream.

After implementing the classes above, I decided I needed a “dummy” Compressor — a compressor that does not execute any compression algorithm but implements the ICompressor interface. My reasoning for doing so is to have a default Compressor be returned via a compressor factory (you will see that I created one further down), and also for ascertaining baseline benchmarks.

Plus, I figured it would be nice to have an object that closely follows the Null Object pattern — albeit in our case, we aren’t truly using this design pattern since our “Null” class is actually executing logic — so client code can avoid having null checks all over the place.

I had to go back and change my Compress and Decompress methods to be virtual in my abstract class so that I can override them within my “Null” Compressor class. The methods just take in the expected parameters and return expected types with no compression or decompression actions in the mix.

using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;

using Sitecore.Diagnostics;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors
{
    class NullCompressor : CompressionStreamCompressor
    {
        private NullCompressor()
        {
        }

        public override byte[] Compress(object uncompressedData)
        {
            Assert.ArgumentNotNull(uncompressedData, "uncompressedData");
            return ConvertObjectToBytes(uncompressedData);
        }

        public override object Decompress(byte[] compressedBytes)
        {
            Assert.ArgumentNotNull(compressedBytes, "compressedBytes");
            return ConvertBytesToObject(compressedBytes);
        }

        protected override Stream CreateNewCompressionStream(Stream stream, CompressionMode compressionMode)
        {
            return null;
        }

        public static ICompressor CreateNewCompressor()
        {
            return new NullCompressor();
        }
    }
}

Next, I defined my custom Sitecore cache with its interface and settings Data transfer object.

An extremely important thing to keep in mind when creating a custom Sitecore cache is knowing you must subclass CustomCache in Sitecore.Caching — most methods that add or get from cache are protected methods, and you won’t have access to these unless you subclass this abstract class (I wasn’t paying attention when I built my cache for the first time, and had to go back to the drawing board when i discovered my code would not compile due to restricted access rights).

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Sitecore.Caching;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Caching.DTO
{
    public class SquashedCacheSettings
    {
        public string Name { get; set; }
        public long MaxSize { get; set; }
        public ICompressor Compressor { get; set; }
        public bool CompressionEnabled { get; set; }
    }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Caching.Base
{
    public interface ISquashedCache
    {
        ICompressor Compressor { get; }

        void AddToCache(object key, object value);

        T GetFromCache<T>(object key) where T : class;

        object GetFromCache(object key);
    }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Sitecore.Caching;
using Sitecore.Data;
using Sitecore.Diagnostics;

using Sitecore.Sandbox.Utilities.Caching.Base;
using Sitecore.Sandbox.Utilities.Caching.DTO;
using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;

namespace Sitecore.Sandbox.Utilities.Caching
{
    public class SquashedCache : CustomCache, ISquashedCache
    {
        private SquashedCacheSettings SquashedCacheSettings { get; set; }

        public ICompressor Compressor
        {
            get
            {
                return SquashedCacheSettings.Compressor;
            }
        }

        private SquashedCache(SquashedCacheSettings squashedCacheSettings)
            : base(squashedCacheSettings.Name, squashedCacheSettings.MaxSize)
        {
            SetSquashedCacheSettings(squashedCacheSettings);
        }

        private void SetSquashedCacheSettings(SquashedCacheSettings squashedCacheSettings)
        {
            SquashedCacheSettings = squashedCacheSettings;
        }

        public void AddToCache(object key, object value)
        {
            DataInformation dataInformation = GetCompressedDataInformation(value);
            SetObject(key, dataInformation.Data, dataInformation.Size);
        }

        private DataInformation GetCompressedDataInformation(object data)
        {
            long size = SquashedCacheSettings.Compressor.GetDataSize(data);
            return GetCompressedDataInformation(data, size);
        }

        private DataInformation GetCompressedDataInformation(object data, long size)
        {
            if (SquashedCacheSettings.CompressionEnabled)
            {
                data = SquashedCacheSettings.Compressor.Compress(data);
                size = SquashedCacheSettings.Compressor.GetDataSize(data);
            }

            return new DataInformation(data, size);
        }

        public T GetFromCache<T>(object key) where T : class
        {
            object value = GetFromCache(key);
            return value as T;
        }

        public object GetFromCache(object key)
        {
            byte[] value = (byte[])GetObject(key);
            return SquashedCacheSettings.Compressor.Decompress(value);
        }

        private T GetDecompressedData<T>(byte[] data) where T : class
        {
            if (SquashedCacheSettings.CompressionEnabled)
            {
                return SquashedCacheSettings.Compressor.Decompress<T>(data);
            }

            return data as T;
        }

        private object GetDecompressedData(byte[] data)
        {
            if (SquashedCacheSettings.CompressionEnabled)
            {
                return SquashedCacheSettings.Compressor.Decompress(data);
            }

            return data;
        }

        private struct DataInformation
        {
            public object Data;
            public long Size;

            public DataInformation(object data, long size)
            {
                Data = data;
                Size = size;
            }
        }

        public static ISquashedCache CreateNewSquashedCache(SquashedCacheSettings squashedCacheSettings)
        {
            AssertSquashedCacheSettings(squashedCacheSettings);
            return new SquashedCache(squashedCacheSettings);
        }

        private static void AssertSquashedCacheSettings(SquashedCacheSettings squashedCacheSettings)
        {
            Assert.ArgumentNotNull(squashedCacheSettings, "squashedCacheSettings");
            Assert.ArgumentNotNullOrEmpty(squashedCacheSettings.Name, "squashedCacheSettings.Name");
            Assert.ArgumentCondition(squashedCacheSettings.MaxSize > 0, "squashedCacheSettings.MaxSize", "MaxSize must be greater than zero.");
            Assert.ArgumentNotNull(squashedCacheSettings.Compressor, "squashedCacheSettings.Compressor");
        }
    }
}

You’re probably thinking “Mike, what’s up with the name SquashedCache”? Well, truth be told, I was thinking about Thanksgiving here in the United States — it’s just around the corner — and how squash is a usually found on the table for Thanksgiving dinner. The name SquashedCache just fit in perfectly in the spirit of our Thanksgiving holiday.

However, the following class names were considered.

public class SirSquishALot
{
}

public class MiniMeCache
{
}

// this became part of this post’s title instead 🙂
public class HoneyIShrunkTheContent
{
}

I figured having a Factory class for compressor objects would offer a clean and central place for creating them.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors.Enums
{
    public enum CompressorType
    {
        Deflate,
        GZip,
        Null
    } 
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Enums;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors.Base
{
    public interface ICompressorFactory
    {
        ICompressor CreateNewCompressor(CompressorType compressorType);
    }
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;
using Sitecore.Sandbox.Utilities.Compression.Compressors.Enums;

namespace Sitecore.Sandbox.Utilities.Compression.Compressors
{
    public class CompressorFactory : ICompressorFactory
    {
        private CompressorFactory()
        {
        }

        public ICompressor CreateNewCompressor(CompressorType compressorType)
        {
            if (compressorType == CompressorType.Deflate)
            {
                return DeflateCompressor.CreateNewCompressor();
            }
            else if (compressorType == CompressorType.GZip)
            {
                return GZipCompressor.CreateNewCompressor();
            }

            return NullCompressor.CreateNewCompressor();
        }

        public static ICompressorFactory CreateNewCompressorFactory()
        {
            return new CompressorFactory();
        }
    }
}

Now, it’s time to test everything above and look at some statistics. I basically created a sublayout containing some repeaters to highlight how each compressor performed against the others — including the “Null” compressor which serves as the baseline.

<%@ Control Language="C#" AutoEventWireup="true" CodeBehind="Squash Test.ascx.cs" Inherits="Sitecore650rev120706.layouts.sublayouts.Squash_Test" %>

<div>
    <h2><u>Compression Test</u></h2>
    Uncompressed Size: <asp:Literal ID="litUncompressedSize" runat="server" /> bytes
    <asp:Repeater ID="rptCompressionTest" runat="server">
        <HeaderTemplate>
            <br /><br />
        </HeaderTemplate>
        <ItemTemplate>
                <div>
                    Compressed Size using <%# Eval("TestName")%>: <%# Eval("CompressedSize")%> bytes <br />
                    Compression Ratio using <%# Eval("TestName")%>: <%# Eval("CompressionRatio","{0:p}") %> of original size
                </div>
        </ItemTemplate>
        <SeparatorTemplate>
            <br />
        </SeparatorTemplate>
    </asp:Repeater>
</div>

<asp:Repeater ID="rptAddToCacheTest" runat="server">
    <HeaderTemplate>
        <div>
            <h2><u>AddToCache() Test</u></h2>
    </HeaderTemplate>
    <ItemTemplate>
            <div>
                AddToCache() Elasped Time for <%# Eval("TestName")%>: <%# Eval("ElapsedMilliseconds")%> ms
            </div>
    </ItemTemplate>
    <FooterTemplate>
        </div>
    </FooterTemplate>
</asp:Repeater>

<asp:Repeater ID="rptGetFromCacheTest" runat="server">
    <HeaderTemplate>
        <div>
            <h2><u>GetFromCache() Test</u></h2>
    </HeaderTemplate>
    <ItemTemplate>
            <div>
                GetFromCache() Elasped Time for <%# Eval("TestName")%>: <%# Eval("ElapsedMilliseconds")%> ms
            </div>
    </ItemTemplate>
    <FooterTemplate>
        </div>
    </FooterTemplate>
</asp:Repeater>

<asp:Repeater ID="rptDataIntegrityTest" runat="server">
    <HeaderTemplate>
        <div>
            <h2><u>Data Integrity Test</u></h2>
    </HeaderTemplate>
    <ItemTemplate>
            <div>
                Data Retrieved From <%# Eval("TestName")%> Equals Original: <%# Eval("AreEqual")%>
            </div>
    </ItemTemplate>
    <FooterTemplate>
        </div>
    </FooterTemplate>
</asp:Repeater>

In my code-behind, I’m grabbing the full text of the book War and Peace by Leo Tolstoy for testing purposes. The full text of this copy of the book is over 3.25 MB.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;

using Sitecore;

using Sitecore.Sandbox.Utilities.Caching;
using Sitecore.Sandbox.Utilities.Caching.Base;
using Sitecore.Sandbox.Utilities.Caching.DTO;

using Sitecore.Sandbox.Utilities.Compression.Compressors;
using Sitecore.Sandbox.Utilities.Compression.Compressors.Base;
using Sitecore.Sandbox.Utilities.Compression.Compressors.Enums;

namespace Sitecore650rev120706.layouts.sublayouts
{
    public partial class Squash_Test : System.Web.UI.UserControl
    {
        private const string CacheKey = "War and Peace";
        private const string WarAndPeaceUrl = "http://www.gutenberg.org/cache/epub/2600/pg2600.txt";
        private const string MaxSize = "50MB";

        private ICompressorFactory _Factory;
        private ICompressorFactory Factory
        {
            get
            {
                if(_Factory == null)
                    _Factory = CompressorFactory.CreateNewCompressorFactory();

                return _Factory;
            }
        }

        private IEnumerable<ISquashedCache> _SquashedCaches;
        private IEnumerable<ISquashedCache> SquashedCaches
        {
            get
            {
                if(_SquashedCaches == null)
                    _SquashedCaches = CreateAllSquashedCaches();

                return _SquashedCaches;
            }
        }

        private string _WarAndPeaceText;
        private string WarAndPeaceText
        {
            get
            {
                if (string.IsNullOrEmpty(_WarAndPeaceText))
                    _WarAndPeaceText = GetWarAndPeaceText();

                return _WarAndPeaceText;
            }
        }

        private long _UncompressedSize;
        private long UncompressedSize
        {
            get
            {
                if(_UncompressedSize == 0)
                    _UncompressedSize = GetUncompressedSize();

                return _UncompressedSize;
            }
        }


        private Stopwatch _Stopwatch;
        private Stopwatch Stopwatch
        {
            get
            {
                if (_Stopwatch == null)
                    _Stopwatch = Stopwatch.StartNew();

                return _Stopwatch;
            }
        }

        protected void Page_Load(object sender, EventArgs e)
        {
            SetUncomopressedSizeLiteral();
            BindAllRepeaters();
        }

        private void SetUncomopressedSizeLiteral()
        {
            litUncompressedSize.Text = UncompressedSize.ToString();
        }

        private void BindAllRepeaters()
        {
            BindCompressionTestRepeater();
            BindAddToCacheTestRepeater();
            BindGetFromCacheTestRepeater();
            BindDataIntegrityTestRepeater();
        }

        private void BindCompressionTestRepeater()
        {
            rptCompressionTest.DataSource = GetCompressionTestData();
            rptCompressionTest.DataBind();
        }

        private IEnumerable<CompressionRatioAtom> GetCompressionTestData()
        {
            IList<CompressionRatioAtom> compressionRatioAtoms = new List<CompressionRatioAtom>();

            foreach (ISquashedCache squashedCache in SquashedCaches)
            {
                byte[] compressed = squashedCache.Compressor.Compress(WarAndPeaceText);
                long compressedSize = squashedCache.Compressor.GetDataSize(compressed);

                CompressionRatioAtom compressionRatioAtom = new CompressionRatioAtom
                {
                    TestName = squashedCache.Name,
                    CompressedSize = compressedSize,
                    CompressionRatio = ((decimal)compressedSize / UncompressedSize)
                };

                compressionRatioAtoms.Add(compressionRatioAtom);
            }

            return compressionRatioAtoms;
        }

        private void BindAddToCacheTestRepeater()
        {
            rptAddToCacheTest.DataSource = GetAddToCacheTestData();
            rptAddToCacheTest.DataBind();
        }

        private IEnumerable<TimeTestAtom> GetAddToCacheTestData()
        {
            IList<TimeTestAtom> timeTestAtoms = new List<TimeTestAtom>();

            foreach (ISquashedCache squashedCache in SquashedCaches)
            {
                Stopwatch.Start();
                squashedCache.AddToCache(CacheKey, WarAndPeaceText);
                Stopwatch.Stop();

                TimeTestAtom timeTestAtom = new TimeTestAtom
                {
                    TestName = squashedCache.Name,
                    ElapsedMilliseconds = Stopwatch.Elapsed.TotalMilliseconds
                };

                timeTestAtoms.Add(timeTestAtom);
            }

            return timeTestAtoms;
        }

        private void BindGetFromCacheTestRepeater()
        {
            rptGetFromCacheTest.DataSource = GetGetFromCacheTestData();
            rptGetFromCacheTest.DataBind();
        }

        private IEnumerable<TimeTestAtom> GetGetFromCacheTestData()
        {
            IList<TimeTestAtom> timeTestAtoms = new List<TimeTestAtom>();

            foreach (ISquashedCache squashedCache in SquashedCaches)
            {
                Stopwatch.Start();
                squashedCache.GetFromCache<string>(CacheKey);
                Stopwatch.Stop();

                TimeTestAtom timeTestAtom = new TimeTestAtom
                {
                    TestName = squashedCache.Name,
                    ElapsedMilliseconds = Stopwatch.Elapsed.TotalMilliseconds
                };

                timeTestAtoms.Add(timeTestAtom);
            }

            return timeTestAtoms;
        }

        private void BindDataIntegrityTestRepeater()
        {
            rptDataIntegrityTest.DataSource = GetDataIntegrityTestData();
            rptDataIntegrityTest.DataBind();
        }

        private IEnumerable<DataIntegrityTestAtom> GetDataIntegrityTestData()
        {
            IList<DataIntegrityTestAtom> dataIntegrityTestAtoms = new List<DataIntegrityTestAtom>();

            foreach (ISquashedCache squashedCache in SquashedCaches)
            {
                string cachedContent = squashedCache.GetFromCache<string>(CacheKey);

                DataIntegrityTestAtom dataIntegrityTestAtom = new DataIntegrityTestAtom
                {
                    TestName = squashedCache.Name,
                    AreEqual = cachedContent == WarAndPeaceText
                };

                dataIntegrityTestAtoms.Add(dataIntegrityTestAtom);
            }

            return dataIntegrityTestAtoms;
            
        }

        private IEnumerable<ISquashedCache> CreateAllSquashedCaches()
        {
            IList<ISquashedCache> squashedCaches = new List<ISquashedCache>();
            squashedCaches.Add(CreateNewNullSquashedCache());
            squashedCaches.Add(CreateNewDeflateSquashedCache());
            squashedCaches.Add(CreateNewGZipSquashedCache());
            return squashedCaches;
        }

        private ISquashedCache CreateNewNullSquashedCache()
        {
            return CreateNewSquashedCache("Null Cache", MaxSize, CompressorType.Null);
        }

        private ISquashedCache CreateNewDeflateSquashedCache()
        {
            return CreateNewSquashedCache("Deflate Cache", MaxSize, CompressorType.Deflate);
        }

        private ISquashedCache CreateNewGZipSquashedCache()
        {
            return CreateNewSquashedCache("GZip Cache", MaxSize, CompressorType.GZip);
        }

        private ISquashedCache CreateNewSquashedCache(string cacheName, string maxSize, CompressorType compressorType)
        {
            SquashedCacheSettings squashedCacheSettings = CreateNewSquashedCacheSettings(cacheName, maxSize, compressorType);
            return SquashedCache.CreateNewSquashedCache(squashedCacheSettings);
        }

        private SquashedCacheSettings CreateNewSquashedCacheSettings(string cacheName, string maxSize, CompressorType compressorType)
        {
            return new SquashedCacheSettings
            {
                Name = cacheName,
                MaxSize = StringUtil.ParseSizeString(maxSize),
                Compressor = Factory.CreateNewCompressor(compressorType),
                CompressionEnabled = true
            };
        }

        private static string GetWarAndPeaceText()
        {
            WebRequest webRequest = (HttpWebRequest)WebRequest.Create(WarAndPeaceUrl);
            HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse();
            
            string warAndPeaceText = string.Empty;

            using (StreamReader streamReader = new StreamReader(httpWebResponse.GetResponseStream()))
            {
                warAndPeaceText = streamReader.ReadToEnd();
            }

            return warAndPeaceText;
        }

        private long GetUncompressedSize()
        {
            return SquashedCaches.FirstOrDefault().Compressor.GetDataSize(WarAndPeaceText);
        }

        private class CompressionRatioAtom
        {
            public string TestName { get; set; }
            public long CompressedSize { get; set; }
            public decimal CompressionRatio { get; set; }
        }

        private class TimeTestAtom
        {
            public string TestName { get; set; }
            public double ElapsedMilliseconds { get; set; }
        }

        private class DataIntegrityTestAtom
        {
            public string TestName { get; set; }
            public bool AreEqual { get; set; }
        }
    }
}

From my screenshot below, we can see that both compression algorithms compress War and Peace down to virtually the same ratio of the original size.

Plus, the add operations are quite expensive for the true compressors over the “Null” compressor — GZip yielding the worst performance next to the others.

However, the get operations don’t appear to be that far off from each other — albeit I cannot truly conclude this I am only showing one test outcome here. It would be best to make such assertions after performing load testing to truly ascertain the performance of these algorithms. My ventures here were only to acquire a rough sense of how these algorithms perform.

In the future, I may want to explore how other compression algorithms stack up next to the two above. LZF — a real-time compression algorithm that promises good performance — is one algorithm I’m considering.

I will let you know my results if I take this algorithm for a dry run.

Gobble Gobble!