(Base One logo) .NET database and distributed computing tools

Competitive Media Reporting’s Web Ad Tracking System
Technical Details

Base One built an income-producing web site for Competitive Media Reporting (CMR), which reports on ad expenditure and occurrence data. The application, CMR's Web Ad Tracking System, is one of the earliest successful examples of using commercial grid computing software to deliver large-scale data processing for a web site.

In order to meet rush deadlines, CMR decided to do the required back-office computing with Base One's "shrink-wrapped" database programming tools and distributed computing middleware. This project illustrates the kind of development speed-up and improved performance that can be achieved with Base One's software.

Delivery in record time

CMR is the leading provider of marketing communication and advertising expenditure information to advertising agencies, advertisers, broadcasters and publishers. In 1999, they needed a new data consolidation and reporting system fast. Competitors were catching up, and ongoing negotiations for the company's acquisition made it critical to build a commercial web site that would give them a commanding lead.

In just seven months, Base One designed and built CMR's fee-based web site and grid computing database application from scratch. The web site (CMRInteractive, subsequently released as www.AdNetTrackUS.com) monitors all brand activity on over 350 top advertising revenue-driven sites. CMR clients query against millions of banner ad occurrences and tens of thousands of companies and brands, with a hierarchical classification system that includes over one thousand industry categories.

Advertisers use the web site to create custom reports showing where their competitors are placing banners, and how much they are spending on each of their products. Similarly, publishers (commercial web sites) use the facility to find out which companies are advertising on their competition.

The site received enthusiastic reviews from some of CMR's best customers. In early 2000, shortly after the new web site's successful introduction, CMR was sold to UK-based Taylor Nelson Sofres, a worldwide marketing information company.

Dramatic performance improvements simply by adding machines

CMR's distributed application extracts banner data collected by web crawlers, and consolidates it for efficient, dynamic reporting. Long-running tasks are split up so that they can be done faster in parallel by multiple computers.

The company chose to use a small number of high-end, Windows multi-processor computers (as opposed to using a larger number of inexpensive PCs or a mix) for their back-office servers. CMR was able to achieve major performance increases through this design, just by adding an additional machine.

For example, starting with the data collected by web crawlers and pricing data from CMR's analysts, a typical database load and transformation taking 6 hours to run on a single computer ran in about 3.5 hours on two computers. In other words, two machines ran the same jobs nearly twice as fast as one machine.

The necessary processing entailed numerous logical dependencies (jobs that should not start until others are finished). Nevertheless, the amount of logically independent processing that could be performed in parallel was sufficient to achieve near-linear scalability on a small number of these powerful machines.

Distributed applications that are easy to maintain and enhance

When CMR's in-house programming staff took over the web site and grid application, they found the system easy to enhance. None of the complexities of parallel processing appear in the back-office source code, because it looks like a typical client/server database application.

In fact, no programming changes at all were required to use the full range of database and administrative functions, both interactively and in batch computing mode. Likewise, no additional code was needed to insure fault tolerance: any job can fail cleanly, be easily restarted, and continue processing, possibly on a different machine. Also, no CMR-specific code was required to define or change the dependencies between jobs. Instead, the distributed computing software handles dependencies automatically, through its database representation of tasks to be done. (This guarantees, for example, that calculations of quarterly and yearly summaries follow completion of the monthly calculations.)

In addition to processing data from web crawlers, the application runs scheduled and ad-hoc scripts for creating, maintaining, upgrading, importing into and exporting from the web site's database. The same application can run these scripts either locally or remotely over the Internet (with Base One's Internet Server handling the communications).

For example, when CMR wanted to add 100 new users to the web site's database in New York, they were able to quickly create a script and run it securely over the Internet from their development center in Pennsylvania. Similarly, they can use the same application locally or across the Internet to submit, schedule, and monitor jobs that run in the New York data center.

Full-featured security system

The CMR application incorporated Base One's standard, pre-built User Administration facility for assigning security privileges and setting security rules. Multiple logons of the same (paying) user are automatically detected and prevented. Users are tracked without cookies and recognized, regardless of what machines they are using.

End-users of CMR's web site are divided into groups, with one or more groups per customer (advertiser or publisher). Users can create their own custom report specifications and share the report specifications of others belonging to the same group. Those belonging to the same group can view each other's reports, copy and run them, but cannot change them.

 The business problems come first

By building the site on top of Base One's shrink-wrapped software, CMR was able to maximize its development investment. Base One's consultants were able to concentrate on the real business problems to be solved.

CMR clients use AdNetTrackUS to monitor brand and industry growth on the Internet and hone their Internet budgeting. Here are some examples of the kinds of questions they can answer:

  • What brands are advertising on the Internet?
  • Which of my brand's competitors are advertising on the Internet?
  • How rapidly is Internet advertising growing?
  • Who are the top spenders in Internet advertising?
  • What industries are showing the greatest online growth?
  • What percent of a company's overall advertising mix is spent on the Internet?
  • How much are my competitors spending, and on which web sites?
  • What is the advertising mix on a given web site?
  • Who is advertising on my competitors' web site and not mine?

The CMR web site provides multiple methods of searching for companies, brands, web sites, and industry categories, which are used to create the selection criteria in report specifications. Copying of selection criteria is done automatically between the 29 different types of reports, so a user can easily preserve and track a custom list of web sites, companies, brands, and industry categories.

The web-based financial reports include:

  • Complete brand and parent company spending with drill-down to web sites
  • Total ad spending by industry or a brand's custom competitive set
  • Expenditure, percent share and percent change reports, by brand, company, and industry class
  • Web site revenue detail with drill-down to brand activity
All reports have a drill-down feature, with links leading to further detail. For example, clicking on a company listed in a Company Expenditure Report leads to a Web Site Revenue Report, showing the web sites where the company has been advertising. Reports initially display results alphabetically by name, and they can be sorted by revenue/expenditure or percent changed just by clicking on the month, YTD, or percent changed column headers.

Besides displaying useful tables online, users can generate printed reports. Because of the web site's unique high-quality printing capabilities, CMR is well on the way to achieving its goal of replacing the sale of paper-based reports and books with high-end, Internet-based content delivery.

Speeding up development and lowering costs with Base One consulting

Base One completed the web site launch in three staged deliveries of source code and documentation. A Base One system architect met with management, sales staff and users, to pin down the web site requirements, design issues and priorities. The business requirements definition, general design, project planning, and detail design of the database layout were done together, to speed delivery. Implementation time was further shortened by doing detailed design of the web site and database maintenance application concurrently with coding and testing of programs to load and clean the data from CMR's banner-collecting web crawlers.

To speed prototype evaluation and source code deliveries, Base One installed an Acceptance Test system at CMR. This consisted of the web site, database, and distributed computing software, all residing on a single, inexpensive Windows server. When the project was ready, Base One assisted CMR in setting up the production web, database, and batch job servers, and in resolving firewall and other security issues.

Finally, after a couple of releases, Base One turned over all programming responsibility to CMR's IT staff. Quickly they were able to make major enhancements to the web site and back-office application, for example, immediately adding support for storing and displaying graphical ad images.

TNS/CMR - AdNetTrackUS - Web Ad Tracking System

Competitive Media Reporting Case Study - Introduction

Visual Studio | Database Technology | Distributed Computing | BFC

Home Products Consulting Case Studies Order Contents Contact About Us

Copyright © 2012, Base One International Corporation