Machine Learning

, Volume 57, Issue 1, pp 83-113

First online:

Lessons and Challenges from Mining Retail E-Commerce Data

  • Ron KohaviAffiliated
  • , Llew MasonAffiliated withBlue Martini Software
  • , Rajesh ParekhAffiliated withBlue Martini Software
  • , Zijian ZhengAffiliated withMicrosoft Corporation


The architecture of Blue Martini Software's e-commerce suite has supported data collection, data transformation, and data mining since its inception. With clickstreams being collected at the application-server layer, high-level events being logged, and data automatically transformed into a data warehouse using meta-data, common problems plaguing data mining using weblogs (e.g., sessionization and conflating multi-sourced data) were obviated, thus allowing us to concentrate on actual data mining goals. The paper briefly reviews the architecture and discusses many lessons learned over the last four years and the challenges that still need to be addressed. The lessons and challenges are presented across two dimensions: business-level vs. technical, and throughout the data mining lifecycle stages of data collection, data warehouse construction, business intelligence, and deployment. The lessons and challenges are also widely applicable to data mining domains outside retail e-commerce.

data mining data analysis business intelligence web analytics web mining OLAP visualization reporting data transformations retail e-commerce Simpson's paradox sessionization bot detection clickstreams application server web logs data cleansing hierarchical attributes business reporting data warehousing