Most large retailers use programmes, nicknamed bots, to skim through their competitors’ prices.
Amazon was a pioneer in “bot-driven pricing”, whereby retailers use algorithms based on sales forecasts, inventory levels and rivals’ prices – collected by bots – to adapt their own prices. Prices can change several times a day and as often as every few minutes during the Christmas shopping season.
Bots can be disguised as real shoppers by programming a cursor to explore a web page before retrieving the price or can be hidden using multiple computer addresses so that a single source cannot be identified.
As many as 80 per cent of clicks on Amazon product listings could be bots, compared with around a third of traffic on other large sites. E-commerce retailers can thwart bots “crawling” on their own website by using captchas – a system which requires a user to distinguish letters and numbers from images to block bots but allow human users to continue.
“It’s an arms race,” said Keith Anderson, a senior vice president at Profitero, an e-commerce analystics firm. “Every week or every month, there’s some new approach from both sides.”
Engineers at Wal-Mart’s research branch, @WalmartLabs, were responsible for using bots to review Amazon product prices, with millions of clicks per day. However, they found themselves blocked from the listings for weeks and forced to retrieve data from a secondary source.
Amazon’s approach to blocking the Wal-Mart blocks was aimed at a specialised web browser called PhantomJS, typically used by programmers rather than ordinary shoppers. Amazon put up a “digital curtain” to hide its listings from PhantomJS users.
A spokesperson for Amazon said that the company is aware of the use of bots by competitors to check its listings and denied that there was a “campaign” to stop them.
“Nothing has changed recently in how we manage bots on our site,” she said. “[But] we prioritise humans over bots as needed.”
Amazon’s expertise with bots allows the company to prevent rivals from gathering its data, while also gathering data about its rivals. With its rapid deliveries, expansion into original TV production and technological prowess, Amazon has dominated its competition to become the largest online retailer in the world, hitting $100 billion in annual sales in 2015.