Despite the disastrous effects of the same policy in Spain, the European Union is flirting with the idea of a link tax. This autumn’s proposals for copyright reform in Europe might contain all sorts of good things — not least, Hargreaves-like rules for content-mining — there is also the real possibility that they will also propose requiring payment for linking to content.
The problem is that some publishers feel that, by including snippets of text from their web-sites, content aggregators such as Google News take eyeballs away from them.
No matter that studies show the opposite — that including a site in such aggregation services actually increases their traffic. This is matter for each publisher to make its own decision about. If they don’t want their content to be aggregated, then they shouldn’t be forced to participate in aggregators.
So I have a proposal: let’s solve the problem with the scalpel of technology rather than the sledgehammer of legislation.
We should introduce a simple standard whereby any publisher that doesn’t want its content to be included places a simple text-file called “robots.txt” at the top level of its web-site. If that site doesn’t want to be included at all, it would simply place these lines into the file:
And a publisher that only wants to omit certain areas of its web-site — the /news/ area, for example — could say “Disallow: /news/” instead, for finer grained control.
Then all we’d need to do is persuade aggregators to implement code that reads and respects this “robots.txt” file, and publishers that don’t want to be included would have a nice, simple way to opt out.
I can’t believe no-one’s thought of this before.