Be aware of "free" API when building any kind of your mash-up or start-up.

Hello all, this is story of my non-profit project that stuck with data from other API.

It's gone couple of years, when I started to make artificialcasting.com

The idea was simple, I have take a global rank of sites and try to figure out the "movers and shakers". Only one service I have found with this kind of information available for free - was Alexa.com

I have looking to find this kind of growing web-sites to predict the NEXT BIG THING.

For me, it was a backside non-profit project. Yes, of course I'm looking to make something usefull to people, but the quality of results was discouraging some time (and still yet). For example: a lot of viruses, torrents, sex oriented web sites still appear in results. That's why we did not annonce to public this project, only for friends and co-workers. I've even made a bayes-classification to predict is this good site or not.

After friend of mine, Alex, was involved in this project, we make much more good looking web site than it was before. 

Also, we make a new kind of algorithm to analyze our database. For this moment db contains 358,000.000 (358 millions) data-points. (Previous data-points that I collected from 2013 we have erased, cause it was not continious time series, with holes).  We have improve algorythm, figure out which moving average/linear regression, time windows, etc . is better to make a good report. 

For next step, we are prepare to mark web sites to their markets. To analyze which market growing faster than any else.. But this plans it seems will never be reality.

Today, I visit my site, to watch new report (it's was rebuilded each morning) and find, that something was wrong. 

The last data was at 18/11/2016. I think it's the last day of life of artificialcasting.com :( 

Since this day, in Alexa file that we used: "Need Alexa Top Sites data?  Find our APIs here: https://aws.amazon.com/alexa/"

API cost for this moment:  $0.0025 per URL returned (e.g. $.25 for 100 URLs) (Alexa Top Sites is sold by Amazon Web Services, Inc.)
For example, to get ranks for the 1 million web sites - we need to pay $2500 USD (each day!).

Of course, we don't plan to buy this data for non-profit project..

I have understanded the risk of going to pay for API from the start of project. I have no claims of course to anybody. But, the lesson was learned: Never stuck to other companies free API.

It sometimes looking good, for first, but after time when it will be cost as such microscopic amount like - $0.0025 per request, what you will do if you need millions of requests?

I think that much more better strategy was to spread the free addon

to collect the user data, like Alexa do. And after make this kind of project. But for this moment we have no more data. That's how business is working in Internet: collect the data from people for free, make it available for the money.