Coding 101 45

Using APIs to scrape product information from websites.

The Code for today's show is available below:
Santa's Little Helper 2/3

Computing into Human Territory

Software has advanced to the point at which event predicting & brute-force recall is better than humans IN THE SMALL SCALE.

-- Deep Blue in 1997 beating Garry Kasparov

Now a team has developed a machine that can out-duel humans in the extraction and cataloging of data IN THE LARGE SCALE.

• A team of researchers at the University of Wisconsin-Madison, led by Christopher Re, a professor at the University, have created a computer that can extract data from scientific publications and place it in a database with tens of thousands of studies.
• This has been VERY difficult for conventional computers to do, because while they can parse data more quickly than the human mind, the categorization of so many studies, all of which their own style of organization, has been borderline impossible.

Example:

• The human mind just understands the connection between the space program and the manufacturing of liquified gasses.
• For a computer, unless there has been an existing correlation made to connect the two, that connection will not be made.

The New machine is called "PaleoDeepDive"

• It is looking at all the data that has been manually entered into the Paleobilogy Database
• That repository is the destination of data from the NSF (National Science Foundation) and many international organizations.
• Ultimately, this is a boon for "Big Data Mining" of disparate scientific sources.

Ivory Tower

Remember our 4 steps for developing an app:

1. Break down the functions of the app.
2. Gather resources: What services/APIs are available for my app?
3. Create the Logic Tree: What are the decisions that need to be made for a functional app?
4. Code each part of the logic tree possible:

Code Warrior

If you want to play with Xamarin. Visit their site and get the free version of their tool suite. As long as you're not compiling HUGE apps, it should be absolutely free.

If you want a tool to help you convert JSON to C#, try JSON2CSharp.

Code Warrior

Lets build a provider against one of the Retailer API's!

1. Construct a common base class that designates exactly what you want it to do. (e.g. GetPriceById, GetPriceFromUrl)
2. Review the API Documentation and provide a REST Query to create the search. - This will be using a base class we created called "UrlEncodingParser" that will all us to construct a URL Query String for each REST search and strongly type it.
3. Build a simpe query and test it out. You get a large JSON blob back and it is easier to use if it is strongly typed.
   • Using tools like Fiddler, Visual Studio Debugger, we can extract the JSON blob and build a strongly typed version of it in .Net using www.JsonUtils.com

You can find the Best Buy AP here

The JSON Utility we demonstrated on the show is here

Get in Touch With Us!

Subscribe and get Coding 101 automatically at https://twit.tv/shows/coding-101!

Follow PadreSJ on Twitter.

Bandwidth for Coding 101 is provided by CacheFly.

• Transcripts: Coding 101 45 (Transcript)