Unless you truly have a unique take or years of experience with what you’re presenting, the quickest and one of the best ways to provide value in content is to provide a unique take on data. And luckily that’s something the web is chock full of.

But have you ever been gathering facts on a site chock full of data and had to look elsewhere because there was no way to copy it all into a spreadsheet? Don’t try to tell me this hasn’t happened to all you content creators out there.

Or you could, but the formatting was all off.

Or there were just too many interactions with the site needed to grab each single data point.

You get the point.

In content work, and data disciplines more generally, getting and cleaning data is often more than half of the battle. A number of surveys have shown that many data teams, likely with much more experience wrangling data than many marketers, can spend up to 80% of their data lifecycle just finding and cleaning data.

While no guide could ever enable you to always have clean, ready, and easy to use data, in this guide I’m going to lay out a few concepts and techniques that can help non-technical marketers and content creators level up their data collection and consumption games.

First: What is an API?

API stands for ‘application programming interface’. It sounds more technical and obscure than it really is. An API basically just facilitates a way for two programs to talk to one another.

You use APIs all the time even if you don’t know it. Nearly every web page you load in your browser is something called a “GET” API request sent to the server associated with the website. Lo and behold, when this request is successful, it “gets” some resource that is then displayed in your browser.

Every time you log into a third-party location with your Facebook or Google accounts, you’re using an API. Nearly every app on your phone utilises APIs.

In short, an API helps you send or receive data from a server. While there are many advanced ways to utilise APIs. Just being aware that these are primarily how we get and send data around the web. Learning a few simple techniques can help you to really level up your data acquisition game.

Second: What does this have to do with content?

While APIs can be configured to send or receive a wide variety of data types and structures, most up-to-date APIs (read more about RESTful APIs) hold to certain conventions.

Perhaps most notably, APIs tend to send data in structured formats including CSV, comma-separated values, basically, spreadsheet format, and JSON, Javascript Object Notation, a nested hierarchy of information.

Data is often organised in some way on the public web as well. But as we mentioned in the intro, it’s often spread across a site. Or requires working through a great deal of pagination or is hidden in a tab. Pulling data from an API often allows you to immediately receive your data in a structured format in which you can begin to take action on the data.

There are readers for both of these main data types online, as well as plenty of converters if you’re more comfortable with a spreadsheet than JSON. Just google “JSON to CSV converter online.”

A second benefit of gathering data via API is that most APIs let you filter what is returned to you. This keeps you covered on two fronts, you can often request a great deal more data than you can gain via content on a site or app, and simultaneously ensure it’s only the portions of this data that you need.

A third benefit is that many APIs are updated as soon as the data powering whatever application they’re from is updated. This isn’t the case for a great deal of data content on the web. If you want the most up-to-date data available from a service, it’s likely not on their blog.

Fourth, so much content regurgitates the same points these days. If you’re looking to go beyond the rehashing of the stats or data that’s already publicly available on the web, APIs can help provide you with enough data for a fresh take.

Third: The basics of an API request

For our purposes, an API request just has a few moving parts. While you can certainly get more advanced, many API requests can be understood with the following information.

If you’re wondering how you could determine the following values, note that this information is often housed in the “documentation” of an API. We’ll be working through Accuweather’s forecast API which has documentation here.

An endpoint (sometimes called a “base url”) is the location you’re requesting to send data to. This is a URL. For example, the endpoint for Accuweather’s APIs is the following:

The path, similar to how this term is used in your local file system, specifies what resource you want from a given endpoint. The path basically adds to the endpoint and is also part of the URL. The path format we would add to the above endpoint to get a weather report by zip code can be seen below.

URL parameters are values we provide to an API by including them somewhere in the URL. They’re often specified in the documentation for the API inside of curly brackets (“{“ and “}”).

We can see an example of a URL parameter above where the path includes “{locationKey}.” In this case, we would replace that section with our postal code of choice. Say, for a weather forecast of Madison, Wisconsin:

Query parameters are added to the end of the URL. They specify what data should be returned. You’ve likely seen query parameters within URLs. They start with a question mark, specify a parameter, and then sometimes an option for the parameter.

Accuweather’s forecast API accepts a query parameter that specifies what unit of measurement you want your data returned in. It defaults to metric being false, but you can change your results to metric format by adding the following “metric=true” values. Note that you will oftentimes have multiple query parameters. These are separated by the ampersand symbol as seen below

Header values aren’t specified in the URL. These are typically additional values that may be needed to get the data you need in a format you prefer. A commonly required field is that of an API token, this is essentially a password that allows you to query an API once you’ve signed up.

A second value often passed in the header is what format you would like your data in. I’ll show you how to include a token and header values in the next section.

Body values can be required when the API you’re querying is doing something to some data that you have. Perhaps you have an API that transforms some files you have on your computer. You might specify this as the data you need to send to the API within the body of the call. Many API calls do not require any values in the body.

In this case, you can pass your API key as body values. This is often the best practice not to expose your API key in the URL as a query parameter. But in our case, we aren’t really building a project anyone else will use, in all likelihood. So we can include our API key as a body value or a query parameter. More on this in the following section.

The method specifies what sort of interaction you want to have with the API you’re connecting to. While this is a choice that was made by those who created the API, typically a “GET” request is used to return data from another application, and a “POST” request is used to send data to another application.

Funnily enough, both GET and POST requests both send and receive data from another application. But this convention largely has to do with the primary purpose of the request. There are a number of other request method types, but for our purposes GET and POST are more than enough.

Fourth: Two easy-to-use, low-technical methods for performing API calls

Next, I’m going to walk you through two easy to follow ways to request data from an API. While these may look technical, as long as you follow along carefully, you should be able to make your first API call in a matter of minutes!

First, most API calls require you to register and receive a token or password. Sort of like how you have to sign up for most services online before you can use them. If you want to follow along with Accuweather’s forecast API, you can sign up for a token by clicking register at the top of this page.

Next, you’ll need to “add an app.” We won’t truly be building an app. But as we mentioned before, APIs are essentially ways for applications to talk with one another.

So this is essentially just a way to register yourself with a particular API provider. I called my app “why-content-creators-should-learn-APIs.” Just fill in random fields for this part and you should be ok.

After hitting “create app” your app will likely be approved immediately, and you can find your “key” by clicking on your app name.

Keep this tab open or copy your API key as you’ll need it soon.

Next, you’ll want to open up your command prompt, or Terminal on Mac. If you’ve never used your command line, never fear.

We’re basically just going to paste in a value. There’s a native command in command prompts titled “cURL.” This command allows you to send a GET or POST request to a given URL and receive data.

In its simplest form you can type:

In our case, we’re going to gather all of the components we walked through in the “basics of an API” section into one URL. Take the example URL below and replace “location_code” with a zip code of your choice, and “api_key” with the API key you received from Accuweather.

Once you’ve replaced those values enter the following command into your command prompt and hit enter.

You should see a response with a great deal of weather data. It won’t be easy to read in this format. But copy it out of your command prompt ensuring you get the start and end brackets, and paste it into a JSON viewer. You can Google for your own, or try out one of these.

You still may not be particularly familiar with this data format. But it should be obvious that you were, in this case, able to gather many weather details about a given zip code almost instantly, with structured results, and with just a single line of code.

This data format (JSON) is actually built to be parsed by machines. It’s human-readable to an extent. But not as friendly for non-technical knowledge workers to wrangle with. Luckily there are a number of ways to convert to CSV. I’ll dive into some additional techniques in a future post. But you can convert JSON to CSV with tools like this online.

Depending on the underlying organisation of the JSON, it may parse to usable CSV more or less easily. But ideally, you’re using this technique to pull in large or more comprehensive data sets than you could manually pull out of websites. So even if your data requires some spreadsheet-level wrangling, you’ve saved yourself time and have gotten more data.

Now that we’ve worked through an initial way to obtain API data from the command line, let’s check out another application that can help you to scale your API queries to a greater extent. Postman is a desktop app primarily built for testing APIs. But it can be a great way for non or semi-technical audiences to jump into API calls as well.

First, you’ll need to download and install Postman. Once you’ve created an account and logged in, click “+ New” in the top left-hand corner of the app. Click create “request” and you should see the following screen:

In the bottom section of the modal, you should be able to create a collection. And this is where Postman really helps you to scale up your ability to make a variety of API calls. You can save related queries that you may want to make on a continual basis inside of a collection.

You can also schedule entire collections of API calls for future times, save results, transform results, and even automatically populate series of API calls with variable values of your choice. For instance, if you would like to query a large set of zip codes with the Accuweather API.

Once you’ve created your request it’s really quite simple to send your first query. “GET” is already specified as the request type, you can paste your base URL and path into the bar to the right. For the zip code I’m querying, that’s the following:

Inside the “params” tab, you can add your query parameters as we did for the previous cURL example. In my case, I’ll add “details” with a value of “true” and “apikey” with a value of my API key. You should see these added to the end of your query URL to create a URL identical to what we used in our cURL request.

Click send and your JSON result should appear below! Note that you can select what format you would like the response data in, as well as save the response data to a file to the right of the results.

Now obviously weather data isn’t likely to be the most valuable data source for most content. But what we’re comparing here is the manual process of going to an Accuweather page and trying to copy a disparate collection of data points. Some will be behind tabs, some in graphical format. And imagine trying to do this for hundreds or thousands of pages to get a valuable data set…

How do I get this to work on an API of my choosing?

One final note before we move on is that some readers may be wondering how to determine what values to pass to an API call. This all comes down to reading the API documentation. For our Accuweather one day forecast API example, the documentation can be seen here (or an image below):

Most API documentation provides similar information. What you’ll want to look out for includes:

• Is it a GET or POST request (top right corner of image)?

• What is the main URL to query (labeled ‘resource URL’ in this example)?

• What parameters are available (under ‘query parameters’)?

• What parameters are required (in this case only ‘apikey’)?

The answers to these questions will help you to construct your own URL to query with our two techniques above. That is assuming a site or service has an API.

Fifth: What to do when a site doesn’t have an API

Finally, some sites and services either don’t provide APIs, provide expensive APIs, or don’t provide the data you want from an API. This doesn’t mean, however, that you have to go back to manually grabbing data.

Most sites with valuable data tend to have data of a few types:

• Product pages and product data

• Discussion and social media post data

• Article data

• One-off dataset of interest

If you want to automate the grabbing of a one-off dataset of interest, you can use a point and click web scraper, such as this free Chrome extension. This can be useful in cases where you want to extract a field or two that is repeated in a visually selectable format again and again. For example, you can’t copy and paste the location of each of the top-ranked universities on U.S. News directly.

But you can specify what type of value you want to this simple point and click scraper and it will find similar values. For a more robust guide on this and other scrapers, check out my recent article on ways to utilise web scraping for content.

The downside to this smaller scraper is you need to specify each field type you’re interested in. And if you want to grab data from a range of pages you’ll have to manually go to each page and run your scraper. It’s quicker than copying and pasting. But it’s a slower process than querying an API.

For other well-defined types of pages, like articles, discussions, products, and more, you can also go with an AI-enabled web scraper that’s trained to pull valuable fields from one or more pages.

Diffbot provides automatic extraction APIs for many of the most valuable page types online. If you can’t find an API for an e-commerce site, simply point Diffbot’s Product API at a given page and get structured data back.

It’s like an API for sites that don’t have APIs…

And you can try it out for free by toggling a page type and entering your URL into the form on this page.

You might recognise the format as similar to what was outputted in our cURL and Postman examples. And querying the API is basically just as simple.

Elements to include in your query include:

• Your token

• The URL you wish data to be extracted from

• And an endpoint that corresponds with the right API (in this case, product, article, analyze, discussion, or event API)

Note that you’ll need to “encode” the URL you want to extract data from. Because you’ll be including the URL to parse the inside of the URL you’re requesting an API response from. This basically just keeps the two separate. If you don’t want to mess with this step, just use Postman as it takes this step automatically.

In conclusion

As content becomes more data-driven, the largest barrier to productivity in content creation comes from the time it takes to gather data that’s able to be used. While APIs may seem entirely foreign to many content creators, there are a variety of low-code options for utilising them for fast data acquisition.

Simply familiarising yourself with the basic components of an API and one or two techniques for getting data from these sources can revolutionise how you get data for content.

In this guide we walked through several steps you can take to easily query APIs using cURL or Postman. In addition, we tackled ways in which you can automate data gathering from sites that don’t provide an API. Notably, by finding an API for sites that don’t have one!

Stuck on these tutorials? Feel free to reach out below and I’ll try and help step you through the process.

Resources:

• Download Postman

• Best Practices For Web Scraping For Content

• Try To Scrape Any Page With Diffbot’s Test Drive

=====