{ "cells": [ { "cell_type": "markdown", "id": "4db86daf", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Extracting data from the web\n", "\n", "#### *Vegard H. Larsen (Department of Data Science and Analytics)*" ] }, { "cell_type": "markdown", "id": "1fd4f4ff", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# 1. Extracting data from the web" ] }, { "cell_type": "markdown", "id": "7b7ef419", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## HTML Scraping" ] }, { "cell_type": "code", "execution_count": 1, "id": "ad3923e9", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Let´s use a package from the Standard Library to open a webpage \n", "\n", "from urllib.request import urlopen" ] }, { "cell_type": "code", "execution_count": 2, "id": "b447a48f", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# We can look at the HTML content of a webpage\n", "\n", "with urlopen('https://www.bi.edu') as response:\n", " bi_homepage = response.read()" ] }, { "cell_type": "code", "execution_count": 6, "id": "e2bf796f", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# We can turn the file into a Python string\n", "\n", "bi_homepage_as_html_text = bi_homepage.decode('utf-8')" ] }, { "cell_type": "code", "execution_count": 7, "id": "10833bf8", "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "'\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n\\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n\\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n\\r\\n \\r\\n\\r\\n \\r\\n\\r\\n\\r\\n \\r\\n \\r\\n \\r\\n BI Norwegian Business School | BI\\r\\n \\r\\n\\r\\n \\r\\n \\r\\n \\r\\n\\r\\n\\r\\n\\r\\n \\r\\n \\r\\n \\r\\n\\r\\n \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n
\\r\\n Skip to main content\\r\\n \\r\\n
\\r\\n\\r\\n
\\r\\n \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n -\\r\\n\\r\\n \\r\\n\\r\\n\\r\\n

Norwegian Business School BI

\\r\\n\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n
\\r\\n \\r\\n
\\r\\n
\\r\\n
\\r\\n \\r\\n
\\r\\n \\r\\n

Meet us abroad

\\r\\n \\r\\n \\r\\n

We are going on an international tour! Meet us in a town near you for more information about our programmes and personal counselling.

\\r\\n
\\r\\n View the schedule\\r\\n
\\r\\n
\\r\\n\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

Bachelor programmes

\\r\\n \\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

Master programmes

\\r\\n \\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

Executive

\\r\\n \\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

PhD

\\r\\n \\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n Faculty and research\\r\\n

Research with impact

\\r\\n \\r\\n \\r\\n

BI's researchers contribute with research at the top international level. This contributes to new knowledge that provides better decisions, smarter ways of doing things, as well as new products and services.

\\r\\n
\\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

Find programmes and courses

\\n

Explore the wide variety of programmes and courses we offer at BI with a simple search on what interests you (e.g. business, culture or finance).

\\r\\n
\\r\\n
\\r\\n \\r\\n
\\r\\n
\\r\\n \\r\\n Search full-time programmes\\r\\n \\r\\n \\r\\n \\r\\n Search part-time programmes\\r\\n \\r\\n \\r\\n
\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n \\r\\n

News

\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n
\\r\\n \\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Sign\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Sign\\r\\n
\\r\\n
\\r\\n
\\r\\n

\\r\\n
\\r\\n \\r\\n

Sign up for our newsletter

\\r\\n \\r\\n
\\r\\n
\\r\\n

I would like to receive news, invitations and other relevant communication from BI Norwegian Business School.

\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"About\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"About\\r\\n
\\r\\n
\\r\\n
\\r\\n

\\r\\n
\\r\\n \\r\\n

About BI

\\r\\n \\r\\n
\\r\\n
\\r\\n

BI Norwegian Business School is an independent, not-for-profit foundation and the main provider of research based knowledge on business and management disciplines in Norway.

\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Smiling\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Smiling\\r\\n
\\r\\n
\\r\\n
\\r\\n

\\r\\n
\\r\\n \\r\\n

Business and Alumni

\\r\\n \\r\\n
\\r\\n
\\r\\n

BI aims to strengthen the quality of interaction with the business society and to structure a beneficial collaboration between attractive national and international businesses.

\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Books\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n \"Books\\r\\n
\\r\\n
\\r\\n
\\r\\n

\\r\\n
\\r\\n \\r\\n

BI Library

\\r\\n \\r\\n
\\r\\n
\\r\\n

Welcome to BI’s Library, one of Europe’s most modern academic libraries.

\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n\\r\\n
\\r\\n
\\r\\n
\\r\\n
\\r\\n\\r\\n
\\r\\n
\\r\\n\\r\\n \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bi_homepage_as_html_text" ] }, { "cell_type": "code", "execution_count": 8, "id": "fcff0065", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/plain": [ "8" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# We can analyze the content on the webpage \n", "\n", "bi_homepage_as_html_text.count('Oslo')" ] }, { "cell_type": "markdown", "id": "dc99d09d", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Making sense of HTML code: BeautifulSoup" ] }, { "cell_type": "code", "execution_count": 9, "id": "fbd2cce2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " BI Norwegian Business School | BI\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " Skip to main content\n", " \n", " \n", "
\n", "
\n", " \n", " \n", " \n", " -\n", " \n", " \n", "

\n", " Norwegian Business School BI\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", "

\n", " Meet us abroad\n", "

\n", " \n", " \n", " \n", "

\n", " We are going on an international tour! Meet us in a town near you for more information about our programmes and personal counselling.\n", "

\n", "
\n", " \n", " View the schedule\n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", "

\n", " Bachelor programmes\n", "

\n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

\n", " Master programmes\n", "

\n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

\n", " Executive\n", "

\n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

\n", " PhD\n", "

\n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", " Faculty and research\n", " \n", "

\n", " Research with impact\n", "

\n", " \n", " \n", " \n", "

\n", " BI's researchers contribute with research at the top international level. This contributes to new knowledge that provides better decisions, smarter ways of doing things, as well as new products and services.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " Find programmes and courses\n", "

\n", "

\n", " Explore the wide variety of programmes and courses we offer at BI with a simple search on what interests you (e.g. business, culture or finance).\n", "

\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", " \n", " Search full-time programmes\n", " \n", " \n", " \n", " \n", " \n", " \n", " Search part-time programmes\n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

\n", " News\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Sign\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Sign\n", "
\n", "
\n", "
\n", "

\n", "

\n", "
\n", " \n", "

\n", " Sign up for our newsletter\n", "

\n", " \n", "
\n", "
\n", "

\n", " I would like to receive news, invitations and other relevant communication from BI Norwegian Business School.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"About\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"About\n", "
\n", "
\n", "
\n", "

\n", "

\n", "
\n", " \n", "

\n", " About BI\n", "

\n", " \n", "
\n", "
\n", "

\n", " BI Norwegian Business School is an independent, not-for-profit foundation and the main provider of research based knowledge on business and management disciplines in Norway.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Smiling\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Smiling\n", "
\n", "
\n", "
\n", "

\n", "

\n", "
\n", " \n", "

\n", " Business and Alumni\n", "

\n", " \n", "
\n", "
\n", "

\n", " BI aims to strengthen the quality of interaction with the business society and to structure a beneficial collaboration between attractive national and international businesses.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Books\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \"Books\n", "
\n", "
\n", "
\n", "

\n", "

\n", "
\n", " \n", "

\n", " BI Library\n", "

\n", " \n", "
\n", "
\n", "

\n", " Welcome to BI’s Library, one of Europe’s most modern academic libraries.\n", "

\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n" ] } ], "source": [ "from bs4 import BeautifulSoup\n", "\n", "soup = BeautifulSoup(bi_homepage_as_html_text, 'html.parser')\n", "print(soup.prettify())" ] }, { "cell_type": "code", "execution_count": 10, "id": "9cc5178b", "metadata": { "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/plain": [ "[

Meet us abroad

,\n", "

Bachelor programmes

,\n", "

Master programmes

,\n", "

Executive

,\n", "

PhD

,\n", "

Research with impact

,\n", "

Find programmes and courses

,\n", "

News

,\n", "

Gender differences are often differences in power

,\n", "

Improving access to drugs: the critical role of supply chain risk

,\n", "

Making the magic happen at Oslo Business Forum

,\n", "

BI’s joint MBA with Fudan ranked top 30 worldwide

,\n", "

Sign up for our newsletter

,\n", "

About BI

,\n", "

Business and Alumni

,\n", "

BI Library

]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soup.find_all('h2')" ] }, { "cell_type": "code", "execution_count": null, "id": "9a3409df-cd34-4af5-97b5-76ac8fbb1103", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "f3e8a469-5b69-45a2-a159-5afef6697f82", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "e507a669-fb13-4200-b6ef-ea71d0b909aa", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.10" } }, "nbformat": 4, "nbformat_minor": 5 }