Selenium is a web automation tool. It can be used to crawl and scrape websites. It requires a web browser installed to use as a web driver: we’re going to use Firefox. We’ll also need to trick Firefox into thinking there is a display attached: we’ll user Xvfb for that.
One setup, you’ll be able to run Selenium scripts at your pleasure.
- A virgin Alpine Linux container
- A medium knowledge of Linux
Add Firefox repository to APK
As of this blog post, Firefox is not available in the main alpine repository. You can search the Alpine repository to find out where you can install Firefox from. In my case, it is available in the
community repository branch
With that information, I can add the following line to
Update your cache to pull from the new repository
apk upgrade --update-cache --available
Install Selenium dependencies
Now you should be install Firefox and all other Selenium dependencies without issue.
apk add xvfb firefox dbus py-pip ttf-dejavu
Here is what we just installed:
|xvfb||A virtual display driver. We're building a headless system, so we need a virtual display for our web browser to run in.|
|firefox||We will use the Firefox driver for Selenium. This is what we will use for scraping web sites.|
|py-pip||Used to install Selenium drivers for Python|
|ttf-dejavu||Fonts! These are required for Firefox to render pages|
Setup Xvfb virtual display server
Test Xvfb is setup properly
We should be able to run the Xvfb virtual display server by running
Xvfb :99 -ac &
Check to make sure your display server is running using the
Note: If you get an error complaining about a machine-id, install the
dbus package and run
dbus-uuidgen > /etc/machine-id. You can then uninstall dbus.
Setup Xvfb to start on system boot
We will use the
local service to create a basic script that will start our service at boot time. We’ll start by enabling the
local service to run at boot.
rc-update add local default
Now we need to create our start script. When
local service is enabled, it will run all executable scripts in the
/etc/local.d/ that end with
.start at boot time. It will run all
.stop scripts when the
local service is stopped
In our case, we just need a single script
/etc/local.d/Xvfb.start with the following lines:
#!/bin/sh /usr/bin/Xvfb :99 -ac &
Now make it executable:
chmod +x /etc/local.d/Xvfb.start
Note: Now is a good time to test that everything is working smoothly. Stop and start your container and then check to see if the Xvfb service is running
Note: Creating a script to shutdown Xvfb is left as an exercise for the reader
Install Selenium with Python
We’re going to use Python Selenium bindings because it’s really easy to get started.
pip install --upgrade pip # ensure pip is upgraded to latest version pip install selenium requests
That’s it. You can checkout the Selenium Python Getting Started Guide.