3 Hub Tools: What They Are and How to Use Them
3.1 Overview
The VBD Hub brings together datasets, tools, and community resources to support vector-borne disease research. In the pre-work, you learnt how to navigate the Hub and where to find its useful resources. This session builds on that by focusing on two tools for searching and retrieving data, Hub Search, the Hub’s web-based search interface, and ohvbd, an R package that allows you to search for and retrieve data directly in R. During this workshop, you will practice using both, learn how to refine your searches, and check what they return, and compare the two tools so you can choose the one that best fits your workflow.
3.1.1 Learning Objectives
By the end of this workshop, you should be able to:
- Use Hub Search filters to search and download datasets relevant to a specific research question.
- Build an ohvbd pipeline in R to search for, filter, and fetch VBD data.
- Refine ohvbd searches using smart search functions.
- Inspect retrieved data to check it matches your research questions and identify whether wrangling is required.
- Understand the differences between the Hub Search and ohvbd package, and choose the right tool for your workflow.
3.2 Searching and Retrieving Data with Hub Search
As we have seen, the Hub Search can be found under the Find Data tab on the VBD Hub website.
The Hub Search makes discovering datasets much easier by allowing you to search across multiple data sources in one place. This means you can identify datasets relevant to your research and explore the associated metadata before downloading, rather than searching each database source individually.
On the Find Data page, you will see a Filter panel with several options that let you refine your search. These include:
- Category - Filter your search based on data type, such as Occurrence, Abundance, Traits, Proteomics, and Epidemiological data.
- Database - Filter by source database, including VecDyn, VecTraits, GBIF, ProteomeXchange, and the VBD Hub.
- Published - Set a date range for when the data was published by entering a start and end date.
- Location - Define a geographic area by drawing a polygon on the map.
- Taxonomy - Search for a specific species or taxonomic group.
- Full text search - Refine your search by using specific keywords.
These filters help you to narrow down your search results and find datasets that are the most relevant to your research.
Note: The Hub Search tool retrieves data from several open-access databases. If you are unfamiliar with these databases, they include:
- VecDyn - Data on vector population dynamics, including how vector populations change over time and across locations.
- VecTraits - Data on vector traits, such as life history, behaviour, and ecology.
- GBIF - Species occurrence records, showing the time and location vector species have been observed.
- ProteomeXchange - Proteomic and molecular vector data.
Once you have run a search, the results will be displayed on the same page. These typically include the dataset name, the source database, and a brief description of the dataset.
You can also access further details, including the metadata, and geographic or temporal coverage. Clicking on the dataset will let you see options to access or download the dataset.
These details can help you decide whether the dataset is suitable for your research before downloading it.
After reviewing your results, you might decide to refine your search. You might consider:
- Relevance - does the dataset answer your research question?
- Coverage - does the dataset include the correct location or time period for your research?
- Structure - is the data in a usable format?
You can refine your search by adding more filters, using more specific keywords, or combining terms, such as “species name + location”.
Tip: Start with a broad search, and then gradually narrow it down.
Remember, you don’t have to download everything - you should focus on datasets that are the most useful for your research and analysis.
Task: Retrieving Ixodes ricinus Data Using Hub Search
Use the Hub Search and relevant filters to find datasets on abundance of Ixodes ricinus. How many results did your search return?
Select one dataset from your search and identify the:
- Dataset name
- Publication date
- Source database
3.3 Searching and Retrieving Data with the ohvbd R Package
ohvbd is an R package developed by the VBD Hub that allows you to search for and retrieve data within R, without needing to download files manually from multiple sources.
It connects to several VBD databases at once, including:
- VecTraits (vt)
- VecDyn (vd)
- GBIF (gbif)
- ProteomeXchange (px)
-
AREAdata, which is accessed with separate functions, such as
fetch_ad()
This means you can pull datasets directly into your R workflow, making data retrieval much more efficient.
3.3.1 Building a Search Pipeline with ohvbd
To get started, we need to install and load the ohvbd package in R:
install.packages("ohvbd")
library(ohvbd)Once installed, we can start searching for data.
The ohvbd package uses a piped workflow, which allows us to build our search step-by-step.
If we wanted to search for data on Ixodes ricinus from the VecTraits database, we can run:
ixodes_ricinus_data <- search_hub("Ixodes ricinus") |>
filter_db("vt") |>
fetch() |>
glean()Let’s break this down a bit so we can understand what each step is doing:
-
search_hub()- Searches for datasets matching your query, here we want to search for"Ixodes ricinus". This gives us the IDs of matching datasets, not the data itself. -
filter_db()- Narrows results to a specific database, in this case"vt", for VecTraits. -
fetch()- Downloads the data for those IDs, in its raw format. -
glean()- Converts the raw data into a usable dataframe.
Tip: We can consider a basic search with ohvbd as 4 stages:
- Find the data.
- Filter the search.
- Fetch the data.
- Format the data.
The data retrieved using ohvbd is often raw, and its structure depends on how the original source database is formatted. This means that after retrieving the data, you will usually need to wrangle the data before you start any analysis.
3.3.2 Quick Searches with ohvbd
If you know which database you want to use, you can make your search more efficient by specifying the details directly into search_hub:
search_hub("ixodes ricinus", db = "vt")This gives the same result as filter_db(), but is more efficient because it only searches the database you need.
Tip: You can write ?search_hub or ?fetch_vt in the R console to access guide documents if you need extra help using ohvbd.
3.3.3 Refining an ohvbd Search
We can make our search more refined by adding more parameters to search_hub():
search_hub(
query = "",
db = c("vt", "vd", "gbif", "px"),
fromdate = NULL,
todate = NULL,
locationpoly = NULL,
taxonomy = NULL,
exact = FALSE,
withoutpublished = TRUE,
returnlist = FALSE
)Let’s have a look at what each of these arguments does:
-
query- What you are searching for, such as a species name. -
db- Which databases we want to search -
fromdateandtodate- Filter by a specified date range. These parameters use ISO format (yyyy-mm-dd). -
locationpoly- Define a geographic search area. -
taxonomy- Search by species ID. -
exact- Return exact matches only. -
withoutpublished- Include results without publication dates. -
returnlist- Return the raw output list of datasets, rather than a formatted dataframe.
Tip:
You do not need to use all of these arguments at once. Start simple and build a search pipeline that aligns with your search aims.
Frequent mistake:
taxonomy and locationpoly need inputs in a specific format, which ohvbd can create for you:
-
match_species()finds species IDs from GBIF, using thergbifpackage. For example,match_species("Culex pipiens"). -
match_countries()creates a country outline, using thernaturalearthpackage. For example,match_countries("United Kingdom").
You can find out more on the match_species() and match_countries() reference pages.
Example: Retrieving Culex pipiens Data Using ohvbd
Imagine we want to retrieve population data on Culex pipiens from VecDyn.
We can start by defining our search query and filter by database using search_hub():
search_hub(
query = "Culex pipiens",
db = "vd")We can also make our search more specific by adding a date filter directly into our search. For example, we might only want datasets published from 2010 onwards:
search_hub(
query = "Culex pipiens",
db = "vd",
fromdate = "2010-01-01")Note that this filters by when the dataset was published, not when the data were collected.
We can also add:
-
exact = TRUEso our search only retrieves exact matches, -
withoutpublished = FALSEso datasets without publication dates aren’t retrieved.
search_results <- search_hub(
query = "Culex pipiens",
db = "vd",
fromdate = "2010-01-01",
exact = TRUE,
withoutpublished = FALSEThis search returns the IDs of the datasets that match our search criteria. We have stored these in an object called search_results, which contains information about each dataset, rather than the data itself. This allows us to review the results before retrieving the full datasets. We can view them by running search_results in the R console.
Once we are happy with the results, we can retrieve the data with fetch(), and use glean() to extract it and convert it into a usable dataframe:
Now we have retrieved the data, we can have a quick look by running:
head(culex_pipiens_data)We can save the dataset for future use using write.csv():
write.csv(culex_pipiens_data, "culex_pipiens_data.csv", row.names = FALSE)
3.3.4 Unexpected Search Results
As you become more familiar with the ohvbd package, you might find that your searches do not always return exactly what you expect.
This is because searches are not only based on exact matches, but also on how datasets are indexed and described within the underlying databases.
For example, a search term may match multiple fields within the dataset:
- Species names
- Dataset titles
- Metadata descriptions
This means that a search can return results that are related to your query, but not exactly what you intended.
Example: Unexpected Results from a Bemisia Search
Let’s have a look at an example of a search that returns unexpected results.
Imagine we are searching for trait data on Bemisia, a genus of whitefly, using the VecTraits database:
df <- search_hub("Bemisia", db = "vt") |>
fetch() |>
glean()
head(df)At first glance, everything may look fine. However, if we explore the dataset further, for example, by checking the unique values in the genus column, we might notice something unexpected:
unique(df$Interactor1Genus)Alongside Bemisia, another genus appears: Axinoscymnus, which is a ladybird. This can seem confusing, we searched for whiteflies, not ladybirds.
If we look more closely at the dataset, we can see that:
-
Axinoscymnus appears in the
Interactor1Genuscolumn -
Bemisia appears in the
Interactor2Genuscolumn
This tells us that the dataset includes interactions between species. So while Bemisia is included in the search results, it is not always the primary species the trait data refers to.
To mitigate unexpected results, the ohvbd package provides more advanced ways to refine your search.
One key approach is to use smart search functions, written as search_x_smart(), where x can be replaced with the code for the appropriate database, such as vt for VecTraits, or vd for VecDyn.
Unlike search_hub(), which searches across multiple fields, smart search functions let you choose which field to search, how to match it (e.g. "equals" or "contains"), and the value to look for. This means you can control exactly where your search term is being matched.
Example: Unexpected Results from a Bemisia Search (cont.)
Instead of searching broadly for Bemisia, we can refine our search using a smart search function.
Our previous code (see previous Example) returned datasets where Bemisia appeared in multiple columns, including as a secondary interacting species.
To make the search more specific, we can use search_vt_smart() to target the column that represents the primary species the traits refer to:
df_smart <- search_vt_smart("Interactor1Genus", "equals", "Bemisia") |>
fetch() |>
glean(
cols = c("Interactor1Genus", "Interactor1Species", "Interactor2Genus"),
returnunique = TRUE
)
unique(df_smart$Interactor1Genus)Here we have also used cols in glean() to keep only the columns we need, and returnunique = TRUE to remove any duplicate rows.
We can see that only Bemisia appears in the primary species column. This mitigates the unintended results, such as Axinoscymnus, that appeared in our earlier search.
By using smart search functions, we can more precisely control how our query is applied, ensuring that the data we retrieve aligns with our research question.
In addition to refining the search itself, it is often necessary to apply data wrangling techniques after retrieval.
These might include:
- Filtering rows based on specific conditions
- Standardising or correcting species names
- Merging columns where necessary
These steps are a normal part of working with integrated datasets, where multiple species, interactions, or naming conventions may be present.
By combining targeted searching with careful data wrangling, we can ensure that the final dataset aligns with our specific research question.
3.4 Hub Search vs. ohvbd
You might have noticed that the ohvbd package and the Hub Search are quite similar, as they both allow you to search across multiple databases.
However, there are some differences between the two resources:
- With ohvbd, once your code is written, it can be re-run to retrieve updated data, which improves reproducibility.
- Additionally, all results can be retrieved and combined directly within R.
- With Hub Search, datasets are typically downloaded manually one at a time.
- ohvbd is most useful for users who work in R, whereas users who work with other software will find the Hub Search more appropriate.
Both tools are useful, and the best choice depends on your workflow and experience.
Task: Retrieving the Same Dataset with Hub Search and ohvbd
Have a go at retrieving the data from this paper using both the Hub Search and ohvbd: Blagrove MSC, Arias-Goeta C, Di Genua C, Failloux A-B, Sinkins SP (2013). A Wolbachia wMel transinfection in Aedes albopictus is not detrimental to host fitness and inhibits Chikungunya virus. PLoS Neglected Tropical Diseases 7(3): e2152.
Remember to think about which filters will narrow your search most efficiently.
For Hub Search, open the dataset, check its metadata, and download it.
For ohvbd, use fetch() and glean() to retrieve the dataset as a dataframe.
Once you have retrieved your dataset, consider:
- Which filters did you use, were these the most efficient?
- Which tool got you to the dataset faster?
- Which tool would you use if you needed this dataset plus several similar ones?
3.5 Key Takeaways
- The Hub Search and ohvbd both let you search several VBD databases at once, so you don’t need to search each source individually.
- Start with a broad search and then narrow down, checking results for relevance, coverage and structure before downloading.
- Searches match across species names, dataset titles and metadata, so results may include data you didn’t expect. Smart search functions with ohvbd let you target a specific column.
- Choose the tool that best fits your workflow. Hub Search suits point-and-click exploration and non-R users. ohvbd is better suited to R users, anyone working with multiple datasets, and anyone who needs a reproducible search they can re-run.